feat(tools): add HF model normalization utility

[caixianzhang]

Summary

• add an XTuner xtuner.tools.model_normalize CLI for HF repack and FP8 conversion
• preserve MTP tensors/configuration and copy tokenizer/chat template/generation config assets
• add bounded FP8 save workers, reference-guided or explicit heuristic policy, and one-click shell wrappers
• add --base-model-dir to stamp a user-supplied LICENSE (Copyright line rewritten to Copyright 2025-2026 Shanghai AI Laboratory) and generation_config.json into the product after conversion (supersedes a single-file --generation-config option, which was dropped before merge)
• add usage documentation and focused CLI/MTP tests

Scope

This PR intentionally provides conversion/repacking only. It does not run full-model validation, SHA256 scans, inference services, or Hub uploads.

--base-model-dir (release assets)

Both repack and to-fp8 accept --base-model-dir <dir>, applied after conversion:

• generation_config.json (if present) is copied verbatim into the output, overwriting any existing file (overwrite is logged).
• LICENSE (if present) is copied with its first Copyright line rewritten to the fixed Copyright 2025-2026 Shanghai AI Laboratory; the license body is left untouched. An existing output LICENSE is overwritten (logged). No Copyright line -> file written unchanged with a warning.
• Omitting the option preserves existing behavior; a source-supplied generation_config.json remains in the output as-is.

Validation

• ruff check xtuner/tools/model_normalize tests/tools/test_model_normalize_cli.py
• PYTHONPATH=. pytest -q --confcutdir=tests/tools tests/tools/test_model_normalize_cli.py (base-model-dir/license tests pass; repack test skipped when safetensors is unavailable)
• bash xtuner/tools/model_normalize/run_model_normalize.sh --help
• End-to-end in the glm5-2-8gpu-99022910-cfcd2 job (8x H200):
  • BF16 repack: 353 output shards, model.safetensors.index.json present, LICENSE Copyright rewritten, generation_config.json overridden — rc=0.
  • FP8 conversion: 177 output shards, 59044 *_scale_inv tensors matching the reference, LICENSE/generation_config.json applied — rc=0.
  • Report: dev_notes/workflow/pr2073_base_model_dir_verification.md.

run_glm52.sh usage

The GLM-5.2 example accepts one positional argument:

$1: bf16 | fp8

It does not define other positional arguments; extra arguments are ignored.

Environment variables:

Variable	BF16	FP8	Default	Description
SOURCE_DIR	required	required	—	Input HF model directory
OUTPUT_ROOT	required	required	—	Output root directory
SHARD_SIZE_GB	optional	optional	4	Target shard size in GB
REFERENCE_DIR	not used	required	—	FP8 reference model directory
MAX_SAVE_WORKERS	not used	optional	4	Number of parallel FP8 shard-save workers
BASE_MODEL_DIR	optional	optional	unset	Directory with user-supplied LICENSE and generation_config.json to stamp into the product

BF16 example:

SOURCE_DIR=/path/to/glm52-hf-source \
OUTPUT_ROOT=/path/to/release \
BASE_MODEL_DIR=/path/to/base-model/glm5-2 \
SHARD_SIZE_GB=4 \
bash xtuner/tools/model_normalize/examples/run_glm52.sh bf16

FP8 example:

SOURCE_DIR=/path/to/glm52-hf-source \
OUTPUT_ROOT=/path/to/release \
REFERENCE_DIR=/path/to/glm52-fp8-reference \
BASE_MODEL_DIR=/path/to/base-model/glm5-2 \
SHARD_SIZE_GB=4 \
MAX_SAVE_WORKERS=4 \
bash xtuner/tools/model_normalize/examples/run_glm52.sh fp8

The output directories are fixed relative to OUTPUT_ROOT:

BF16: ${OUTPUT_ROOT}/20_hf_bf16_mtp
FP8:  ${OUTPUT_ROOT}/20_hf_fp8_mtp