Full model card: method, results, format spec, per-layer table

README.md

@@ -0,0 +1,176 @@
+---
+license: mit
+base_model:
+- zai-org/GLM-5.2
+tags:
+- glm
+- moe
+- mxfp4
+- gptq
+- quantization
+- calibrated
+- vllm
+- dgx-spark
+library_name: vllm
+---
+
+# GLM-5.2 GPTQ-Calibrated MXFP4 Routed Experts
+
+**Drop-in replacement expert weights** for [aidendle94/GLM-5.2-Hybrid-FP8-MXFP4](https://huggingface.co/aidendle94/GLM-5.2-Hybrid-FP8-MXFP4):
+the same MXFP4 format, same size, same kernels — but every rounding decision chosen by
+GPTQ against each expert's real routed activations instead of round-to-nearest.
+
+**Result: full-model teacher-forced KL divergence vs BF16 drops 0.098 → 0.080 (−17.7%)**
+at byte-identical memory footprint and serving speed.
+
+| Metric (10-repeat gate, wikitext window) | RTN experts (v2) | **GPTQ experts** |
+|---|---|---|
+| Mean KLD vs BF16 | 0.0976 | **0.0803 (−17.7%)** |
+| Median per-token KLD | 0.00123 | **0.00083 (−33%)** |
+| Top-1 agreement with BF16 | 0.9325 | **0.9391** |
+| Decode speed (4× DGX Spark, TP4+DCP4, MTP) | 24.4 tok/s | 24.4 tok/s |
+| Reasoning-length inflation vs BF16 service | +63% | **+56%** |
+| Per-expert output error (mean of 19,200) | 0.181 | **0.146 (−19.6%)** |
+| Experts regressed vs RTN | — | **0 / 19,200** |
+
+## What's in this repo
+
+- `L{3..77}.safetensors` — 75 MoE layers × 256 routed experts each, HF per-expert layout:
+  `model.layers.{L}.mlp.experts.{E}.{gate,up,down}_proj.weight` (uint8 nibble-packed
+  MXFP4, low nibble first) + `.weight_scale` (uint8 e8m0, bias 127, block-32 along input dim).
+  Byte-compatible with the AMD-Quark MXFP4 layout the base hybrid uses.
+- `L{3..77}_metrics.json` — per-expert held-out validation (RTN vs GPTQ relative output
+  error, token counts, mode).
+- `run-logs/` — full quantization run logs.
+
+These are **routed experts only** (layers 3–77). Attention, dense MLPs, shared experts,
+router, MTP draft and tokenizer live in the base hybrid repo.
+
+## Method
+
+Per expert (256 per layer, 19,200 total):
+
+1. **Calibration data**: 117 windows × 2048 tokens (code-heavy + prose) captured from a
+   serving GLM-5.2 hybrid with an MoE-forward hook recording each layer's true input
+   hidden states and top-8 routing — so every expert is calibrated on **the tokens it
+   actually serves** (~1K–20K tokens/expert, traffic-weighted by construction).
+2. **GPTQ** with act-order and error feedback onto the MXFP4 grid; scales fixed
+   amax/e8m0 (AMD-compatible). `down_proj` is calibrated against the **quantized**
+   gate/up intermediate (within-expert error propagation).
+3. **Heavy damping (`damp=1.0` of mean diagonal)** — the key hyperparameter finding:
+   per-expert Hessians are rank-deficient (tokens < 6144 dims; rank ≈ token count), and
+   the textbook 1% damping makes GPTQ *worse* than RTN (−13%) by chasing null-space
+   compensation that MXFP4 clipping destroys. At damp=1.0 the sweet spot is wide (0.1–10).
+4. **Validation**: 10% held-out routed tokens per expert; every expert ≥ RTN (worst
+   single expert: +0.7%).
+
+Improvement is depth-graded — early layers gain most (up to −57.9% error at L6), which
+compounds through the network — and traffic-graded (high-traffic experts −14.4% vs
+−8.5% for rare ones, since hot experts get the most calibration tokens automatically).
+
+Run on 4× rented H200s in ~3.5 h (~$100): layer-streamed BF16 fetches (19.3 GB in flight,
+never the full model), one GPU per layer, 240K-token capture uploaded from the serving
+cluster.
+
+## Using these experts
+
+With the base hybrid's sharded vLLM deployment, splice per-rank slices into the
+`sharded_state` files (`w13_weight` = per-rank rows of [gate;up] packed, `w2_weight` =
+per-rank input-column bytes of down — row/byte slicing never crosses the nibble packing;
+see the surgery scripts referenced in the base repo). For HF-layout loaders, these tensors
+directly replace the corresponding expert entries in the base repo's index.
+
+## Per-layer results
+
+<details><summary>75-layer table (mean over 256 experts each)</summary>
+
+| Layer | RTN err | GPTQ err | Improvement |
+|---|---|---|---|
+| 3 | 0.1398 | 0.0595 | 57.9% |
+| 4 | 0.1370 | 0.0733 | 47.1% |
+| 5 | 0.1325 | 0.0726 | 45.9% |
+| 6 | 0.1400 | 0.0835 | 41.0% |
+| 7 | 0.1507 | 0.0967 | 36.3% |
+| 8 | 0.1592 | 0.1037 | 35.0% |
+| 9 | 0.1619 | 0.1103 | 31.8% |
+| 10 | 0.1583 | 0.1104 | 30.6% |
+| 11 | 0.1572 | 0.1103 | 30.2% |
+| 12 | 0.1592 | 0.1126 | 29.6% |
+| 13 | 0.1661 | 0.1216 | 26.9% |
+| 14 | 0.1665 | 0.1246 | 25.3% |
+| 15 | 0.1699 | 0.1316 | 22.7% |
+| 16 | 0.1693 | 0.1316 | 22.4% |
+| 17 | 0.1720 | 0.1331 | 22.8% |
+| 18 | 0.1757 | 0.1394 | 20.7% |
+| 19 | 0.1787 | 0.1433 | 19.8% |
+| 20 | 0.1781 | 0.1424 | 20.2% |
+| 21 | 0.1782 | 0.1415 | 20.7% |
+| 22 | 0.1761 | 0.1391 | 21.2% |
+| 23 | 0.1777 | 0.1342 | 24.5% |
+| 24 | 0.1782 | 0.1378 | 22.7% |
+| 25 | 0.1772 | 0.1405 | 20.7% |
+| 26 | 0.1779 | 0.1427 | 19.8% |
+| 27 | 0.1764 | 0.1439 | 18.5% |
+| 28 | 0.1754 | 0.1450 | 17.4% |
+| 29 | 0.1739 | 0.1452 | 16.6% |
+| 30 | 0.1732 | 0.1462 | 15.8% |
+| 31 | 0.1748 | 0.1480 | 15.4% |
+| 32 | 0.1756 | 0.1498 | 14.8% |
+| 33 | 0.1778 | 0.1520 | 14.6% |
+| 34 | 0.1778 | 0.1537 | 13.6% |
+| 35 | 0.1790 | 0.1564 | 12.7% |
+| 36 | 0.1800 | 0.1575 | 12.6% |
+| 37 | 0.1811 | 0.1599 | 11.8% |
+| 38 | 0.1810 | 0.1604 | 11.5% |
+| 39 | 0.1821 | 0.1614 | 11.5% |
+| 40 | 0.1816 | 0.1590 | 12.5% |
+| 41 | 0.1822 | 0.1590 | 12.8% |
+| 42 | 0.1824 | 0.1607 | 12.0% |
+| 43 | 0.1830 | 0.1608 | 12.2% |
+| 44 | 0.1848 | 0.1636 | 11.6% |
+| 45 | 0.1851 | 0.1643 | 11.3% |
+| 46 | 0.1854 | 0.1643 | 11.5% |
+| 47 | 0.1853 | 0.1633 | 12.0% |
+| 48 | 0.1862 | 0.1632 | 12.4% |
+| 49 | 0.1842 | 0.1594 | 13.6% |
+| 50 | 0.1864 | 0.1607 | 13.9% |
+| 51 | 0.1906 | 0.1647 | 13.7% |
+| 52 | 0.1909 | 0.1641 | 14.1% |
+| 53 | 0.1910 | 0.1640 | 14.2% |
+| 54 | 0.1913 | 0.1624 | 15.2% |
+| 55 | 0.1911 | 0.1605 | 16.1% |
+| 56 | 0.1928 | 0.1624 | 15.8% |
+| 57 | 0.1939 | 0.1647 | 15.1% |
+| 58 | 0.1936 | 0.1641 | 15.3% |
+| 59 | 0.1928 | 0.1639 | 15.0% |
+| 60 | 0.1939 | 0.1641 | 15.4% |
+| 61 | 0.1932 | 0.1622 | 16.1% |
+| 62 | 0.1941 | 0.1642 | 15.4% |
+| 63 | 0.1944 | 0.1657 | 14.8% |
+| 64 | 0.1947 | 0.1660 | 14.8% |
+| 65 | 0.1949 | 0.1664 | 14.6% |
+| 66 | 0.1954 | 0.1667 | 14.7% |
+| 67 | 0.1951 | 0.1669 | 14.5% |
+| 68 | 0.1945 | 0.1652 | 15.1% |
+| 69 | 0.1924 | 0.1614 | 16.2% |
+| 70 | 0.1917 | 0.1609 | 16.1% |
+| 71 | 0.1922 | 0.1608 | 16.4% |
+| 72 | 0.1895 | 0.1546 | 18.5% |
+| 73 | 0.1881 | 0.1519 | 19.3% |
+| 74 | 0.1888 | 0.1505 | 20.4% |
+| 75 | 0.1903 | 0.1492 | 21.6% |
+| 76 | 0.1908 | 0.1472 | 22.9% |
+| 77 | 0.1856 | 0.1318 | 29.2% |
+
+</details>
+
+## Provenance & credits
+
+BF16 source weights: [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) (MIT).
+Baseline RTN MXFP4 experts: AMD (Quark) via
+[festr2/GLM-5.2-BF16-AMDMXFP4experts](https://huggingface.co/festr2/GLM-5.2-BF16-AMDMXFP4experts).
+Calibration capture, GPTQ run, validation and packaging by this repo's author, on
+4× NVIDIA DGX Spark (GB10) + rented H200s. Credits to the b12x community
+(Koush, David Young, Dooner, Festr, Luke and others).
+
+MIT, as inherited from all sources.