julianmb/Qwen-3.8-27B-ROCmFP4-FAST-GGUF

Qwen 3.8 27B ROCmFPX (GGUF for AMD Radeon GPUs)

This repository contains optimized ROCmFP4 and ROCmFP8 GGUF releases of Qwen 3.8 27B, custom-engineered for AMD Strix Halo (Ryzen AI Max+ 395 / Radeon 8060S) and AMD Radeon discrete GPUs.

🛠️ Unified HaloFPX Server: github.com/julianmb/halofpx (Multi-model zoo, hot-swapping, OpenAI API)
📖 Qwen 3.8 Dedicated Deep-Dive: github.com/julianmb/q38rocm (Standalone optimization report, benchmarks & tuning)

---

📦 Available GGUF Quantizations

File Name	Precision	File Size	Raw Decode	MTP Speculative Decode	Best For
Qwen3.8-27B-ROCmFP4-FAST.gguf	4.26 bpw	13.55 GiB	14.02 tok/s	🔥 30.56 – 36.04 tok/s	Maximum Speed (Gold Standard, Fits 16GB+ GPUs)
Qwen3.8-27B-ROCmFP8.gguf	8.25 bpw	26.25 GiB	7.66 tok/s	18.96 tok/s	Maximum Precision (<0.003 PPL loss vs FP16)
Qwen3.8-27B-ROCmFP4-STRIX_LEAN.gguf	4.34 bpw	13.82 GiB	13.90 tok/s	30.00 – 34.00 tok/s	FP16 embeddings & norm preservation
Qwen3.8-27B-Q3_K_S.gguf	3.59 bpw	11.40 GiB	16.69 tok/s	20.44 – 26.11 tok/s	Fastest unassisted decode
Qwen3.8-27B-Q3_K_M.gguf	3.95 bpw	12.56 GiB	15.15 tok/s	25.00 – 28.50 tok/s	Balanced 3-bit medium quantization
Qwen3.8-27B-ROCmFP2.gguf	2.69 bpw	8.56 GiB	12.82 tok/s	17.50 – 19.00 tok/s	Ultra-compact memory footprint

---

🔬 Why ROCmFP4 & ROCmFP8 Improve Performance on AMD Hardware

In auto-regressive decoding, generation speed is physically bound by memory bandwidth:

• Bandwidth Math: Strix Halo's 256-bit memory controller delivers 190–200 GB/s sustained read bandwidth. Loading an unquantized FP16 model (54.6 GB) caps generation at **4.9 tok/s**.
• 8-Bit (ROCmFP8 - 26.25 GB): Doubles memory streaming throughput to 18.96 tok/s with MTP speculation while providing lossless precision (<0.003 PPL loss vs FP16).
• 4-Bit (ROCmFP4 - 13.55 GB): Slashes memory payload by 75.2% vs FP16, raising unassisted streaming decode to 14.02 tok/s and MTP speculative throughput to 30.56 – 36.04 tok/s (with 33.80 tok/s sustained decode at K=4) with ~99% benchmark retention.

---

🧠 Optional: AMD XDNA 2 NPU Acceleration (Qwen 3.8 27B Only)

⚠️ Scope note: All NPU benchmarks below were only tested on Qwen 3.8 27B (dense, ROCmFP4_FAST).

Strix Halo features a 50 TOPS XDNA 2 NPU (/dev/accel/accel0). Extensive empirical benchmarking demonstrates:

• Sustained Decode Ceiling: 33.8 tok/s via embedded MTP (iGPU only) is the practical ceiling. The NPU does not improve sustained decode speed.
• Proven NPU Value: 1.8× faster first-token TTFT on long prompts (870 ms vs 1587 ms) via hybrid NPU-burst → iGPU handoff (scripts/run_pipeline.py).
• See docs/NPU_INTEGRATION.md for complete setup instructions.

---

📊 Performance Benchmarks (AMD Ryzen AI Max+ 395)

Measured on AMD Ryzen AI Max+ 395 (40 CU Radeon 8060S @ 2.9 GHz, 128 GB 256-bit LPDDR5X, Linux 7.0, Mesa 26.0 RADV):

Optimization Profile	Model Size	Unassisted Decode (Measured)	MTP Speculative Decode (Measured)	Speedup vs Baseline	TTFT (Prompt Eval) (Measured)
Stock Q4_K_M (Baseline)	15.92 GiB	12.27 tok/s	N/A	1.00×	526.7 ms
ROCmFP4_FAST (This Model)	13.55 GiB	14.02 tok/s	N/A	1.14×	468.3 ms
ROCmFP4_FAST + Strict Greedy MTP	13.55 GiB	14.02 tok/s	34.82 tok/s	2.84×	442.8 ms
ROCmFP4_FAST + Sustained MTP (K=4)	13.55 GiB	14.02 tok/s	33.80 tok/s sustained	2.40×	510.0 ms
ROCmFP4_FAST + MTP (n6/p0.60)	13.55 GiB	14.02 tok/s	30.56 – 34.82 tok/s	2.50× – 2.84×	439.4 ms
ROCmFP4_FAST + Deep Spec (n7/p0.35)	13.55 GiB	14.02 tok/s	🔥 36.04 tok/s (JSON/Code)	🔥 2.94×	445.8 ms

---

💾 Context Scaling & Memory Footprint

Using Asymmetric TurboQuant KV cache (-ctk q8_0 -ctv turbo4):

Context Window	Model Weights	TurboQuant KV Cache	Total RAM Footprint
8K tokens	13.55 GiB	0.62 GiB	14.17 GiB (Fits 16GB VRAM)
16K tokens	13.55 GiB	1.23 GiB	14.78 GiB (Fits 16GB VRAM)
32K tokens	13.55 GiB	2.45 GiB	16.00 GiB (Ideal for 64GB Strix Halo)
64K tokens	13.55 GiB	4.90 GiB	18.45 GiB
128K tokens	13.55 GiB	9.80 GiB	23.35 GiB
262K tokens (Full)	13.55 GiB	20.08 GiB	33.63 GiB

---

📥 Quick Download

# Download ROCmFP4_FAST (13.55 GiB)
hf download julianmb/Qwen-3.8-27B-ROCmFP4-FAST-GGUF Qwen3.8-27B-ROCmFP4-FAST.gguf --local-dir .

# Or download ROCmFP8 (Zero-loss 8-bit precision, 26.25 GiB)
hf download julianmb/Qwen-3.8-27B-ROCmFP4-FAST-GGUF Qwen3.8-27B-ROCmFP8.gguf --local-dir .

---

🚀 How to Run

Option 1: Via HaloFPX (Recommended Multi-Model Server)

git clone https://github.com/julianmb/halofpx.git
cd halofpx
pip install -e .

halofpx load qwen38-27b

Option 2: Standalone via q38rocm

git clone https://github.com/julianmb/q38rocm.git
cd q38rocm

./quickstart.sh

---

🔒 Limitations & Safety

• Custom Backend: Requires the ROCmFPX toolchain (pinned build: e87d53e (213)).
• Hardware Target: Optimized for AMD Strix Halo (gfx1151) and AMD discrete Radeon GPUs.
• Base Alignment: Inherits base safety characteristics and knowledge capabilities of Qwen 3.8 27B.

---

📜 License & Attribution

• Base Model: Qwen 3.8 27B by Alibaba Cloud
• Quantization & Optimizations: Apache 2.0 License.
• Community Research: Multi-slot concurrency metrics referenced from MrWidmoreHK and kujetic.

📏 Quantization Fidelity (wikitext-2 validation)

Independent perplexity checks (llama-perplexity, ROCm0, ctx 512):

Build	PPL ↓
ROCmFP4_FAST (this repo, original build)	7.205 ± 0.121
ROCmFP4_FAST requantized from unsloth Q8_0	7.138 ± 0.120

Speed is statistically identical between both builds (13.56 vs 13.64 tg128) — decode is bandwidth-bound. Requantizing any high-precision source (Q8_0/BF16, --allow-requantize) into ROCmFP4 presets is a validated recipe; avoid k-quant sources (double quantization).

🔬 Batch Scaling (why speculative decoding works here)

llama-batched-bench, tg128 per-sequence throughput, Vulkan0:

Verify batch	tok/s/seq
1	13.53
8	55.44 (4.1×)

Speculative decoding converts memory-bandwidth headroom into tokens: the flat batch-cost curve is what makes MTP's ~2.5× sustained gain possible.