| --- |
| language: en |
| license: mit |
| library_name: mlx |
| base_model: |
| - Qwen/Qwen3.8-27B |
| - z-lab/Qwen3.8-27B-DFlash2 |
| tags: |
| - qwen3.8 |
| - dflash2 |
| - speculative-decoding |
| - block-diffusion |
| - apple-silicon |
| - mlx |
| - 4-bit |
| - long-context |
| - agentic |
| - omlx |
| - benchmark |
| inference: false |
| --- |
| |
| # LocalFlash: Qwen3.8-27B + DFlash2 serving configuration for Apple Silicon |
|
|
| This repository documents a **measured, reproducible deployment recipe** (not new weights): |
| Qwen3.8-27B at MLX 4-bit, accelerated by the **DFlash 2** block-diffusion drafter under the |
| z-lab **oMLX fork**, tuned for long-context coding-agent workloads on an M4 Max / 64 GB. |
|
|
| ## Measured results (M4 Max, 64 GB, macOS 27.0) |
|
|
| | Metric | llama.cpp baseline | This configuration | |
| |---|---|---| |
| | Decode (median) | ~15 tok/s | **809 tok/s** | |
| | TTFT @ 32.5k fresh prompt | 435 s every turn | 486 s once | |
| | TTFT @ cached prefix turn | — | **8–16 s** | |
| | Context window | 32k | 262k native | |
| | Needle recall @ 70k | — | 5/5 ordered | |
|
|
| Raw measurement records: [GitHub repo](https://github.com/chengyixu/qwen38-dflash2-bench) → `results/raw/`. |
|
|
| ## Files |
|
|
| No weight files are hosted here — use the upstream artifacts: |
|
|
| - Target: [`mlx-community/Qwen3.8-27B-4bit`](https://huggingface.co/mlx-community/Qwen3.8-27B-4bit) |
| - Drafter: [`z-lab/Qwen3.8-27B-DFlash2`](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2) |
| - Server: [`z-lab/omlx-fork` release `0.6.2-dflash2`](https://github.com/z-lab/omlx-fork/releases) |
|
|
| ## Engine configuration |
|
|
| Place as `~/.omlx/model_settings.json`: |
|
|
| ```json |
| { |
| "version": 1, |
| "models": { |
| "Qwen3.8-27B-4bit": { |
| "dflash_enabled": true, |
| "dflash_draft_model": "/Users/<you>/Models/mlx/Qwen3.8-27B-DFlash2", |
| "dflash_draft_quant_enabled": true, |
| "dflash_draft_quant_weight_bits": 4, |
| "dflash_draft_quant_activation_bits": 16, |
| "dflash_draft_quant_group_size": 64, |
| "dflash_block_size": 5, |
| "dflash_verify_mode": null, |
| "dflash_in_memory_cache": true, |
| "display_name": "Qwen3.8-27B 4bit + DFlash2" |
| } |
| } |
| } |
| ``` |
|
|
| Notes: block size ≤5 per z-lab guidance for quantized targets; adaptive verify; |
| L1 prefix cache is what turns multi-turn agent sessions from minutes-per-turn |
| into seconds-per-turn. |
|
|
| ## Integrity requirement |
|
|
| Hash-verify both models against the hub manifests before first serve. Two of our |
| three shards arrived corrupt after resumed downloads — with valid safetensors |
| headers and correct sizes. The model loads and then emits fluent gibberish. |
| `bench/hf_fetch.py` in the GitHub repo wraps download + verification. |
|
|
| ## Sampling (per Qwen3.8 model card) |
|
|
| Thinking mode: `temperature=1.0, top_p=0.95, top_k=20`; reasoning effort |
| xhigh/medium/low via the server's chat-template kwargs. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{localflash2026, |
| title={Losing the Draft, Keeping the Speed}, |
| author={chengyixu}, |
| year={2026}, |
| url={https://github.com/chengyixu/qwen38-dflash2-bench} |
| } |
| ``` |
|
|
