ChengyiX/Qwen3.8-27B-DFlash2-LocalFlash-M4Max

LocalFlash: Qwen3.8-27B + DFlash2 serving configuration for Apple Silicon

This repository documents a measured, reproducible deployment recipe (not new weights): Qwen3.8-27B at MLX 4-bit, accelerated by the DFlash 2 block-diffusion drafter under the z-lab oMLX fork, tuned for long-context coding-agent workloads on an M4 Max / 64 GB.

Measured results (M4 Max, 64 GB, macOS 27.0)

Metric	llama.cpp baseline	This configuration
Decode (median)	~15 tok/s	809 tok/s
TTFT @ 32.5k fresh prompt	435 s every turn	486 s once
TTFT @ cached prefix turn	—	8–16 s
Context window	32k	262k native
Needle recall @ 70k	—	5/5 ordered

Raw measurement records: GitHub repo → results/raw/.

Files

No weight files are hosted here — use the upstream artifacts:

• Target: mlx-community/Qwen3.8-27B-4bit
• Drafter: z-lab/Qwen3.8-27B-DFlash2
• Server: z-lab/omlx-fork release 0.6.2-dflash2

Engine configuration

Place as ~/.omlx/model_settings.json:

{
  "version": 1,
  "models": {
    "Qwen3.8-27B-4bit": {
      "dflash_enabled": true,
      "dflash_draft_model": "/Users/<you>/Models/mlx/Qwen3.8-27B-DFlash2",
      "dflash_draft_quant_enabled": true,
      "dflash_draft_quant_weight_bits": 4,
      "dflash_draft_quant_activation_bits": 16,
      "dflash_draft_quant_group_size": 64,
      "dflash_block_size": 5,
      "dflash_verify_mode": null,
      "dflash_in_memory_cache": true,
      "display_name": "Qwen3.8-27B 4bit + DFlash2"
    }
  }
}

Notes: block size ≤5 per z-lab guidance for quantized targets; adaptive verify; L1 prefix cache is what turns multi-turn agent sessions from minutes-per-turn into seconds-per-turn.

Integrity requirement

Hash-verify both models against the hub manifests before first serve. Two of our three shards arrived corrupt after resumed downloads — with valid safetensors headers and correct sizes. The model loads and then emits fluent gibberish. bench/hf_fetch.py in the GitHub repo wraps download + verification.

Sampling (per Qwen3.8 model card)

Thinking mode: temperature=1.0, top_p=0.95, top_k=20; reasoning effort xhigh/medium/low via the server's chat-template kwargs.

Citation

@misc{localflash2026,
  title={Losing the Draft, Keeping the Speed},
  author={chengyixu},
  year={2026},
  url={https://github.com/chengyixu/qwen38-dflash2-bench}
}