Qwen

/

Qwen3.8-Flash-Next

Like 5.39k

Follow

Qwen 105k

I gonna ask question I have 32gb ram 16gb vram and fast nvme ssd can i run this model decent speed.

yes, you will need to push -ngl 48 and try with -ncmoe with 40s and lower quant model of 1/2 bit... if total file size is great then 48 gb as it is in my case you need for now --mmap too. TPS is generally good in 20s... prompt is bit slow in 40s-80s for me ... few folks are already working to ensure that like ngl if we can push experts onto ram, leaving ngram for mmap ... that will boost up prompt prefill over 400 ...

Wow this is an outright killer model... anyone can now run this with mere 32 gb ram & a fast nvme drive ... darn cheap

I’d like to ask: following the author's description, how do I run this using vLLM via Docker? Could you provide the specific parameters?

The -ngl, -ncmoe, and --mmap settings above are llama.cpp/GGUF options; they are not vLLM Docker parameters, so the 48 GB result should not be transferred to vLLM as-is. With vLLM, the answer depends on an actually supported quantized checkpoint, its loader/kernel support, context length, and the runtime’s CPU/GPU offload behavior.

I maintain FitLLM. We pinned the official checkpoint metadata and deliberately report this model as numeric-null until those runtime/allocation details can be verified:
https://fitllm.run/model-status/qwen3-8-flash-next

If someone shares the exact quant repo, runtime + commit, launch command, context, peak RSS/VRAM, and tok/s, I’ll turn it into a reproducible hardware profile instead of guessing.

You can try this, with 48GB your decode tok/s will be in the range of 10-15:

https://github.com/Pummelchen/NVMAI

Thanks — I checked NVMAI. It is an Apple-Silicon-specific SSD/NVMe expert-streaming runtime with its own model format, not a standard 48 GB RAM/VRAM-resident load. Its README currently reports 5.40 tok/s for Qwen3.8-Flash-Next 4-bit on a 24 GB M3 and explicitly notes that model state, KV cache, and runtime scratch use memory beyond the configured expert-cache budget. I therefore won't use the 10–15 tok/s claim as a generic 48 GB FitLLM profile without the exact machine, NVMAI commit, quantization, RAM budget, context, peak RSS/VRAM, and raw benchmark. I'll keep this as a separate experimental storage-streaming path, not normal FitLLM hardware guidance.