Writeups
Benchmarks, failure notes, and the commands to reproduce them.
Qwen3.8-Flash-Next on 2x3090 + DDR4
A 177B mixture-of-experts model that does not fit in 48 GB of VRAM. Expert weights live in host memory.
- Part 04
The expert cache moves out of the way while the prompt runs
Releasing the expert cache and the decode buffers for the length of a prompt lets prefill run at a 2048-token micro-batch: 2.2x faster at 8k and 2.4 to 2.5x at 37k to 119k, with decode unchanged and a 240-pair quality screen finding no consistent regression.
- Part 03
A top-k fallback that was sorting far more than it needed to
Long-context decode was 9 to 12% faster once the sparse-attention indexer stopped fully sorting each row, and a 480-request quality screen found no consistent regression.
- Part 02
25-29 to 37-41 t/s with UD-Q4_K_XL and MTP
A quant swap, MTP on top of the cache, a 4x faster load, a bug in the cache PR, and the discovery that my RAM had been thermal throttling the whole time.
- Part 01
17 to 25-29 t/s decode with the expert cache PR
All 48 expert layers in host RAM, an LRU cache of hot experts in VRAM, and the VRAM budget that made it pay.
llama.cpp
Changes I have sent upstream, and the measurements behind them.
- Note
A silent -ot downgrade under mmap, and a 9-line loader fix
Why -ot ...=CUDA_Host did nothing on master, what my PR #28223 changes, and where the 8 minutes of load time went.