Evans
Running models on hardware that shouldn't run them. Writing down what it costs.
I’m Evans. I run language models locally on a dual-Xeon box with two RTX 3090s and 192 GB of DDR4. Sometimes that means a mixture-of-experts model far too big for 48 GB of VRAM, with its expert weights out in host memory and the interesting work happening in the gap. Sometimes it means a dense model that fits on the cards and the question is how fast it can go.
What I publish is measurements: what a change did to decode and prefill, what broke, and what it took to find out. The code I’ve sent upstream to llama.cpp came out of the same runs.
Prefill is a factor of four better and still spends most of its time moving the same weights more than once.
After part 4 an 8k prompt still moves 278 GB to the first GPU, 3.6 times the expert set, and that is 72% of the time to first token. A patch that shares one staged copy across the consumers in a batch is written and reviewed, not built. The bar: at least 1.5x on prefill, decode within 3%, the same quality screen.
Radix-select top-k in the CUB fallback for wide rows, the change behind part 3. Waiting on workflow approval. #28223 (host buffer overrides under mmap) is closed for now under the one-open-PR rule for new contributors and comes back after this one.
Qwen3.8-Flash-Next on 2x3090 + DDR4
A 177B mixture-of-experts model that does not fit in 48 GB of VRAM. Expert weights live in host memory.
- Part 04
The expert cache moves out of the way while the prompt runs
Releasing the expert cache and the decode buffers for the length of a prompt lets prefill run at a 2048-token micro-batch: 2.2x faster at 8k and 2.4 to 2.5x at 37k to 119k, with decode unchanged and a 240-pair quality screen finding no consistent regression.
- Part 03
A top-k fallback that was sorting far more than it needed to
Long-context decode was 9 to 12% faster once the sparse-attention indexer stopped fully sorting each row, and a 480-request quality screen found no consistent regression.
- Part 02
25-29 to 37-41 t/s with UD-Q4_K_XL and MTP
A quant swap, MTP on top of the cache, a 4x faster load, a bug in the cache PR, and the discovery that my RAM had been thermal throttling the whole time.
- Part 01
17 to 25-29 t/s decode with the expert cache PR
All 48 expert layers in host RAM, an LRU cache of hot experts in VRAM, and the VRAM budget that made it pay.
llama.cpp
Changes I have sent upstream, and the measurements behind them.
- Note
A silent -ot downgrade under mmap, and a 9-line loader fix
Why -ot ...=CUDA_Host did nothing on master, what my PR #28223 changes, and where the 8 minutes of load time went.
Can query-selected KV paging fit a 262k context on a single 24GB GPU? Measured on Qwen3.6-27B: no. Even a perfect oracle selector falls far short of the attention mass needed. Harness plus the negative result, so nobody else has to run it.
Downloads that finish. A supervisor around aria2c for huge model files. Survives stalls, expired links, and reboots without losing progress. SHA256-verified. Moved about 200GB with it.
Self-hosted personal organiser you talk to. Whisper, a local LLM, SQLite, FastAPI, and a React PWA. Nothing leaves your machine. Runs end to end on 12GB of VRAM.
