⚡ TL;DR
Two flags tripled decode speed for a 220B-class MoE on one 24GB GPU: a GPU-resident expert cache and --load-mode none.
- Decode: 23.0 t/s → 65.6 t/s (2.85×)
- Prefill: 355 t/s → 762–826 t/s (2.2×) on real text
- The win is not the cache alone. The cache without fixing residency was slower (30.7 t/s) — mmap was keeping only half the model in RAM
- Bonus: n-gram speculative decoding added +21% on prose and +45–50% on code, at zero VRAM cost
- And a warning: our model file had been silently corrupt on disk for weeks. Cached reads lied. Verify with
iflag=directagainst the publisher's hash
The Setup
This is about serving a large mixture-of-experts model on consumer hardware — a single RTX 4090 (24 GB), a Ryzen 7 5700X (8 cores / 16 threads), and 94 GB of DDR4. The model is Qwen3.8-Flash-Next-Uncensored, a ~220B-parameter MoE with roughly 6B parameters active per token, in two quantizations:
- IQ3_XXS — 85.2 GB across 2 shards (~20 t/s decode when we started)
- AD-4.27 — a mixed 4.27 bpw recipe, 94.5 GB across 33 shards
Both are well beyond VRAM, so expert weights are offloaded to system RAM and streamed per token. That single fact governs everything that follows.
Why MoE Decode Is a Bandwidth Problem
A dense 220B model would be hopeless on this box. MoE makes it possible because each token only activates a small slice of the network — but that slice still has to arrive from memory, every single token.
The bandwidth budget is the whole story:
RTX 4090: ~1000 GB/s (GDDR6X)
DDR4 dual-channel: ~45-50 GB/s (about 20x slower)
Practical consequence: every expert that has to come from system RAM caps your tokens per second
So decode speed is not a compute question. It is a question of how few bytes per token have to cross that DDR4 link. Everything that helped here reduced that number: either by keeping hot experts on the GPU, or by making sure the host side never re-read from disk.
The Numbers
All figures below are from clean-room runs on the same box: real text (Wikipedia) for prefill, 1024 forced-token generations for decode, page cache evicted before each load, and every leg checked for coherent output. VRAM is with a 65k context.
| Configuration | Decode | Prefill (7.7k / 31k) | VRAM | System RAM |
|---|---|---|---|---|
| ncmoe 38, no cache, mmap | 23.8 t/s | 142 / 175 t/s | 15.5 GB | partial |
ncmoe 38, no cache, --load-mode none |
23.0 t/s | 377 / 355 t/s | 15.5 GB | ~66 GB |
| ncmoe 48 + expert cache 128, mmap | 30.7 t/s | 115 / 143 t/s | 17.7 GB | ~41 GB |
ncmoe 48 + expert cache 128, --load-mode none |
65.6 t/s | 826 / 762 t/s | 17.7 GB | ~79 GB |
Output at the top configuration was verified coherent: 8 of 8 generations passed a repetition/degeneracy check and all three correctness probes answered right. Note what that does and does not cover — with --load-mode none the weights live in anonymous memory, so hashing the file on disk tells you the file is good, not the copy in RAM. Known-answer probes are the practical check for a no-mmap load; the page-cache-vs-O_DIRECT hash trick only compares like-for-like when the model is mmap'd.
Depth matters, and the table does not show it. The 65.6 t/s decode figure is a shallow-context measurement. With a 75k-token context loaded, decode at depth falls to about 20 t/s — attention has to read the whole quantised KV window every token — while prefill on the same request still ran at 394 t/s over a 300k-character prompt, and long-context recall stayed correct. If your workload is agentic with long contexts, plan around the lower number and treat the cache as the thing that keeps the expert half of the cost off DDR4.
Why the Expert Cache Works
The MoE expert cache (currently landing upstream as llama.cpp PR #27861) keeps a fixed number of hot experts per layer, on the GPU. We gave it 128 slots, which costs about 12.9 GB of VRAM.
The reason such a small slice helps so much is routing skew. Expert selection is extremely uneven — a small set of experts absorbs a large share of the traffic. Caching the hot set converts most expert matmuls from DDR4 streaming into GPU-resident reads.
Compare that to the older approach, --n-cpu-moe, which simply keeps the experts of the first N layers on the CPU:
ncmoe 38 (10 layers of experts on GPU):
helps 10 of 48 layers = ~21% of expert work, at the cost of 18 GB VRAM
ncmoe 48 + cache 128 (all experts on host, hot set cached):
helps all 48 layers, at the cost of 12.9 GB VRAM
Coverage across every layer beats owning a few layers outright. That is the whole trick.
The Trap That Cost Us a Day: mmap Residency
Here is the part worth repeating. The expert cache alone made things worse than we expected — 30.7 t/s instead of something like 60. And prefill collapsed to a third of its previous rate. For a while we believed the cache was stealing memory bandwidth from prefill.
It wasn't the cache. It was memory mapping.
With llama.cpp's default mmap loading, model weights live in the kernel page cache. Our models are 85–94 GB and the machine has 94 GB of RAM, so the kernel cannot keep the whole working set resident. It evicts expert pages, the model immediately faults them back in, and the result is disk thrash in the middle of generation. Measuring the process told the story immediately:
AD-4.27, ncmoe 38, mmap: RSS 43 GB of a 94.5 GB model
Same config, --load-mode none: RSS 75 GB (the full host working set)
Prefill: 46/69 t/s (mmap) → 384/369 t/s (--load-mode none)
--load-mode none reads the weights into ordinary anonymous RAM instead of page-cache-backed mappings, so nothing the model needs gets evicted underneath it. It costs about 60 extra seconds at load time. It is, by a wide margin, the single highest-value flag we found.
This also explains a whole day of confusing numbers. Our prefill measurements had ranged from 60 to 1,660 t/s on nominally identical configurations. That was never a property of the model or the flags — it was the page cache, and whether the model happened to be resident at that moment.
Wait. Was the Model Corrupt?
Partway through, output went strange: a model that had been answering normally started emitting an endless string of a single digit. We assumed we had broken something with the expert cache. We had not.
The actual discovery was worse and more interesting: our model file on disk was corrupt.
The tell was a hash mismatch against the publisher's manifest. Two shards, identical byte lengths to the originals, different SHA256 sums:
| Shard | Size | Local SHA256 | Upstream SHA256 |
|---|---|---|---|
| IQ3 shard 1 | 44,637,691,008 B | 6eac4f45… |
aaf57046… |
| IQ3 shard 2 | 40,564,977,024 B | d65e862d… |
a19cf9bb… |
Same length, different content. That is not a stale download or a different upload — that is corruption in place. It had been sitting there for weeks, serving subtly wrong weights to every session, and the model was coherent enough most of the time that nothing looked obviously broken.
It also explains the pathology that had been blamed on the expert cache: on this machine, a page-cache read of a verified file hashed differently from an O_DIRECT read of the same file, in the same run, three separate times. The resident copy of the weights was being corrupted after load. Bad RAM does that. Fixing the model file did not fix the cause; it removed one of its victims.
Verify your own weights before you tune anything
A cached read can lie; a direct read cannot (though it can still be corrupted in flight by bad RAM, which is how the original download was damaged). Check the file itself, then check it again while it is loaded:
# 1. against the publisher's hash
dd if=model-00001-of-00002.gguf iflag=direct bs=64M | sha256sum
# 2. the embedded copy, with the model running (catches post-load corruption)
sha256sum model-00001-of-00002.gguf # page-cache read
dd if=model-00001-of-00002.gguf iflag=direct bs=64M | sha256sum # same file, direct
# if those two disagree, your resident weights are not what you think they are
Three Things That Turned Out to Be Wrong
1. "The expert cache halves prefill"
Our own earlier measurements said so. They were an artifact: the cache made VRAM pressure worse, which changed residency under mmap. With --load-mode none, the cache configuration has the best prefill we measured (826 t/s). No trade-off exists.
2. "1,660 t/s prefill"
Our initial prefill test used one sentence repeated 600 times. Repetitive prompts are not a benchmark. On real text the same configuration does 287 t/s. If your test prompt would look strange pasted into a document, the number is fiction.
3. "The expert cache produces garbage output"
An upstream bug report describes degenerate output with this cache on this model family. We reproduced something that looked exactly like it, filed it mentally as a cache bug, and were wrong: 32 consecutive cache-enabled generations came out coherent, and the one bad run traced to the RAM corruption described above. We are not saying the upstream report is mistaken — only that on our box the cause was hardware, and we would have shipped a wrong conclusion had we not kept checking.
Measuring Honestly on a Flaky Machine
Most of the engineering effort here was not in finding speed, it was in trusting the numbers. If you benchmark on hardware that occasionally flips a bit, you need rules:
- Evict the model from the page cache before each measured load. Otherwise you are measuring the kernel's mood.
- Check output coherence on every leg, not just speed. Degenerate output reuses the same experts, which inflates every throughput number you collect. A fast number from broken output is worse than no number.
- Hash the resident weights. Reading the loaded file two ways and comparing is the only cheap way to know whether the model you are measuring is the model you downloaded.
- One server at a time, with a RAM gate. Two 80 GB models on a 94 GB machine will silently thrash instead of failing cleanly.
- Keep the raw generations. Every claim in this post can be re-checked against stored output.
Bonus: n-gram Speculative Decoding
Speculative decoding usually means running a small draft model alongside the big one. There is a cheaper variant: propose the next tokens using n-grams already present in the context, then verify them with the model in a single batched pass. Because drafts are verified, output is unchanged — it is a pure throughput trick, and it costs no VRAM.
That last property matters a lot here, because our VRAM is fully committed. On a 6k-character C++ file with a code-continuation prompt:
| Speculative mode | Prose (256 tok) | Code (256 tok) | Code (1024 tok) |
|---|---|---|---|
| none | 20.3 t/s | 18.8 t/s | 18.9 t/s |
ngram-simple |
21.4 t/s | 20.5 t/s | 19.6 t/s |
ngram-mod |
24.5 t/s | 27.2 t/s | 28.3 t/s |
+21% on prose, +45–50% on code. Code benefits most because code repeats itself — the same patterns, the same idioms, the same structures re-emitted. If you are running a coding agent locally, this is close to free money:
llama-server -m model.gguf ... \
--spec-type ngram-mod --spec-ngram-mod-n-min 24
A Note on Quality
Speed tuning is only worth anything if the model still answers correctly. On this model family we measured quality with perplexity on held-out text, hard reasoning questions under a high effort setting, and a long-context needle test. Two findings worth passing on:
- Reasoning budget is a server-side cap. Our test harness was not passing
--reasoning-budget, so a thinking model could spend the entire request on reasoning and get truncated mid-thought. That is what made one model look broken. With the budget set, the same questions answered 3/3 correct with no truncation. If your local model "keeps rambling and never answers", check that flag before blaming the model. - Long context held up. With a 100k-token prompt built from Wikipedia text and three facts planted at different depths, the model retrieved all three, at ~280 t/s prefill at that depth.
Reproduce It
The full recipe, in the order the flags matter:
# the configuration that gave 3x, at 200k context
llama-server -m Qwen3.8-Flash-Next-Uncensored-IQ3_XXS-00001-of-00002.gguf \
-c 204800 -np 1 -ngl 99 \
--n-cpu-moe 48 \
-ctk tbq4_0 -ctv tbq4_0 -t 8 --jinja \
--moe-expert-cache 128 --moe-expert-cache-inserts 2 \
--load-mode none \
--reasoning on --reasoning-effort medium --reasoning-budget 8192
# optional, on top: n-gram speculative decoding
--spec-type ngram-mod --spec-ngram-mod-n-min 24
# drop the model's pages from the page cache first (see "was the model corrupt")
python3 -c "
import glob,os,sys
for p in glob.glob('/path/to/model/*.gguf'):
fd=os.open(p,os.O_RDONLY); os.posix_fadvise(fd,0,0,os.POSIX_FADV_DONTNEED); os.close(fd)"
Budget on this machine: ~21 GB VRAM and ~80 GB of system RAM at 200k context. The RAM requirement is the real constraint — the whole trick only works because all expert weights are host-resident and the hot ones are cached on the GPU. If you have less RAM, drop to --n-cpu-moe 38 and no cache: you keep ~23 t/s decode and ~370 t/s prefill with a much smaller footprint.
What We Would Try Next
- MTP / multi-token prediction. This model family ships a draft head trained for exactly this; the reference numbers from the model's own author sit at 46–55 t/s, and the flag (
--spec-type draft-mtp) is already compiled into our build. Our blocker is a partial draft head that the loader rejects, not the concept. - Fine-grained tensor placement.
--n-cpu-moeis a blunt instrument ("first N layers").--override-tensorlets you place individual tensors, which would let the statistically hottest expert layers live on the GPU instead of whichever ones happen to come first. - Sliding-window attention to fund a bigger cache. Attention is not our bottleneck, so shrinking the KV cache does not speed up decode directly — but every gigabyte it frees on the GPU is another slot or two hundred in the expert cache, and that does.
- Fix the RAM. Everything above is measurable only because we kept re-checking; the underlying bit-flip corruption is a hardware problem, and the honest next step is a memory test rather than another flag.
Sources
- Expert LRU cache — llama.cpp PR #27861 (GPU-resident cache for host-offloaded MoE experts)
- Model — Qwen3.8-Flash-Next-Uncensored (orcarouter) and the AD-4.27 mixed-bpw quant (Navin-Models)
- Runtime — llama.cpp with the qwen4exp port, TBQ4 KV quantization
- Related reading — Q4 KV Cache: Surprising Results, Turbo4 KV Cache Benchmarks