⚡ The short version
GLM-5.3-Flash is a 320-billion-parameter mixture-of-experts model. Our quantised copy is about 96 GB, more than the 24 GB GPU and 90 GB of RAM on our desk can hold at once. We ported it to our own fork of the Strata engine. The first run decoded at 1.75 tokens/s. Three days later it decodes at 25 t/s and prefills at ~400 t/s, with a 512K-token context.
It runs uncensored through a runtime abliteration LoRA, so the model file is never edited, and it is served over an OpenAI- and Anthropic-compatible API that our coding agent plugs into. The code is open: Indras-Mirror/Strata-GLM.
Why GLM-5.3-Flash
Before starting we compared the three big open "Flash" models. GLM-5.3-Flash led the Artificial Analysis Intelligence Index (42, against DeepSeek V4.1 Flash at 39 and MiMo-V2.6-Flash at 38). It also beat DeepSeek on the coding-agent benchmarks we care about: Terminal Bench 84.3 vs 82.7, DeepSWE 63.4 vs 54.4. People who use it report the cleanest code of the three and fixes aimed at root causes. The usual complaint is that it's slow, and speed is the part an engine can change.
It is also an odd architecture to run. It takes the hyper-connections from DeepSeek V4: four parallel residual streams, re-mixed by a Sinkhorn normalisation in every layer. 34 of its 45 layers use Kimi Delta Attention, a linear attention whose state stays the same size however long the context gets. The other 11 use MLA with a lightning indexer, which picks the 2,048 most relevant tokens to attend to. Each layer has 288 experts and 8 fire per token.
We had already built a DeepSeek-V4 engine on Strata (Niko1221's MoE engine). Its expert tier carried over unchanged, and ggml already had every operation the dense half needed. What we had to write was a new graph builder for the dense half, plus the wiring.
Day one: 1.75 tokens per second
The first real run was coherent and very slow: 1.75 t/s. Only 18.6% of the experts a token needed were in VRAM, so most were computed on the CPU or read off the SSD mid-token. The usual fix is to cache the popular experts, and that's where we found the model's first surprise.
GLM's routing is flat. There's no small set of popular experts. Over 512 tokens the busiest 10% of experts take only 18% of the routes, and a frequency-ranked VRAM cache hits about 10%, which is just its share of the experts. Consecutive tokens share less than one expert in eight beyond chance. So decode speed depends on how much of the model is resident, not on which parts. That changed the plan:
- No duplicates. An expert in VRAM is no longer also kept in RAM. Decode went 1.75 → 3.95 t/s.
- Byte-sized cache slots in place of slots sized for the largest expert: 951 → 1,851 experts in VRAM.
- Chunked prefill. Each layer's experts stream into the GPU once per chunk of up to 4,096 tokens and run as one batched matrix multiply. Prefill went from 1.7 t/s to 80–90, and later ~400.
- Soft REAP pruning. We scored every expert by how much it contributes (REAP saliency) and kept the least useful 25% out of the hot tiers. They're not deleted: if the router really wants one, it still runs. Perplexity stays at the unpruned level.
Uncensoring without touching the weights
No uncensored quantisation of GLM-5.3-Flash fits this machine, so we apply the uncensoring at run time. GCSA AiLab's Abliterix v2 is a rank-1 LoRA that edits attention outputs, the shared expert and the routed experts. Its StrongREJECT refusal rate is 1.7%. Applying it at run time means we can serve the same file stock or uncensored, and that the delta has to be applied on every path an expert can take: VRAM, PCIe, CPU, and the batched prefill kernels.
That last requirement caused the hardest bug of the project. With the LoRA on, a hard refusal test made the model fall into an endless repetition loop instead of answering. The stock model handled the same prompt cleanly. The temptation was to call the adapter weak, but we treated the loop as a bug, and it was one.
One of our speed tricks skips a cache-miss expert when its routing weight is tiny. A skipped expert is never computed, so its LoRA delta is never applied, while the experts around it get theirs. Each token became a mix of ablated and un-ablated experts, and prefill and decode skipped different ones. The model was being pulled in two directions at once. The fix was --renorm-skip: skipped experts are dropped cleanly and the remaining weights renormalised, so the ablation is all-or-nothing again. With the loop gone, the ablated server got 3× faster (5.4 → 16.3 t/s) because it could use the skip path again.
Maya: a better file
Halfway through we changed the model file. peasantsmith's Maya-S-v2 quantises the experts from the FP8 release into importance-weighted 2–3-bit formats and keeps the dense half at Q6_K. It's smaller than the 3.0-bit file we started with, and on the same flags it was faster straight away (decode 17.8 → 21.7 t/s, prefill 158 → 205), with better chat perplexity. Our LoRA worked on it unchanged.
Next came a live census: the engine counts which experts each conversation actually uses and fills the VRAM cache from those counts rather than from a static profile. We added an elastic VRAM cache that gives memory back before a long prompt and takes it again afterwards, and let --pcie auto balance cache misses between the CPU and the PCIe bus. That got us 24–25 t/s at 512K.
The crash at 19,000 tokens
Then a 19K-token request killed the server. It wasn't RAM. The CUDA backend records a reusable CUDA graph the second time it sees an identical pass, and the fourth prefill chunk was the second identical one. So it tried to record a graph for a whole 4,096-token prefill pass, about 1.5 GB, into the ~0.9 GB the elastic cache had left free. Our 12.7K-token benchmark prompt was never long enough to trigger it.
There were two fixes. A small ggml patch stops recording graphs for batched passes (prefill gets nothing from them anyway), and chunks now halve whenever the long-context indexer's scratch would outgrow the headroom. A 50K-token prompt went from crashing to 400 t/s, and 100K prefills at ~330 t/s. 512K stayed.
The profile that corrected us
We had a theory for why decode stalled at ~24 t/s. The host was stopping to synchronise with the GPU eight times per layer, 356 times per token. It's a reasonable story. We fixed it properly: async calls, pinned buffers, and the next layer's expert prediction fused into the current layer's graph. Speed changed by 0%.
A full NVIDIA Nsight trace with every graph node expanded showed why. The GPU's compute kernels take 16 ms per token. The host-to-device copy engine is busy 26 ms per token at about 22 GB/s, which is the PCIe 4.0 ceiling. Decode is PCIe-bound. The syncs were never the bottleneck, so we wrote down that our previous session's conclusion was wrong, kept the cleanups, and changed approach: move fewer bytes.
- Prefetch less, more accurately (
--pf-b 1.0): 23.7 → 25.1 t/s. Over-prefetching was crowding out the experts we actually needed. - Keep pruned experts in spare RAM. Soft-pruned experts the census sees in use now fill leftover arena space instead of being read off the SSD (~4 ms/token): 25.1 → 26.2.
- A lighter dense half (Q4_K): 27.2 t/s.
Then we chose quality over two tokens
To test that last change we measured perplexity with exact math: no skipped experts, 4,000 tokens, identical on re-runs.
| Dense half | Chat ppl | Code ppl | Decode |
|---|---|---|---|
| Q6_K, native (shipped) | 5.27 | 3.60 | 25.0 t/s |
| Q4_K requant | 5.35 (+1.6%) | 3.65 (+1.5%) | 27.2 t/s |
| Q5_K requant (the old default) | 5.60 (+6.2%) | 3.64 (+1.1%) | ~26 t/s |
Two things came out of this. The 5-bit requant we'd been shipping was quietly costing 6% on chat, and it was worse than 4-bit, which we still can't explain. And the native 6-bit file was the best of the three. Q4_K is 2 t/s faster for 1.5% more perplexity. We decided that wasn't worth it and shipped native Q6.
Where it stands
| Stage | Prefill | Decode |
|---|---|---|
| First real run (3.0-bit file, cold) | ~1.6 t/s | 1.75 t/s |
| 3.0-bit file, ablated, tuned (300K) | 158 t/s | 17.8 t/s |
| Maya-S-v2, first wiring (300K) | 205 t/s | 21.7 t/s |
| Now: Maya-S-v2 + LoRA, Q6 dense, 512K | ~400 t/s | 25.0 t/s |
All measured on one PC: an RTX 4090 (24 GB), a Ryzen 7 5700X, 90 GB of DDR4 and an NVMe drive. The workload is a real 12.7K-token chat turn followed by 256 generated tokens, with the abliteration on. Prefill is ~400 t/s up to 50K tokens and ~330 at 100K. One strata-glm-quetza command starts the engine and drops you into QuetzaCodetl, our coding agent.
A few ideas didn't make it. Speculative decoding, which is a 2× trick on dense models, buys +2% at best here: checking a draft touches more distinct experts, and with only ~40% of experts resident the extra fetches cancel the saved passes. Biasing routing toward cached experts lowered the hit rate. Baking the abliteration into the weights wasn't viable. Each of these is written up with its measurements in the repo.
What we took from it
Measure before you conclude, and write it down when you were wrong. The flat-routing finding changed our plan on day one. A sound-looking theory about host syncs cost a session, and a single node-level profile disproved it. Most of the speed came from looking at what the hardware was actually doing: the PCIe bus, the SSD reads, the scratch memory. Clever kernels contributed much less.
The engine, the ggml patch, the serve configs and every measurement behind these numbers (28 sessions of findings) are on GitHub: Indras-Mirror/Strata-GLM. Next up: coding and agentic benchmarks through the served API, and a fast-start mode so you don't wait minutes for the RAM arena to load.
Want a frontier-class model running privately on hardware you already own? We set these up.