Xolotl: Four Frontier Models on One Gaming PC

So what is Xolotl

Hey all, it's been a busy few weeks so I figured it was time for a proper write-up. Xolotl started because I wanted a frontier model as my own coding agent, on my own machine, with nothing leaving the house and no API bill. The big open models are good enough for that now, the problem is they're enormous. DeepSeek-V4.1-Flash is 552 billion parameters and even heavily quantised it's 152 GiB, about six times what my graphics card holds.

The trick is that mixture-of-experts models only use a small slice of themselves for each token. DeepSeek-V4.1 has 15,360 experts and a token only touches 240 of them, so you never need the whole thing at once, you need the right few experts on the GPU at the right moment. Xolotl keeps the hottest experts in VRAM, tens of GB in a pinned RAM arena the GPU copies from directly, and the rest on NVMe, and it moves the ones each token asks for a layer ahead. It's a fork of Niko1221's Strata with about 71,000 lines on top. I went through the engine side in detail in the DeepSeek-V4.1 post, so this one is more about what it runs now, how good the models actually are, and where it's heading.

What one PC runs now

The machine is a Ryzen 7 5700X with one NVMe drive and, until this week, 90 GB of DDR4. "Coding tasks" is our quick screen: 21 HumanEval problems, 10 tool-call tasks and 3 multi-step bug-fix loops, run through the real server with thinking on. The DeepSeek and GLM generation speeds are measured during those tasks, the Qwen3.8-Flash-Next ones at 8K and 16K prompts with a 1,024-token answer, so they're not quite the same test, but they're all real runs on this box.

DeepSeek-V4.1 V4: the one that caught the API

The October checkpoint of DeepSeek-V4.1-Flash (they call it 1017) got three builds from us, and V4 is the one I'm properly happy with. It's the same 152 GiB as V3 but the quality jump was big:

On the full HumanEval (all 164) it got 162. On our Gauntlet, seven multi-turn agentic coding tasks with hidden tests graded after every turn, it solved 6/7 (score 6.50), against the DeepSeek API's 5/7 (6.41) and our September build's 4/7 (6.20). That's one run each, so I wouldn't read too much into 6 vs 5, but a few months ago I honestly didn't think a gaming PC would be sitting level with the API on agent work.

Two things did most of it. The 1,056 hottest experts are kept exactly as DeepSeek trained them, in their native 4-bit format, instead of being squeezed further. And every other expert went through XQ, our own refinement step that learns each expert's quantised weights on the tokens that expert actually sees, through the real model output. It only keeps a result when it beats the plain fit on held-out text, so it can't make an expert worse.

Qwen3.8-Flash-Next at 45 GB: fast and properly good

The new one this week is a 45 GB build of Qwen3.8-Flash-Next. It's built from Swift 1.5, UkisAI's reasoning-efficient fine-tune, with the refusal direction removed from the full-precision weights before quantising. The routed experts average 2.61 bits per weight, the dense backbone is left bit-identical to the 8-bit parent, and the model's big n-gram table (another 29 GB) is read off the disk as it goes.

On the 4090 it does 199 tok/s generating and 3,204 tok/s on a 16K prompt (160 and 2,819 at 8K). On the Gauntlet it solved 5/7 with a score of 6.68, the best we've measured on this model: our previous 55.8 GB build got 6/7 at 6.50, and Qwen's own API got 5/7 at 5.50. It's also the first local run to finish the ray tracer task's second scene, which every other run, the API included, stalls on halfway. It refused 0 of 100 held-out harmful prompts and 0 of 30 normal ones.

I'm testing a 50 GB version of it right now, now that the box has more RAM. The early numbers say it tracks the original model a bit more closely and decodes a bit slower, so I'm running a proper set of agent tasks before deciding which one should be the default.

How we measure, because tok/s isn't the whole story

A fair question that comes up every time someone posts speeds is "but does it actually work properly?". So every build we release goes through the same gates before it goes up:

• Agreement with the original model. We run the same held-out text through the quant and the full model and count how often they pick the same next token, plus the KL divergence between their predictions. This is the number that tells you how much the quantisation actually cost.
• Real agent work. The Gauntlet tasks run with real tools in a sandbox and the hidden tests decide the grade, so a model can't just look right.
• Refusals, both ways. For abliterated builds we check it answers held-out prompts it never saw during the edit, and that it still behaves normally on ordinary ones.
• Speed through the real server, on a fresh prompt and during the tasks, not a synthetic micro-benchmark.

Where something is one run, I say so. The Gauntlet especially swings a bit run to run, which is why the 45 vs 50 GB comparison is getting several runs per task instead of one.

More RAM, more model in RAM

I upgraded the box to 128 GB this week (well, 128 minus a gigabyte, one of the second-hand sticks has a bad chip that I fenced off at boot rather than lose the lot). For DeepSeek-V4.1 that means the RAM arena went from 64 to 80 GB, so fewer experts come off the NVMe each token. With the dense half re-quantised to 6-bit at load, which is the default, decode is about 10% faster than at 8-bit, for a very small quality cost.

Someone on r/LocalLLaMA posted their setup this week, a single RTX 3090 Ti with 128 GB of RAM, getting around 35 tok/s on DeepSeek-V4.1 while coding and 130–150 tok/s on Qwen3.8-Flash-Next. That's on six-year-old hardware, which is exactly the kind of machine I built this for. Code tends to decode faster than prose because the speculative drafts get accepted more often on it, and I'm measuring that properly on my end too.

Small models too

It isn't all giant models. I've been building quants of LFM3-Omni, a 2B model that does text, images and audio, using the same measured approach. A 1.3 GB build agrees with the 8-bit model on 95% of next tokens and runs at about 430 tok/s on the 4090, and our Android port runs LFM3-Omni on a Galaxy A53 at around 12 tok/s. The plan is a small fast helper that sits beside the big models, and something that actually works on a phone. Still testing, so no release yet.

It's not just me anymore

Honestly the best part has been other people picking it up and sending real work back. This week alone nine pull requests went in: Xiaomi2008 added support for Unsloth's 1-bit DeepSeek-V4.1 builds and fused the router into two GPU ops, NektoDron sent request cancellation, smarter prompt-cache checkpoints for agents that edit their history, and a stack of serving fixes, and kelheor tracked down why the Windows build broke and sent the fix. Before that, NektoDron built the conversation pool and batch slots, AlexeyMatskevich added upstream llama.cpp GGUF support and OpenZFS accounting, and Xiaomi2008 got it running natively on Windows. Every new feature lands switched off by default, so nobody's setup changes under them.

Xolotl also loads other people's quants now, ISTA-DASLab's GSQ-RCO and Unsloth's DeepSeek-V4.1 files included, so you're not stuck with mine. Thanks heaps to everyone who's tested it on their own hardware and reported back.

Qwen 4, day one

Qwen 4 is due any day now. If Qwen 4 Flash keeps Qwen3.8-Flash-Next's architecture the engine should already run it, and the 27B looks dense so it's going into the dense engine with the extra attention pieces added. The plan is to have an engine build and a first measured quant out as close to release day as I can. What's known so far is in the repo.

Run it yourself

You'll want Linux (or Windows 11 for DeepSeek), an NVIDIA card with 24 GB, and as much RAM as you can get. Build it:

git clone https://github.com/Indras-Mirror/Xolotl.git xolotl && cd xolotl
bash tools/xolotl/fetch-llama.sh
cmake -S . -B build-xolotl -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON \
  -DSTRATA_GGML_CUDA=ON -DSTRATA_MMQ_KQUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=89 \
  -DSTRATA_GGML_DIR=$PWD/third_party/llama.cpp -DSTRATA_DSV41_FQ_OP=ON -DSTRATA_DSV41_HC_MIX=ON
ninja -C build-xolotl

Use -DCMAKE_CUDA_ARCHITECTURES=86 for an RTX 30 card or 120 for RTX 50. Then grab a model from Hugging Face and serve it. For Qwen3.8-Flash-Next it's one line:

./scripts/xol-qfn.sh --model <shard 1 .gguf> --pack <pack-qfn45>

The DeepSeek and GLM configs are in the repo, and the README has the full steps. Point Claude Code, Codex, opencode or Open WebUI at http://127.0.0.1:<port>/v1 and you've got a local agent. If you run it on different hardware I'd love to see your numbers.

Or have us build one for you

The models above are free and they'll stay free. What we do take on is custom work: a quant of the model you want, sized to the GPU and RAM you actually have, abliteration on a model you own, or a frontier MoE fitted to your workstation with the engine set up and tuned. Every build comes with the same measured report as ours, so you know exactly what you're getting. Small models start at A$120. Have a look at quantisation & abliteration, or book a free call if you're not sure what your hardware can run.