DeepSeek-V4.1 at 41 tok/s on a Gaming PC: Xolotl's Fast Mode

Where things are at

It's only been a week since the last write-up, but the DeepSeek numbers moved enough that I wanted them written down properly. Quick recap if you're new: Xolotl is our open-source engine for running huge mixture-of-experts models on a normal PC. A token only uses a small slice of the model's experts, so the engine keeps the hottest ones on the GPU, a big pinned chunk in RAM, and the rest on the NVMe, and fetches what each token needs as it goes. It's a fork of Niko1221's Strata with a lot on top.

The main change since last week is how I measure. I've stopped leading with HumanEval. Every model we run scores 19 to 21 out of 21 on our quick HumanEval, tool-call and bug-fix screen, so it can't tell them apart anymore. The Gauntlet can. It's a set of hard agentic coding tasks, ten turns each, with real tools in a sandbox and hidden tests deciding the grade after every turn. The full-precision DeepSeek API doesn't ace it either, which is the point. We use the v3-small set now: 6 tasks, thinking on, every model served exactly as I run it day to day. The speeds are measured during those tasks, so they're the speeds you'd actually get from a coding agent.

Score is the sum of each task's progress (1 for a solve, partial credit from the hidden tests otherwise). The API column is each model's own full-precision API on the same tasks. The Gauntlet does swing a bit between runs, so read it as "these are in the same league" rather than a ranking to the decimal, but it's nice to see our DeepSeek and Qwen builds sitting at or above their own APIs.

Fast mode: smaller copies of the experts that cost the most

Once the hot experts are on the GPU, DeepSeek-V4.1's speed on my box comes down to one thing: how many bytes of expert weights have to come across the RAM bus and off the NVMe for each token. The biggest experts, the ones in DeepSeek's native 4-bit format and our 3-bit ones, are about half of those bytes. Fast mode keeps a 2-bit copy of just those two kinds in a separate file and reads that instead. The model file itself isn't touched, and turning it off is unsetting one variable.

Code decodes faster than prose because the speculative drafts get accepted more often on it, which is why the task number is higher than the 8K probe. It is lossy, which is why it's opt-in. On the agent tasks I couldn't measure any cost, but the next-token check shows it changes a token or so in a hundred, so if every token matters, use normal mode. The 2-bit copies are quantised with an importance matrix from DeepSeek's own activations, which is what got the loss under a point. A plain 2-bit copy cost about two and a half.

The idea came from PROMETHEVS by Ryan Flavell, which runs DeepSeek-V4.1 on a 3060 by computing everything with a 2-bit copy of the model and then correcting the tokens it's unsure about with the exact weights. I tried their approach on my box too: a 2-bit copy of every expert, built straight from DeepSeek's original weights. It's a 178 GB file, so more of it has to come off the drive, and it ran at 17.6 tok/s against fast mode's 38 on the same test, and agreed with the full model a bit less often. So on a machine like mine, our build plus fast mode is the better trade. What I haven't tried yet is their correction step, where the unsure tokens get recomputed exactly. That's queued now, with our normal experts doing the correcting so it needs no extra files. Big thanks to Ryan for publishing it all.

Loads in 44 seconds, and NVMe straight into VRAM

DeepSeek used to take about a minute and a half to load, most of it converting the dense half of the model at startup. That converted copy now gets saved next to the model the first time, so every load after that takes 44 seconds, with the same output.

The more experimental one is Xolotl Direct. Normally an expert read off the drive goes NVMe to RAM to a staging buffer and then over to the GPU, crossing the RAM bus several times, and on a DDR4 box the RAM bus is the wall. With Xolotl Direct the drive writes the expert straight into the graphics card's memory, on a normal GeForce card and the normal NVIDIA driver. One expert takes 3.1 ms instead of about 11, with no RAM traffic at all, and the output is identical. On its own it only gives about 3% more decode, because those reads were already mostly hidden behind other work, but it opens up a few things I want to try next. It's Linux only, needs Resizable BAR and two small kernel modules, and I've only tested it on my own machine, so treat it as experimental and I'd love reports from other cards and drives.

The Qwen side

Qwen3.8-Flash-Next got two quieter speed-ups this week, both on by default now: RTX 40 specific kernel settings took decode on a 16K prompt from 139 to 150 tok/s, and running the 1-bit experts through the faster matrix path sped up long prompts by 13 to 17%. I also finished comparing the 45 GB and 50 GB builds over several Gauntlet runs. They came out even on tasks and the 45 GB one is faster, so 45 GB stays the default.

The community keeps sending good stuff

NektoDron cut the wait for a short request after a long one, with the disk prompt cache on, from about 9 seconds to a third of a second. jesdga95 ported a batch of fixes and kernel tweaks from his Basalt project, cpacsm75 sent RTX 5090 numbers at 256K and 512K context, and kelheor found and fixed the Windows build again when my new code used Linux-only calls. Thanks heaps, all of you. The full list is in the repo's credits.

What's next

An RTX 3060 arrives this week, so I'm getting the engines ready to use a second card as an extra layer of fast memory. On DeepSeek that means more experts off the RAM and the drive. For voice mode it means the small LFM3 voice model can live on the 3060 and give the big model the whole 4090 back. I'm also reading through the new speculative decoding papers, D3D in particular, which drafts on a separate GPU while the main one checks the draft, and that sounds like a good fit for a two-card box. And Qwen 4 still isn't out, but there's a watcher on Hugging Face that starts the download the minute it lands.

Try it

The README has the build and serve steps, and there's a page on whether it'll run on your machine. For fast mode, download the v4-fast/ folder from the model's Hugging Face page and point XOLOTL_SHADOW at it, or build your own with ds41_generate --build-shadow. If you run it on different hardware, I'd really like to see your numbers.

Or have us build one for you

The models stay free. What we take on is custom work: a quant of the model you want, sized to the GPU and RAM you actually have, abliteration on a model you own, or a frontier MoE set up and tuned on your workstation. Every build comes with the same measured report as ours. Have a look at quantisation & abliteration, or book a free call if you're not sure what your hardware can run.