DeepSeek-V4.1-Flash at 27 Tokens/s on One RTX 4090: Our Xolotl Engine

Why build an engine at all

So the honest reason is I wanted a frontier model as my own coding agent, running on my own machine, with nothing leaving the house and no API bill. The big open models are good enough for that now, the problem is they're huge. DeepSeek-V4.1-Flash is 552B parameters, and even heavily quantised it's 152 GiB, about six times what a 24 GB card can hold. The usual runtimes either can't load it on this kind of machine or crawl along at a few tokens a second.

What makes it possible is how mixture-of-experts (MoE) models work. DeepSeek-V4.1 has 40 layers, and each layer has 384 "experts", small feed-forward networks that each get good at different things. For every token, a router in each layer picks just 6 of the 384. So out of 15,360 experts, a token only touches 240 of them. You never need the whole model at once, you need the right few experts at the right moment. If you can get those to the GPU fast enough, everything else can live somewhere slower. That's the whole idea behind Xolotl.

Where the 15,360 experts live

Xolotl started as a fork of Niko1221's Strata, which gave us a solid MoE engine, server and kernels to build on, and it has grown about 71,000 lines on top since. It keeps every expert in one of three tiers:

While one layer is running, the engine is already working out which experts the next layer will probably want and starting to move them, so the copies overlap with the maths. The VRAM tier is a cache that evicts the least recently used expert in each layer, and it grows and shrinks with how much VRAM a request actually leaves free. During decoding most of the experts a token needs are already on the card.

There's also a small nudge in the router during generation. When two experts are nearly tied, it prefers the one that's already in VRAM and avoids ones that would have to come off the NVMe. It only changes which experts get picked, never how their outputs are mixed, and every expert can still be chosen if the router really wants it. It's the difference between about 3.5 tok/s with perfectly exact routing and 25+ tok/s. It isn't totally free: code perplexity goes up a bit, while agentic text actually gets better (the quant had already given its cheapest formats to the rarely used experts the nudge steers away from). The coding and agent tests didn't move though, still 21/21, 10/10 and 3/3.

Where a token's 38 milliseconds go

At about 26 tok/s each token takes roughly 38 ms. Half of that, ~19 ms, is the "dense" part of the model (attention and the shared layers every token goes through), which is limited by how fast the 4090 can read its own memory, about 880 GB/s. The other ~16 ms is the expert tier: computing the 240 experts, with any misses being copied in from RAM or disk.

That split is what got the speed up over time. The very first working GPU run of this model did 3.4 tok/s generating and 27 tok/s on prompts, and took 211 seconds to load. Now it's 25.5–26.3 tok/s on a short prompt, 22.7–28.2 tok/s inside real agent sessions at 20–32K context, and it loads in about 50 seconds. There wasn't one magic fix, it came from a long list of small changes, each one measured before the next, and every step is written up in the repo if you want the gory details.

On top of that there's DSpark, speculative decoding with exact sampling. A small draft model guesses a few tokens ahead and the big model checks them all in one pass. Each pass of a 552B model is expensive, so landing about 3 tokens per pass is a big deal. "Exact sampling" means the output follows the same distribution as without it, so it's a pure speed trick.

Prompts: fast, and exact

Prompts take a different path. Instead of going token by token, a long prompt is processed in big chunks, and each layer's experts are streamed onto the card once per chunk and run as one batched multiply on the 4090's int8 tensor cores. That's how a 552B model gets to 244–248 tok/s on an 8K prompt, 288 tok/s at 16K and 266 tok/s at 20K. A fresh 20,000-token prompt takes about 75 seconds.

One rule we kept the whole way: every prompt speed-up has to give byte-identical results. We check it with a hash of the final logits on an 8,192-token prompt, so nothing on the prompt side trades quality for speed.

Agent turns: the hard case

Agents are tricky for an engine like this. An agent re-sends the whole conversation every step, and each new step adds a small chunk like a tool result. The server keeps the previous turn's cache so it only processes the new bit, but on DeepSeek-V4.1 even a +180-token turn touches about 7,000 different experts, and around 2,600 of those only live on the NVMe. So small turns end up limited by the disk.

Two things cut that down. Small turns now only load the experts they're actually routed to, which took a +189-token turn from 9.83 to 8.45 seconds. And the arena adapts to the conversation: the engine counts which experts this conversation keeps using, and while the agent is busy running its tools, it quietly swaps those experts from the NVMe into the RAM arena. That cut disk reads per small turn by 21–30% and took small turns from 8.48 to 7.60 seconds. Every swapped expert is byte-checked (0 bad out of 2,500), and the coding and agent results didn't move.

Then there's the conversation pool, which NektoDron built. Every conversation is saved at its turn boundary in pinned RAM, and a new request resumes from the longest matching start, pictures included. With three agents taking turns at 30–60K tokens each (measured on a 5090), a turn went from 47.5 seconds to about 4. There are also batch slots that run several requests in one pass, and prompts past 1M tokens now finish without running out of memory.

The DeepSeek-specific weirdness

DeepSeek-V4.1 has a few unusual parts that each needed their own engine work. Engram is an n-gram memory of about 200 GB that gets read from disk while the model runs, around 48 small random reads per token. We learnt the hard way that the filesystem matters heaps here: on exFAT those reads get queued one at a time, about 13,000 reads/s, while the same NVMe on ext4 or NTFS does about 290,000. That alone can be 5–10 ms a token.

It also uses four parallel residual streams that get re-mixed in every layer, and a compressed sparse attention. To fit a 512K context on a 24 GB card alongside the expert cache, we built a 4-bit KV cache (the model's memory of the conversation so far) that gives the same results as full precision.

One engine, four models

Every number is measured on that one machine (Ryzen 7 5700X, 90 GB DDR4, one NVMe), through the real server, while it solved real tasks. All four sit behind the same server with each model's own chat template and tool-call format, so Claude Code, Codex, opencode or Open WebUI can point straight at it, with streaming, tool calls and a thinking budget. GLM-5.3-Flash has its own engine path with an abliteration LoRA applied on every route, which we wrote up in Strata-GLM.

It runs on more than my PC now

Honestly the best part lately has been other people picking it up and sending real work back. Xiaomi2008 got it building and fast on native Windows and added a way to use a second GPU through a helper process (40 tok/s on a 5090 + 4080). AtticusG3, Niko1221 and bsvinay did the multi-GPU layer split, and there's an experimental path for old Tesla V100s. NektoDron did the conversation pool, batch slots and the 1M-token fixes. Thanks heaps to everyone who's tested it on their own hardware and reported back, it's made the engine way better than I could've on my own.

The quants that feed it

An engine like this is only as good as the model file you give it, so we build our own quants too. A plain quant gives every weight the same number of bits. That wastes bytes, because some experts get routed constantly and are sensitive to error, and others barely get used. So our builds give each expert its own tier: ternary (three values per weight), 2-bit, 3-bit, or left at the native 4-bit, with the bit budget spent where it measurably buys quality.

Each expert is fitted with GPTQ, scale refitting and coordinate descent, and then XQ, our own refinement step, which learns the quantised codes and scales on each expert's own routed tokens through the real model output. It only keeps the result when it beats the plain fit on a held-out slice, so it never makes an expert worse. One thing that surprised me early on: you can't rank these experts from the weights alone, DeepSeek trained them in FP4 so they all look almost the same on paper. You have to measure what the model actually does with them.

The comparison I care about most is against DeepSeek's own API on our Gauntlet suite, seven multi-turn agentic coding tasks with hidden tests graded every turn:

That's about 97% of the API's score from a gaming PC, which is way closer than I expected when I started. It's not free though. Across the whole Gauntlet both used about the same number of tokens, but on single hard coding problems the quant often thinks 3 to 10 times longer than the API to reach the same answer (6.5K thinking tokens where the API needed 2.1K, or hitting a 27.6K cap where the API was done in 2.4K). So it gets there, it just takes more thinking to do it, and at 27 t/s that's real time. Closing that gap is a big part of what V4 is for.

What's next: V4

V4 of the October checkpoint quant is running right now: a much larger calibration set from real agent and code data, a fresh tier map, and XQ on every expert, at the same 163 GB as V3. Four layers in, each layer's expert error on held-out text is 29–48% lower than the same layer in V3. That's an in-progress number, not the final word, the release will have the full gates (perplexity, agreement with the full model, Gauntlet, decode speed). This week we also got the quant pass itself about twice as fast, so it should land around 26–27 October. I'm not calling it before the gates are in, but I reckon it's going to be a good one.

Try it, or have us build one for you

Everything is open: the engine, the serve configs and every benchmark behind these numbers are on GitHub, and the models are on Hugging Face. If you run it on different hardware, I'd love to see your numbers.

And if you've got a big model that needs to run on hardware you already own, with a quality bar you can actually check, that's what we do: model quantisation as a service, from 27B dense models up to 500B+ MoE, plus the engine setup to serve it. Or if you just want a frontier model running privately in your office, we set those up too.