Cost Per Token Is Where the Fight Moved

While the financing story sucked up the week's oxygen, the open-source serving stack did something quieter and, if you actually run models, more useful. It kept making DeepSeek V4 cheaper to serve, fast. NVIDIA's claim is that its full inference stack cut DeepSeek V4 token cost by up to 5x in about one month on Blackwell. Vendor math, caveats below, but the direction is real and the mechanisms are worth understanding, because they tell you where to spend your own optimization budget.

Here's the thing about a 5x cost drop in a month: it doesn't come from one clever trick. It comes from four independent wins that happen to compound. Let's take them in order.

vLLM 0.24.0: KV-cache tiering grows up

On June 30, vLLM shipped 0.24.0, 571 commits from 256 contributors. The release notes are dense, but two lines matter for cost-per-token.

The first is multi-tier KV-cache offloading. KV cache is the memory tax you pay for context; every token you've already processed sits in HBM waiting to be attended to, and at long context it dominates your memory budget. DeepSeek V4 is already unusually kind here, its architecture holds KV cache to about 9.62 GiB per sequence at 1M context, roughly 8.7x smaller than a comparable V3.2-style stack, and vLLM further squeezes it with fp4 for the indexer cache and fp8 for the attention cache. But the real shift in 0.24.0 is that cache no longer has to live entirely in HBM. You can now offload to a secondary object-store tier, with per-request offloading policy, integrated with the Hybrid Memory Allocator. In plain terms: you can serve more concurrent long-context sessions on the same GPU because the cold cache spills somewhere cheaper. That directly moves your cost-per-token, because it moves how many requests one GPU can hold.

The second is online FP8 per-token-per-channel (PTPC) quantization (#44132). Per-token-per-channel is the compromise that makes online FP8 actually usable: instead of one scale factor for the whole tensor (cheap, lossy) or offline calibration (accurate, inflexible), you scale per token and per channel on the fly. You get most of the accuracy of calibrated quantization without the calibration step. For anyone serving a model they can't or won't pre-quantize, this is the difference between "FP8 in theory" and "FP8 in production."

Also in the release: MiniMax-M3 support, Model Runner V2 now default for Llama and Mistral dense models, and a coordinated security-hardening batch (a Starlette bump for CVE-2026-48710 plus several DoS fixes). Worth a read of the full notes before you upgrade, though, there are real breaking changes: C++20 required, Transformers v4 deprecated, and a handful of models removed outright (ERNIE, Xverse, Dots1, Bamba, Mono-InternVL). Check your fleet against that list before you pip install -U.

SGLang: the kernels are being written by agents now

On July 2, LMSYS published "Agent-Assisted SGLang Development", documenting something that would have read as science fiction eighteen months ago: coding agents landing production kernel optimizations upstream. Their KDA-Pilot system authored PRs, real ones with numbers attached, delivering B200 wall-clock geomean speedups from 1.13x to 2.75x on tracked production rows.

Sit with that for a second. The frontier serving stack, the thing that determines the cost of every open-weight token served worldwide, is increasingly co-developed by the same kind of agent you'd point at your own codebase. If you do AI-assisted coding for a living, this is the clearest signal yet that the technique has crossed from "autocomplete with delusions" into "writes the CUDA norm-scale-shift fast path and the maintainers merge it." The stoic framing: the work that can be delegated is expanding, which means the work worth your attention is narrowing to judgment, architecture, and the things an agent still can't hold in its head at once.

On the serving side proper, SGLang delivered DeepSeek V4 on GB300 at Day-0 with 5x higher throughput at the same interactivity, and Spec V2 is now the default speculative-decoding path. The stack reportedly powers 400,000+ GPUs across xAI, AMD, NVIDIA, Oracle, Google, Azure, and AWS.

Where the compounding actually comes from

NVIDIA's token-cost post is the clearest map of the four wins that stack to the 5x, so credit where due even if the numbers are vendor-stated. The recipe:

• Disaggregated serving — split prefill and decode onto separate hardware pools so each runs at its own optimal batch shape, instead of one compromise config for both.
• Large expert parallelism over NVLink — for MoE models like DeepSeek V4, spread the experts across GPUs and lean on NVLink's bandwidth so the all-to-all routing doesn't become the bottleneck.
• NVFP4 precision — 4-bit floating point for the parts of the model that tolerate it, roughly halving memory traffic versus FP8 on those tensors.
• Multi-token prediction — the decode-side analog of speculative decoding, generating and verifying multiple tokens per step.

Individually, each is a modest win. Stacked, NVIDIA claims up to 20x throughput per GPU. Named production users include Baseten (+50% tokens/sec on DeepSeek V4 Pro via TensorRT-LLM), Cognition using Dynamo for RL, and DigitalOcean/Hippocratic AI reporting +30% throughput at sub-0.5s time-to-first-response across 10M patient calls.

The actionable version

If you run open-weight models, the current cost frontier is legible and mostly free:

The two levers that moved this month are KV-cache tiering (now first-class in vLLM 0.24.0, so you can hold more concurrent sessions per GPU) and speculative decoding (SGLang Spec V2 as the default path, plus multi-token prediction on the decode side). Layer NVFP4 or online PTPC FP8 quantization on top and you're on the same curve NVIDIA is quoting, via the open-source day-zero path rather than a proprietary one. The CUDA-native day-zero deployment story, vLLM and SGLang supporting new architectures the day they drop, is the durable advantage now that token economics dominate infra decisions. That's the moat, and it's an odd one: NVIDIA's is partly built on open-source software it doesn't own.

Two honest caveats, because the instructions demand it and because they're true. The "up to 5x" and "up to 20x" figures are vendor-stated, citing SemiAnalysis InferenceX benchmarks, and haven't been independently reproduced. And the whole inference-economics story rests on DeepSeek V4 costs continuing to fall on Blackwell; if that curve flattens, the narrative cools with it. But the mechanisms, KV tiering, quantization, disaggregation, speculative decoding, are real, upstream, and yours to use today. That part doesn't depend on anyone's press release.

Sources: vLLM 0.24.0 release notes; LMSYS/SGLang blog; NVIDIA inference software post; vLLM DeepSeek V4 attention writeup. Throughput and cost multipliers are vendor-stated and not independently reproduced.