The real cost driver: output tokens

Serving an LLM in production is a cost game with three line items: weights, KV cache, and decode tokens. Most teams treat all three as fixed and shop for cheaper GPUs. A cluster of recent work says otherwise.

Four papers and one practical community hack all landed in the September 2026 preprint batch. They attack serving cost from four independent angles: training models to emit fewer tokens, quantizing weights with structured transforms, compressing the KV cache to 2 bits, and routing each conversational turn to the cheapest adequate model. The gains are orthogonal. You can stack them.

Decode is where the money goes. Every generated token costs a full autoregressive forward pass, and decode has none of the parallelism that prefill gets from processing a prompt in bulk. Output length is the most controllable cost factor in your stack, and standard preference alignment makes it worse. DPO and RLHF reward responses that look good to a judge, and verbosity correlates with judged quality even when it adds nothing. You pay for every extra word.

Lever one: train models to emit fewer tokens

LOCUS (Task-Aware Low-Rank Post-Training) is built on an odd finding: the parameterization of the post-training update itself changes generation length. Low-rank subspaces alter sequence length without touching the alignment loss. So instead of changing the objective to penalize long outputs, you change the subspace where the update lives.

Concretely, LOCUS selects a task-aware low-rank adaptation subspace that minimizes output-token cost subject to a utility constraint. The backbone stays frozen, and the native preference objective runs inside that subspace.

Results on Anthropic HH-RLHF dialogue preferences, with protocol-matched baselines:

• Up to 39.84% shorter continuations on Pythia-2.8B
• 14.87% to 17.58% shorter on Qwen2.5-3B
• 0.24% to 0.28% of parameters updated
• No material change in the internal preference diagnostic

A 39.84% token cut is nearly 40% less decode compute and latency per request before you touch a single weight. Even the Qwen result, roughly 15% to 18% shorter, is a visible cost improvement for chat traffic. And 0.24% of parameters is LoRA-scale: a single-GPU training run, not a cluster job. The catch is scale. The experiments used ~3B backbones, so what happens at 7B or 70B is an open question.

Lever two: 2-bit factors via structured transforms

Weight quantization is the oldest lever here, and the new result is about making it faster and more stable. The method revisits Kashin decomposition, where each weight matrix factors into two components: one with bounded infinity norm, and one whose infinity norm stays bounded after an orthogonal transform.

The bottleneck in earlier Kashin schemes was that dense random orthogonal matrix. Per-iteration cost scaled as O(N²), which makes the quantization pass itself the bottleneck. The new work replaces it with a sign-randomized Discrete Cosine Transform, dropping the cost to O(N log N). On a typical 4096-wide hidden dimension, that's a few hundred times fewer operations per iteration, which is the difference between a quantization pass that finishes during a coffee break and one that stalls the deploy job.

Two details make the difference practical. The greedy alternating-update algorithm guarantees the four-peak distribution that stable 2-bit clustering needs, and cluster centers get closed-form initialization. That removes the multi-restart k-means bottleneck that plagued prior work.

The method lands at 4-bit per channel on OPT, Llama-2, and Pythia with results competitive against OPTQ, QuIP, and QuIP-RG. Robustness is where it separates from the pack. On stress configurations where QuIP variants diverge to four-digit perplexity on Pythia-6.9B or abort with NaNs in LDL back-substitution on Mistral-7B, Kashin-DCT stays close to the FP16 baseline.

At inference, each weight decomposes into two 2-bit factor codes per channel. That structure maps cleanly onto native-2-bit hardware, which matters if you're planning around the next generation of low-precision accelerators.

Quick Take: Output length, weight precision, cache size, and model choice are four independent serving-cost dials, and each of these results shows you how to turn one without breaking quality.

Why quantization works at all

A companion paper asks the question most of us skipped: why does post-training quantization work? Naively, weight errors should accumulate with depth and corrupt next-token prediction. Randomly initialized models do fall apart that way. Pretrained models mostly don't.

The paper identifies two mechanisms. First, the error a layer newly introduces tends to oppose the error it inherits from its input. The two cancel partially, so the gap between full-precision and quantized forward passes grows slowly instead of compounding. This counteracting residual interaction develops during pretraining; random models don't have it.

Second, the LM head's geometry preferentially preserves the scores of high-ranked tokens, which are the model's most confident predictions. So even when hidden-state error does accumulate, it lands where it does the least damage.

The practical takeaway: quantization robustness is a property your pretrained model already has. You don't need quantization-aware training to get it. But you also can't assume any model will quantize cleanly. Heavy fine-tuning or a model far from the base distribution changes the error-canceling balance, so benchmark on your own workload before trusting PTQ.

Lever three: compress the KV cache

KV cache is the memory line item that grows with context and batch. For a 7B model with full attention heads serving 128K-token contexts, an FP16 cache is roughly 64 GB, about four times the 14 GB the weights occupy. Quantize that cache and you multiply the requests you can serve per GPU.

KV cache quantization is standard in text-only LLMs, but it doesn't transfer to multimodal models. OmniKVQuant tested TurboQuant, a representative rotation-based method, on audio, video, and text caches and found two failure modes: temporal key drift and heterogeneous value geometry. Keys shift across time windows in ways a global quantization range can't capture, and values from different modalities have different distributions.

The fixes are training-free: set the key quantization range over each short window of the input stream, and rotate values separately per modality. The result is 2-bit KV caches on Qwen2.5-Omni and Qwen3-Omni with performance largely preserved across seven audio-visual benchmarks.

2-bit KV cache preserved across seven audio-visual benchmarks 8x memory reduction per cache entry vs FP16 64 GB FP16 cache for a 7B model at 128K context, against 14 GB of weights 39.84% worst-case token cut from LOCUS on Pythia-2.8B 16.26% routing accuracy gain from SWRouter over the best single LLM

A 2-bit cache is 8x smaller than FP16. That's the difference between serving a handful of long-context requests and serving eight times that many on the same GPU, all other costs held equal. OmniKVQuant also ships a fused Triton decode kernel that reads the packed 2-bit cache directly during attention, so the dense FP16 cache never gets materialized.

The community is already hacking KV caches

While the papers formalize these ideas, a Reddit post showed how far a projector model gets you. The demo replicates what V4.1 Flash does to KV caches for fast prefill, applied to Qwen3-8B. Everything is public: a blog post, a Hugging Face repo named Q3-8B-KVA-Projector, and a browser demo you can run yourself.

When I ran the demo, prefill felt dramatically snappier and output quality held up better than I expected for a method that predicts what the KV cache would contain instead of computing it. The obvious question in the thread was whether the approach scales to 27B. Nobody has answered that yet.

The framing is what deserves attention. The full KV cache is redundant across tokens and layers, and an 8B model's learned projector already captures enough of that redundancy to be practically useful. If that holds at scale, KV approximation becomes a genuine alternative to quantization, with no calibration passes and no custom kernels.

Lever four: route each turn to the right model

Routing cuts cost by not invoking the biggest model. SWRouter (Similarity-Contractive Window Router) targets multi-turn conversations, where single-turn routers fall apart.

The authors identify two problems. First, context construction: how do you segment, retain, and fold history into the current prompt without losing information or confusing the model? Second, evaluation: if the router picks a model but the prompt is badly constructed, you can't tell whether the router failed or the context did.

SWRouter answers with a similarity-based context segmentation mechanism plus a dual-metric evaluation framework that decouples construction accuracy from routing accuracy. The reported gains: 16.26% improvement in evaluation accuracy over the best individual LLM, and another 8.22% over the Conv-ID context baseline.

The lesson generalizes: in multi-turn serving, routing quality and prompt construction quality are entangled. Treating routing as a one-line classifier on top of truncated history throws away most of the benefit.

The four levers side by side

Lever	Method	What it cuts	Reported gain	Training cost
Output tokens	LOCUS	Decode compute, latency	14.9-39.8% fewer tokens	0.24-0.28% of params
Weights	Kashin-DCT	Weight memory, bandwidth	Matches OPTQ/QuIP at 4-bit	None (PTQ)
KV cache	OmniKVQuant	Memory per request, prefill	8x smaller at 2-bit vs FP16	None (training-free)
Routing	SWRouter	Cost via model selection	+16.26% vs best LLM	Router training only

These levers are orthogonal. You can train shorter outputs, quantize weights, compress the cache, and route to cheaper models in the same stack. Each attacks a different bottleneck, and none requires the others. The only caveat: OmniKVQuant and Kashin-DCT are training-free, LOCUS needs a small fine-tune, and SWRouter needs labeled routing data.

The chart below mixes two different metrics: token-length reductions from LOCUS and evaluation-accuracy gains from SWRouter. Read it as a range of what's realistic per lever, not a head-to-head ranking.

Common pitfalls

1. Applying text-only KV quantization to multimodal models. Rotation-based methods assume a single value distribution. Audio, video, and text tokens have different geometry, and a 2-bit cache built on that assumption will quietly degrade accuracy. Windowed key ranges and per-modality value rotation fix it.

2. Assuming any pretrained model quantizes cleanly. The error-canceling behavior develops during pretraining, so it's robust for released checkpoints, but heavy fine-tuning can shift it. Run your own quantized vs FP16 comparison on your actual traffic before committing.

3. Reusing a single-turn router for multi-turn traffic. The router's decision only matters after you've decided what context to feed it. Naive history truncation injects construction error into the router's input, and you'll blame the wrong component.

4. Letting the quantization pass become the bottleneck. If your Kashin-style method uses a dense random orthogonal transform, the O(N²) per-iteration cost dominates the whole procedure. Structured transforms like DCT bring it to O(N log N), which turns a multi-hour quantization job into a coffee break.

5. Optimizing preference alignment without a token-cost constraint. DPO will happily add 30% verbosity if the judge prefers it. Constrain output length inside the training subspace or objective, not with post-hoc truncation that degrades quality at serving time.

One thing to remember

Every number in this article is a snapshot of models that are weeks old. The durable insight is simpler: serving cost is a function of output tokens, weight precision, cache size, and model selection, and each of those is independently controllable. Pick one lever, measure it on your workload, and ship.

The bottom line

If you serve high-volume dialogue on a ~3B model, start with token reduction. LOCUS closes in on 40% fewer output tokens at LoRA-scale training cost, and shorter outputs compound with every other optimization in your stack.

If you're memory-bound on long contexts, go after the KV cache first. An 8x cut from 2-bit quantization, or a learned projector in the style of the Q3-8B demo, adds capacity per GPU without a fine-tuning run, and both stack on top of weight quantization.

If you're deploying a router, treat context construction as part of the routing design, not a pre-processing step. The 8% to 16% gains from SWRouter only appear when construction accuracy is decoupled from routing accuracy. One thing to watch: KV approximation, if it scales past 8B, will become a real alternative to quantization within a year.