Appearance
Squeezing LLMs: Five Levers That Cut Inference Cost Without Cutting Quality
Every part of the LLM stack is getting squeezed at once. This month alone brought a parallel-decoding method that hits 3x throughput without a draft model, a pruning pipeline that shrinks a pathology encoder by 25% while improving its accuracy, a quantization scheme that keeps video generators sharp at 3-bit weights, a watermark that costs less than 1% latency, and a 90M model running on a Sony PSP from 2004.
They read like five different fields reporting separately. All of them attack the same constraint: the compute, memory, and latency between a good model and a deployable one.
Five cracks in the cost wall
The timing is not an accident. As models get better, the gap between "runs in a paper" and "runs in production" widens. The responses are splitting into distinct strategies: generate faster, shrink the model, quantize the weights, lower the auxiliary overhead, and push the floor down until even decade-old hardware can join.
Each strategy has its own failure mode, and knowing which one applies to your bottleneck is most of the battle.
Parallel decoding without a draft model
Autoregressive generation is sequential because each token conditions on the previous one. Speculative decoding fixes that with a draft model that guesses several tokens ahead, then the target model verifies the guesses in parallel. But draft models are a moving part: they need to stay aligned with the target's distribution, and when they drift, acceptance rates and speedups collapse.
Uno takes a different route. Keep the AR weights, train a second set of lightweight diffusion weights that draw multiple tokens in parallel from the same distribution. The AR side uses the standard next-token objective. The diffusion side is learned through a short Diffusion Distillation phase. Since the underlying distribution never changes, the acceleration is lossless by construction. No draft model to train, tune, or keep aligned. The accompanying Psi-Spec samplers enable inference-time scaling at a fixed context length.
The results are hard to argue with. Uno beats leading speculative-decoding methods at every batch size tested, with up to 3x throughput over the base AR model even at the largest batch size the hardware supports. That lands as three served streams where there was one on the same GPUs. The 8B Uno model outperforms the 26B DiffusionGemma and the proprietary Mercury 2 across agentic tool use, coding, and long-context reasoning. A third of the parameters, better results, higher throughput.
Free Pause Tokens attack a related problem. Thinking tokens improve next-token predictions by giving the model extra compute per step, but they do it by extending the sequence, which costs context length and KV cache. This approach carries that compute in a parallel prediction stream over a weight-shared backbone instead of as an extra token. On a 1B parameter model it improves next-token prediction by 2-3 centinats (0.02-0.03 nats). Small per token, but it adds no context length, no KV cache, and essentially no latency at inference. The only bill is training: about 1.14x a standard pretraining run.
Quick Take: None of these methods is a silver bullet. Each one targets a different bottleneck, and the models that ship to production will combine several.
Pruning that beats the original
TAP-Path compresses Virchow2, a 631M-parameter pathology foundation model, without distilling it into a smaller student. It keeps the encoder architecture and makes four targeted changes: validation-driven transformer-block selection, physical removal of redundant blocks, input-adaptive patch-token pruning, and a lightweight gated task head with multi-depth feature recovery.
The retained model keeps 24 of 32 transformer blocks and 70% of patch tokens. Encoder parameters drop from 631.24M to 473.70M, a 24.96% cut, and analytical encoder compute falls from 340.13G to 220.40G FLOPs, a 35.20% reduction. On gigapixel whole-slide images, which pathologists tile into thousands of patches, cutting 30% of tokens and a third of the FLOPs changes the workflow from batch-only to interactive.
Then there's quality: the compressed model hits 87.98% test accuracy on a 32-class histopathology benchmark, beating the full Virchow2's 86.89% and UNI2-h's 87.67%. Calibration is strong too. Brier score 0.1800 ± 0.0005 and failure-detection AUROC 0.9047 ± 0.0060, which matters in medical settings where the model should know when it doesn't know. Frozen external evaluation on 433 CPTAC samples yields 91.22% accuracy, so the compression generalizes beyond the training distribution.
Encoder FLOPs drop from 340.13G to 220.40G, a 35.2% cut. Parameters go from 631.24M to 473.70M. Test accuracy rises from 86.89% to 87.98%, and a validation-only rare-aware objective improved rare-class balanced accuracy in a secondary analysis. The catch: TAP-Path needs task-specific validation data to decide what to prune.
Quantization that respects the pipeline
Video diffusion models are the worst case for quantization. A single generation runs the denoiser dozens of times, and the memory traffic adds up fast. Quantization-aware training is the natural fix, but conventional QAT on video models produces a distinctive failure: prompt semantics, global layout, and coarse motion survive, while textures, sharpness, and fine detail fall apart.
DSAQuant traces this to timestep-agnostic quantization. Denoising is not uniform. Early steps establish global structure and motion. Middle and late steps refine local appearance and high-frequency detail. Training one quantization scheme across all of them maximizes error exactly where perceived quality is decided.
The fix is two-sided. In training, Denoising-Stage Oriented Supervision keeps teacher distillation for early steps, where structure needs stability, then shifts to target-driven optimization for later steps, where detail needs direct supervision. In inference, Denoising-Stage Gated Guidance disables CFG in the final denoising steps, so classifier-free guidance doesn't amplify quantization error into high-frequency artifacts.
Across the Wan and CogVideoX families under W4A4 and W3A3, DSAQuant beats the previous state of the art by up to 6.60 VBench average points under W3A3, the most aggressive setting, while preserving text-video alignment. To translate: at 3-bit weights and activations, quantization no longer has to cost visual fidelity.
The same compression pressure shows up at the distribution layer. ISTA-DASLab's Qwen3.8-27B-GSQ-RCO-GGUF release is a 27B model shipped as a GGUF file, ready for llama.cpp-style runtimes. Quantization has stopped being an experiment and become a packaging step.
Watermarks that cost under 1%
Watermarking usually has a price. KGW's vocabulary permutation and SynthID's multi-layer tournament both add real work per generated token. Stateless Bernoulli Watermarking (SBW) reduces that to a single comparison against a counter-based random number generator. Green-list membership is decided by independent per-token Bernoulli trials, membership checks are O(1), and the whole thing runs as one kernel with zero intermediate allocations.
The proofs hold up: the z-score test remains N(0,1) under the null hypothesis, so detection guarantees are identical to fixed-size green lists. The stateless design also enables full-vocabulary self-salt watermarking, which biases the entire vocabulary with candidate-dependent seeding. That was impractical before: SBW does it over 6000x faster than KGW's self-salt and 2x faster than SynthID, despite biasing more of the vocabulary. It also works across distributed inference, since no state needs to be shared between shards.
End-to-end generation overhead lands below 1% at every batch size. There's also a newly exposed axis here: hash function design. A GPU-native Jenkins hash improved null calibration by 1.8x and produced more diverse text, with ROC-AUC differences under 0.01 across two seeding schemes and eight configuration pairs. For API providers who have been treating watermarking as overhead to budget around, sub-1% changes the decision.
The long tail of small models
At the other end of the spectrum, I pushed a 90M conversational model onto a Sony PSP from 2004 via the LLMPSP project, just to find the floor. It tops out at 0.5-0.6 tokens per second, which means one to three minutes per reply. The model writes crappy poems and short stories, produces non-functional code, and hallucinates freely on basic facts. By production standards it's useless. As a proof that the efficiency floor keeps dropping, it's remarkable.
What these five results share is a refusal to accept the old trade-off where efficiency cost you quality. Uno is lossless. TAP-Path improves accuracy while cutting compute. DSAQuant recovers visual fidelity at 3-bit weights. SBW keeps detection guarantees at O(1) cost. And the PSP experiment shows what happens when you keep shrinking the model until it fits anywhere.
| Lever | Bottleneck targeted | Quality impact | What it costs you |
|---|---|---|---|
| Uno diffusion-augmented decoding | Sequential token generation | Lossless; 8B model beats 26B competitors | One diffusion distillation pass |
| Free pause tokens | Per-token reasoning compute | +2-3 centinats on a 1B model | ~1.14x training compute |
| TAP-Path pruning | Encoder size and FLOPs | +1.09% accuracy over full Virchow2 | Task-specific validation data |
| DSAQuant stage-aligned QAT | Memory bandwidth at low bit widths | +6.60 VBench over prior QAT methods | One-time QAT, CFG gating at inference |
| SBW stateless watermark | Provenance overhead | ROC-AUC within 0.01 of KGW | One RNG comparison per token |
Common pitfalls
Five mistakes show up repeatedly when teams try to apply these techniques.
Pruning without task validation. TAP-Path works because block selection is validation-driven and token retention is input-adaptive. Drop blocks based on magnitude heuristics or a different task's validation set, and reliability metrics drift even when accuracy looks fine. In a medical context, Brier score and failure-detection AUROC are the numbers that matter, not just top-1 accuracy.
Implementing pause tokens as extra tokens. The entire point of the parallel prediction stream is that it adds no context length and no KV cache. If you only have API access or can't touch the training pipeline, you'll pay both costs, and the technique stops being free. This one is only available to teams that control the training run.
Timestep-agnostic quantization for video models. Train W3A3 QAT without stage-wise alignment and you get the classic failure: structure survives, textures collapse. And keep CFG off in the final denoising steps. Guided generation amplifies quantization error into visible high-frequency artifacts.
Reaching for speculative decoding without a good draft model. Uno's edge over speculative decoding is precisely that it removes the draft model. If your draft lags behind the target's distribution, acceptance rates crater and the speedup inverts. Measure the draft-target alignment before committing.
Assuming watermarking must cost several percent. SBW pushes end-to-end overhead below 1% at all batch sizes with the same detection guarantees. The remaining trap is hash quality: a poorly chosen hash ruins null calibration. The GPU-native Jenkins hash alone improved calibration by 1.8x.
One thing to remember: these levers compose. TAP-Path and DSAQuant shrink different resources, parameters versus memory bandwidth. Uno and SBW touch different parts of the request path, generation versus token emission. A production stack can run stage-aligned quantization, a parallel decoder, and a stateless watermark in the same pipeline, as long as each one was designed not to interfere with the others.
The Bottom Line
If you're serving text LLMs under throughput pressure, adopt diffusion-augmented decoding like Uno. It's lossless, needs no draft model, beats speculative decoding at every batch size, and the 8B variant outperforms 26B competitors in agentic tool use, coding, and long-context reasoning.
If you're compressing generative video models, use stage-aligned quantization in the style of DSAQuant. Generic QAT at W3A3 destroys texture and sharpness, while stage-aligned training recovers up to 6.60 VBench points and CFG gating keeps artifacts out of the final frames.
If you're shipping a domain-specific encoder or a regulated API, prune in place with validation-driven selection rather than distilling, and treat watermarking as a sub-1% cost. One thing to watch: stateless watermarks are becoming default infrastructure, and SBW-style provenance should show up in high-volume APIs within the next six months.