Appearance
Local Inference Is No Longer a Hobby Project
The last two weeks of local inference research all point one direction: the price of serving tokens is dropping on every front at once. Cross-model KV sharing takes a cache from one LLM and feeds it to another, cutting prefill cost by up to 67%. A team ran a 175B protein model on a laptop with 32GB RAM and 8GB VRAM. Hugging Face released 207 WebGPU kernels that run 2.57x faster than ONNX Runtime Web on the same GPU. These are separate projects, but they all answer one question: how do you get production-quality inference without datacenter hardware?
Cross-model KV sharing makes context portable
KV caches are usually disposable. Every model recomputes its prefill from scratch, even when another model in the same system already processed the identical prompt. A paper on arXiv (2608.30963) proposes translating KV states between models that differ in scale, architecture, attention configuration, tokenizer, and even family. The translator is learned, not hand-coded.
The numbers are encouraging. Qwen2.5-7B translated to Qwen2.5-1.5B lifts LongBench2 accuracy from 27.59% to 34.48%, a 6.89-point gain over the native 1.5B baseline. Cross-family Qwen2.5-1.5B to Gemma-2-2B cuts target-side prefill cost by up to 67.05% at 4K context, with decoding perplexity holding close to native. The most heterogeneous case, Llama3.1-70B to Qwen2.5-7B, delivers 44.0% accuracy against 45.7% native, while measured latency drops from 899ms to 138ms. That's an 84% latency reduction for 1.7 points of accuracy.
If you're building a multi-agent system where different models handle different steps, you no longer need to prefill every turn from scratch. The KV cache becomes a transferable artifact, like an embedding cache or a retrieval index.
Key numbers
- 67% target-side prefill reduction with cross-model KV handoff (Qwen2.5-1.5B → Gemma-2-2B at 4K)
- 84% latency drop when Llama3.1-70B KV states feed Qwen2.5-7B (899ms to 138ms)
- 2.57x geometric mean speedup of Hugging Face WebGPU kernels over ORT WebGPU
- 100x throughput of an 8-card A100 cluster on a single RTX 4060 laptop, running a 175B model
Quick Take: The KV cache is becoming a first-class computational artifact. Treating it as model-local waste is becoming the expensive choice.
Consumer GPUs are now fast enough to matter
Puget Systems put two AMD Radeon AI PRO R9700 cards through a full inference benchmark. Each card has 32GB of GDDR6 and 640 GB/s of memory bandwidth. At Puget's pricing, two cards cost about $3,760, which is less than half the cost of two RTX 5090s with the same 64GB aggregate VRAM.
The 64GB tier looks like this:
| Model | Params | Type | FP16 VRAM | Runs on 2x R9700? (64 GB) |
|---|---|---|---|---|
| Qwen2.5 3B Instruct | 3B | Dense | ~6 GB | Yes |
| Qwen3 8B | 8B | Dense (thinking) | ~16 GB | Yes |
| Llama 3.1 8B Instruct | 8B | Dense | ~16 GB | Yes |
| Qwen3.6-27B | 27B | Dense | ~54 GB | Tested (Q4 via llama.cpp) |
| Gemma 4 31B | 31B | Dense | ~62 GB | Requires bfloat16 |
| DeepSeek V4 Flash | 284B | MoE | ~568 GB | No |
Single-GPU decode is memory-bandwidth-bound. The R9700's 640 GB/s ceiling means parameter count matters less than you'd think. Single-stream numbers from Puget's concurrency=1 runs:
Then reality hits: stock vLLM can't span two R9700s. Both tensor parallelism and pipeline parallelism fail during RCCL initialization on RDNA 4, with a HIP error before the model loads. The two-card recipe that worked on vLLM's old v0 engine no longer works on v1 images. llama.cpp's ROCm backend sidesteps RCCL by using direct HIP transfers, and it works reliably. That's the multi-GPU path today.
The 3090 is the workhorse that won't die
While AMD fights the software gap, the used 3090 market keeps turning. The club-3090 repo collects battle-tested compose files for serving LLMs on one or two RTX 3090s across vLLM, llama.cpp, and ik_llama. The maintainers report 89/127 TPS for Qwen3.6-27B on dual 3090s with vLLM, and a single-card llama.cpp config that handles 200K context without prefill cliffs.
The most aggressive optimization I've seen this month is a custom engine that pushed Qwen3.8-27B to 2,000 prefill tokens per second and 132 decode tokens per second on a single RTX 3090. The key was a custom kernel that matches fp32 output quality at int8, with cosine similarity of 0.99997. That is the difference between a toy setup and something that can serve a code assistant.
On Reddit's LocalLLaMA and GitHub threads, the pattern matches my own experience. Long-context runs on a single 3090 are safe, but only if you use llama.cpp. vLLM single-card hits prefill OOMs past ~50K tokens. The other trap is benchmarking reasoning models: Qwen3 8B recorded 0.0 tok/s in a 30-second window because its thinking phase ate the entire interval. Extend the window to 120 seconds and you get 27.7 tok/s. If you trust the first number, you'll think the setup is broken when it isn't.
WebGPU kernels bring inference to the browser
Hugging Face released @huggingface/kernels, a JavaScript loader for 207 versioned WebGPU kernels published on the Hub. Each kernel carries a manifest, correctness tests, benchmark cases, and WGSL shader templates. They benchmarked 809 matching cases against ONNX Runtime Web on an Apple M4 GPU. The geometric mean speedup was 2.57x; the median was 1.90x.
The per-operation numbers tell a cleaner story:
| Operation | HF WebGPU kernel | ORT WebGPU | Speedup |
|---|---|---|---|
| Add | 0.064 ms | 0.227 ms | 3.52x |
| MatMul | 0.115 ms | 0.131 ms | 1.14x |
| Softmax | 0.114 ms | 0.240 ms | 2.11x |
| LayerNormalization | 0.061 ms | 0.135 ms | 2.22x |
Some individual cases are absurd. A 4096-size bilinear Einsum ran in 0.136 ms versus 1,396 ms, more than 10,000x faster. That's an outlier, but it shows what happens when a general runtime hits a slow path. These kernels are the primitive layer, not a full model runtime. Versioned contracts mean you can pin a kernel and swap implementations underneath. That kind of foundation is what makes in-browser inference reliable enough to ship.
175B on a laptop: the absurdity that works
The most unexpected result in this roundup is a virtual screening workflow running DeepSeek 175B on a laptop with an RTX 4060, 32GB RAM, and 8GB VRAM. The team processed 200k protein-ligand pairs across 20 targets in 72 hours, with an average binding affinity error of 0.88 kcal/mol, inside the 1.0 kcal/mol chemical accuracy threshold. They report 100x throughput of an 8-card A100 cluster baseline on the same task.
The catch: 72% of execution time goes to heterogeneous memory management. The model quantization and optimization contribute less than 10% to prediction error. So this stack works for offline scientific workloads, not interactive chat. But the barrier for entry just dropped by orders of magnitude.
Common pitfalls
Don't assume stock vLLM spans consumer cards. On RDNA 4, TP and PP fail during RCCL initialization. Use llama.cpp's ROCm backend, or wait for the fixes tracked in vLLM #40980 and ROCm rocm-systems #5480.
Don't benchmark a reasoning model with a 30-second window. Qwen3 8B shows 0.0 tok/s because its thinking phase swallows the interval. Use 120 seconds, or filter out hidden reasoning tokens.
Don't compare cross-model KV handoff without accounting for tokenizers. The paper's translator handles tokenizer differences, but a naive cache copy will corrupt attention.
Don't expect WebGPU kernels to speed up every operation the same way. Most wins come from slow-path cases. Measure on your own device, which is why Hugging Face built the Fleet benchmark suite.
Don't push a single 24GB card to maximum context. Club-3090 documents prefill OOMs past ~50K tokens on vLLM single-card. Use a dual-GPU config or llama.cpp's more conservative memory management.
One thing to remember
Every number above is a point in time. The WebGPU speedups will age. The 175B-on-laptop pipeline will get faster as memory management improves. The KV-cache translator is initial evidence, not a production tool. What won't age is the pattern: local inference is now a legitimate deployment target, not a hobbyist proof of concept.
The bottom line
- If you're building a multi-agent system where different models handle different steps, adopt cross-model KV sharing for any shared context. It cuts prefill cost by up to 67% and drops latency from 899ms to 138ms, with a small accuracy tradeoff.
- If you're constrained to consumer hardware, skip the RTX 5090 and buy two R9700s or used 3090s. Two R9700s give you 64GB for less than half the cost, and llama.cpp's multi-GPU path works today on both. If you need a single card, a 24GB 3090 running Q4 quants is the reliability sweet spot.
- One thing to watch: the RCCL/RDNA 4 blocker is a software gap, not a permanent one. Expect vLLM multi-GPU to work on these cards within months as fixes land in ROCm. When that happens, the $1,880 price point will make 27B-class models the default for local serving.