Inference Opts: Quant/Arch + New Models + Scaling + Cost Discipline
Key Questions
What efficiency gains does JetSpec speculative decoding provide?
JetSpec delivers a 9.64x speedup in inference through speculative decoding techniques.
How does MiniMax Sparse Attention reduce compute requirements?
MiniMax Sparse Attention cuts attention compute to 1/20th of standard methods while maintaining performance.
What is notable about NVIDIA GB300 in MLPerf 6.0?
NVIDIA GB300 achieved the top results in MLPerf 6.0 benchmarks for inference performance.
What funding did Etched raise for its inference chip?
Etched raised $800M and exited stealth with a working Sohu inference chip, secured TSMC partnership, and over $1B in customer contracts.
How has OpenAI reduced inference costs?
OpenAI achieved a 50% reduction in inference costs through software-only optimizations, lowering GPU requirements for guest-tier workloads to hundreds of units.
Why is memory becoming the new bottleneck in AI inference?
Noam Brown's keynote highlighted that inference-time scaling shifts the hardware race toward memory capacity and cost, elevating suppliers like Samsung and SK hynix.
What is Tencent Hy3 and its benchmark performance?
Tencent released the 295B-A21B Hy3 model on Hugging Face, scoring 71.7 on TerminalBench-2.1 and 28 on DeepSWE while fitting 2x Spark configurations.
What practical guides address KV cache and inference disaggregation?
New guides cover KV cache architecture, memory allocation, paging, and compression, plus decision rules for prefill-heavy workloads and dynamic rate matching in disaggregated inference.
Users shifting from 'tokenmaxxing' to efficiency. JetSpec speculative decoding 9.64x speedup. MiniMax Sparse Attention (1/20 attention compute). LCLMs (16 tokens as 1). NVIDIA GB300 tops MLPerf 6.0. Baseten raises $1.5B. SubQ claims 12M context/1000x compute reduction (unverified). VibeThinker-3B rivals Opus 4.5 via RL. Meta plans to sell excess AI compute. Together AI raises $800M with $1.15B bookings, ATLAS adaptive speculative decoding 500 tok/s on DeepSeek-V3.1, 500MW capacity commitment. Practical inference engineering guide published. MTP technique for up to 3x speedup. NVIDIA TwoTower diffusion LLM 2.42x throughput, open weights. Runpod Overdrive up to 2.45x throughput. ELDR routing for MoE. OpenAI collaborates with Broadcom on 'Jalapeño' chip. EAGLE-3 speculative decoding explained. OpenAI halves inference costs with software-only optimizations, dropping guest-tier GPU count to hundreds. Etched raises $800M for Sohu inference chip (working, TSMC partnership, $1B+ customer contracts). HOLA paper achieves 16x training length extrapolation on RULER via compressive recurrent state + small exact cache for linear attention. Runway's GPU capacity controller using queueing theory to reallocate idle research GPUs to production overnight is a practical pattern. Noam Brown keynote reframes hardware race—inference-time scaling makes memory the new bottleneck, making memory suppliers (Samsung/SK hynix) strategic. DSpark from DeepSeek achieves 60-85% speedup with zero quality loss via Markov head and suffix decay; open-source DeepSpec code. Practical inference disaggregation guide with concrete decision rules (prefill-heavy, >10B models, dynamic rate matching). LOCOS (Logit-Contribution Scoring) identifies non-literal retrieval heads; ablation shows ROUGE-L collapse from 0.401 to 0.000 on Qwen3-8B, key for long-context interpretability. A deep practical guide on KV cache architecture covering memory allocation, paging, fragmentation, and compression trade-offs—essential for production inference decisions. A practical guide on scaling laws for inference cost decisions—Chinchilla vs Kaplan comparison and model selection framework (7B for simple tasks, 30-70B for complex reasoning) directly actionable for production planning. New: FlashMorph paper from Fudan/ByteDance/CUHK proposes converting Transformers to hybrid attention models via budget-constrained layer selection, outperforming heuristic methods on Qwen3 backbones—practical for long-context optimization. New: @EliasEskin to present 'Routing with Generated Data' at ACL2026, signaling ongoing work in routing techniques for inference. New: DigitalOcean practical comparison of non-GPU inference hardware (Inferentia2, TPU, Groq, Tenstorrent) provides actionable tradeoffs for production LLM serving. New: A practical primer on AI power efficiency metrics (PUE, TFLOPS/W, tokens/W) from a non-expert source but good reference for infrastructure cost analysis. New: Tencent Hy3 (295B-A21B) released on HuggingFace with TerminalBench-2.1 71.7 and DeepSWE 28, fitting 2x Spark configs—concrete model release for hardware-specific audience.