Inference Efficiency Breakthroughs for LLM Deployment

Perplexity AI's split-compute method cuts LLM inference costs by 60% by running early layers locally and offloading complex ones to cloud. CoreWeave's unified agentic AI platform claims 40% cost reduction. vLLM's Transformers backend bridges Hugging Face compatibility, now matching native speed via torch.fx-based graph rewriting and runtime fusion. ThoughtFold reduces reasoning tokens by 56% via introspective preference learning. NF-CoT uses normalizing flows for latent reasoning while preserving CoT's left-to-right generation and KV-cache compatibility. OPRD (On-Policy Representation Distillation) aligns hidden representations to eliminate sampling variance. Andrew Ng's vLLM course with Red Hat offers practical deployment knowledge. Gemma 4 QAT checkpoints reduce E2B to 1GB for mobile/laptop efficiency. Mellum2 (JetBrains) is a 12B MoE coding model with GQA, SWA, MTP, and 128K context. Databricks published a detailed post on reliable LLM inference at scale. NVIDIA Nemotron 3 Ultra also contributes to inference efficiency with its MoE architecture and million-token context. CrewAI's practical guide on agent cost optimization highlights hidden token waste from reasoning chains, context loops, and RAG inputs. AdaCodec reduces token budget for video MLLMs by 7x with 5x faster time-to-first-token. NVIDIA's Locate Anything model introduces Parallel Box Decoding, achieving 2.5x speedup for grounding systems. A new intelligence per watt paper finds 71.3% of real-world queries can be served with 100x less compute, supporting model routing. ADASPEC achieves up to 2.3x speedup over EAGLE-2 for multilingual speculative decoding. A conceptual overview of inference-time compute scaling (System 2 reasoning, CoT, tool loops) reinforces the paradigm shift. FlashMemory-DeepSeek-V4 uses Lookahead Sparse Attention to reduce KV cache to 13.5% at 500K context. End-to-End Context Compression (LCLMs) improves Pareto frontier for long-context agents. A new paper on pruning and distilling MoE into dense models shows diversity-aware scoring yields +6.3 pp. One Token per Multimodal Evidence compresses multimodal evidence into single latent tokens, achieving 3-10x savings. NVIDIA's LatentMoE paper optimizes MoE inference by compressing latent space. New: Zamba2-VL (Zyphra) hybrid Mamba2-Transformer VLMs cut time-to-first-token by ~10x. A video on hybrid attention architectures explains the shift toward KV cache efficiency. A new paper on sub-quadratic vision transformers addresses the O(n²) bottleneck. A new proposal suggests precomputing and distributing KV caches as portable artifacts to eliminate redundant prefill, achieving 9-50x compute savings for agent-heavy workloads. V-Zero reformulates OPD as negative-free stop-gradient alignment with contrastive evidence gating to remove answer labels for fine-grained visual reasoning, achieving 5-10x speedups. Causal-rCM achieves 10x faster convergence with only 1-2 sampling steps for video generation. iLLaDA validates masked diffusion LMs as a viable alternative to autoregressive models at 8B scale. CAVEWOMAN finds input compression backfires by causing longer, costlier responses while output compression delivers savings. JetSpec achieves up to 9.64x speedup on MATH-500 via parallel tree drafting that resolves the causality-efficiency dilemma. rasbt's empirical test shows 30B MoE models hit a sweet spot at 40 tok/sec on consumer hardware for coding agents, matching GPT 5.5 Pro speeds with token efficiency gaps between Claude Code (2x) and Codex. FlashMorph provides a principled method for hybrid attention layer selection, improving long-context efficiency. A new edge transformer architecture (ET) optimizes for energy-efficient edge inference, replacing computationally intensive components. A detailed deep-dive on KV cache memory architecture covers capacity, bandwidth, and allocation pressures for production serving. A large-scale NAS study (HARMONY) finds heterogeneous mixing of Transformer, MoE, and Mamba-2 outperforms homogeneous, with best config at 2.38B params, 1.0874 perplexity, 4320 tok/s. A new study finds AI agents consume 136.5x more energy than standard chatbots, with 54.5% GPU idle time, raising sustainability concerns for agentic workloads. OrbitQuant presents data-agnostic quantization for diffusion transformers using rotated basis and fixed codebooks, achieving SOTA PTQ at low-bit settings across multiple models and transferring to video without retuning. New: Carmack suggests using flash memory for inference to lower cost for large models, a pragmatic hardware hack. New papers: Nemotron-Labs-Diffusion (tri-mode LM unifying AR, diffusion, self-speculation, 6x token throughput over Qwen3-8B), HiLS Attention (hierarchical sparse attention for infinite context, 64x extrapolation, convertible from full-attention models). Cerebras achieves 2300 tok/s with Gemma 4, enabling real-time multimodal agents. vLLM's Transformers backend now matches native speed via torch.fx graph rewriting, eliminating separate model ports. Dynamo-MoE achieves 6.75× TTFT reduction over vLLM for MoE inference via dynamic parallelism switching between tensor and expert parallelism. A new comparative study of linear attention architectures (DeltaNet, Gated DeltaNet, Kimi Delta) with cross-layer routing (CLVR) provides practical guidance for efficient attention. Recent addition: Sparse Delta Memory scales linear RNN state capacity via sparse memory, addressing long-context recall gap vs softmax attention with clear isoFLOP gains. New: KronQ achieves 2-bit quantization on LLaMA-3-70B where GPTQ diverges, via Kronecker-factored Hessian. New: S-TTT offers up to 15% relative improvement on long-context benchmarks via self-guided test-time training. New: ShortOPD recovers pruned LLMs with short-to-long on-policy distillation, achieving 9x recovery over baseline. New: PalmClaw is a native on-device agent framework for mobile phones, achieving 94.9% faster completion. Recent reading: Byte-exact KV-cache grafting: frozen Gemma-4-12B jumps from 80% to 93.3% on AIME 2025 with 6574x fewer tokens and ~8700x energy savings. These developments address the scaling cost problem and enable more practical edge-cloud hybrid deployment.