Core LLM inference & training advances
Key Questions
What inference optimizations improve LLM throughput and speed?
EAGLE 3.1 merged into vLLM with speculative decoding, while NVIDIA NVFP4 delivers 1.66x gains over FP8. DiffusionGemma runs 4x faster on consumer GPUs, and DSpark achieves 60-85% speedup via confidence-scheduled speculative decoding. Nemotron-Labs-Diffusion supports 6x tokens per forward pass in tri-mode operation.
Which new quantization and compression techniques enable on-device deployment?
NanoQuant achieves sub-1-bit quantization with 25.8x compression of Llama2-70B to run on 8GB GPUs. BiSCo-LLM uses binary spherical coding for sub-2-bit weights with minimal perplexity degradation. KronQ addresses gradient covariance for stable 2-bit LLaMA-3-70B quantization.
How are RL post-training methods scaling for large models?
Ring-Zero scales zero RL to 1T parameters with emergent self-verification. GFlowRL scales distribution-matching RL to 235B MoE models, achieving Codeforces 2048 at 14B. Microsoft research explores conditional FLOP allocation to optimize RL training budgets without collapse.
What architectural innovations reduce training and inference costs?
CoFrGeNets replace transformer attention with continued fraction ladders for fewer parameters and faster training. HiLS Attention enables 64x context extrapolation with full-attention performance. M+Adam optimizer separates mantissa-exponent for low-precision training from Anima Anandkumar's group.
How do interpretability advances support agentic reasoning monitoring?
Anthropic's Jacobian lens reveals a 'global workspace' in Claude for real-time monitoring of hidden goals and conscious-access bottlenecks. This breakthrough allows viewing what models think but do not say. Self-guided test-time training further improves long-context adaptation by up to 15% on LongBench.
Inference Looping (+2.6% MMLU-Pro); ScheduleFree+ (31% faster); EAGLE 3.1 merged into vLLM; NITP (+5.7% MMLU-Pro); DiffusionGemma (4x faster on consumer GPUs); DeepSeek V4 Compressed Sparse Attention; vLLM v0.22 speculative decoding; NVIDIA NVFP4 (1.66x over FP8); ProbMoE; VibeThinker-3B matches DeepSeek V3.2; VIMPO; OpenAI Jalapeño chip; NVIDIA TwoTower diffusion LLM (2.42x throughput, 98.7% quality, 8% pretraining cost, open-source); M+Adam low-precision training optimizer (mantissa-exponent separation) from Anima Anandkumar's group — signal for reduced training costs. ReContext (long-context reasoning without retraining). dOPSD: on-policy self-distillation for diffusion language models — improves reasoning via teacher privilege from denoising trajectory, gains in math and code. NanoQuant: sub-1-bit quantization via low-rank binary factorization and ADMM, 25.8× compression of Llama2-70B to run on 8GB GPU; practical breakthrough for on-device deployment. Anthropic's 'global workspace' (J-space) in Claude — Jacobian lens reveals a conscious-access bottleneck; can see what Claude is thinking but not saying, and modulate it. Major interpretability breakthrough for real-time monitoring of agentic reasoning and hidden goals. Nemotron-Labs-Diffusion — tri-mode LM (AR, diffusion, self-speculation) from NVIDIA; 6x tokens per forward pass, 4x throughput on SPEED-Bench; self-speculation outperforms MTP; speed-of-light analysis shows 76.5% more tokens per forward pass. CommonSyn — synthetic data for diversified commonsense reasoning (ACL 2026); two-stage generation method; addresses lack of diverse commonsense training data. OmniOpt — unified taxonomy and large-scale benchmark for modern optimizers; covers 100+ methods, evaluates 24+ from 60M to 1B params; fills gap in systematic comparison for training efficiency. NVIDIA Puzzle-75B-A9B compression — joint structural search across heterogeneous MoE pruning, Mamba pruning, distillation, RL, quantization, and MTP head; doubles interactive server throughput on single 8xB200 node while holding quality on reasoning, coding, and agentic benchmarks. HiLS Attention — hierarchical sparse attention with end-to-end learned chunk selection; achieves full-attention performance while enabling 64x context extrapolation; lightweight continued pretraining conversion. NVIDIA Kimi-K2.7-Code NVFP4 — 1T param MoE quantized to NVFP4, matches INT4 on agentic/coding benchmarks (SWE-bench 74.3%, Terminal-Bench 72.5%); practical deployment optimization for Blackwell GPUs. DSpark — confidence-scheduled speculative decoding with semi-autoregressive generation; reduces waste from parallel drafters; DeepSeek production deployment with 60-85% speedup. Distributionally Robust RLHF/DPO — addresses prompt distribution shift with theoretical convergence guarantees; shows OOD improvements on reasoning tasks; practical robustness method for deployment. New: On-policy distillation for recovery after domain SFT — IF-eval 85%→45%→83%, actionable technique for maintaining post-training behavior. New: Jet-Long — dynamic bifocal RoPE for efficient long-context extension; tuning-free, 1.39x FA2 throughput, strong on RULER/HELMET-RAG; directly addresses short-context fidelity trade-off for agentic workflows. New: CoFrGeNets — IBM Research replaces transformer attention/FFN with continued fraction ladders; fewer parameters, faster training/inference, matches GPT2-xl and Llama-3.2B on GLUE/perplexity; plug-and-play with existing pipelines; early-stage but promising for efficiency. New: BiSCo-LLM — lookup-free binary spherical coding for sub-2-bit LLM weights; Qwen3-8B shows ~0.5 perplexity and ~2% accuracy degradation at extreme compression; practical for on-device/agentic deployment. New: KronQ — Kronecker-factored Hessian quantization, addresses gradient covariance gap; 2-bit LLaMA-3-70B (7.93 perplexity vs GPTQ divergence). New: Self-guided test-time training for long-context LLMs — selects evidence spans before adaptation, up to 15% relative improvement on LongBench-v2/Pro. New: Mechanistic study on memorized knowledge failing to generalize in fine-tuning — self-patching reveals routing issue, simple heuristic recovers 58-75% gap. New: GFlowRL — scaling distribution-matching RL to LLMs; removes learned partition function, uses in-batch Monte Carlo; achieves Codeforces 2048 at 14B, scales to 235B MoE. New: Ring-Zero — scaling zero RL to 1T parameters; emergent self-verification and parallel reasoning; structured CoT evaluation framework. New: Microsoft GFlowNet-style RL scaling for large reasoning models — distribution-matching property avoids RLHF collapse; practical for RL training innovations. New: Where Should RL Post-Training Compute Go? — FLOP-accounting framework and conditional allocation frontiers; challenges total FLOPs reporting; actionable for practitioners optimizing RL training budgets. New: H^2SD — Hybrid Hindsight Self-Distillation for RLVR training; modulates update based on trajectory correctness; beats OPSD/RLSD on reasoning benchmarks.