Scaling Laws: Efficiency, Long-Context, and New Architectures
Key Questions
What is NVIDIA Nemotron 3 Ultra and what efficiency gains does it offer?
NVIDIA Nemotron 3 Ultra is a 550B MoE model with 55B active parameters using hybrid Mamba-Transformer architecture, NVFP4 quantization, LatentMoE, multi-token prediction, and Multi-Teacher On-Policy Distillation. It delivers 30% cost savings compared to prior models.
What is Flash-WAM and how does it improve world action models?
Flash-WAM uses modality-aware distillation to achieve a 23x speedup for world action models. It was introduced as a new advance in the highlight.
How does LLMCodec apply video compression techniques?
LLMCodec uses video compression methods to shrink giant LLM weights. This enables more efficient model storage and deployment.
What are the key features of Gemma 4 QAT models?
Gemma 4 QAT models focus on mobile and laptop efficiency, with the E2B variant fitting in 1GB. They optimize quantization-aware training for on-device performance.
What does Imagine Before You Predict introduce for video reasoning?
It proposes interleaved latent visual reasoning that improves performance by +24.4 on FutureBench. The method interleaves visual imagination before prediction steps.
How does AdaCodec benefit video MLLMs?
AdaCodec provides a predictive visual code that allows video MLLMs to operate at 1/7 the usual budget while beating baselines. It reduces computational requirements for multimodal video tasks.
What is Video2LoRA and its token efficiency gain?
Video2LoRA is a hypernetwork that generates LoRA adapters directly from video input. It achieves a 1500x reduction in tokens for vision-language models.
Which paper won best paper at CVPR 2026?
DeepMind's D4RT won best paper at CVPR 2026 for advances in diffusion and representation learning. Oxford VGG also achieved back-to-back wins in the conference awards.
NVIDIA Nemotron 3 Ultra (550B MoE, 55B active) with hybrid Mamba-Transformer, NVFP4 quantization, LatentMoE, multi-token prediction, and Multi-Teacher On-Policy Distillation; 30% cost savings. Other advances: Echo-Infinity, AAD-1, ThoughtFold, BenchEvolver, NF-CoT, OPRD, LoomVideo. New today: Flash-WAM (modality-aware distillation, 23x speedup for world action models); LLMCodec (video compression for LLM weights); Gemma 4 QAT (mobile/laptop efficiency, 1GB E2B); Imagine Before You Predict (interleaved latent visual reasoning, +24.4 on FutureBench); AdaCodec (predictive visual code for video MLLMs, 1/7 budget beats baselines); Video2LoRA (hypernetwork generates LoRA from video, 1500x token reduction). Also: LightReasoner, CPT, RT-Lynx, MobileMoE, scale vectors, LLaVA-OneVision-2, IndexMem, RTDMD, On-Policy Adversarial Flow Distillation, OSP-Next, NEO-ov, Gamma-World, ZAYA1-8B, Thinking Before Constraining, GASP, Why Larger Models Learn More, GPIC, minWM, YoCausal, Trajectory, Graphon, Cosmos 3, JEPA, dMoE, VLM3, LongTraceRL, Light Interaction, StateKV, Representation Forcing, NITP, VideoMLA, VLMs as teachers, World Models Meet Language Models, TrOPD, MERIT, VaSE, GPU kernel surrogate, Decoupled Residual Denoising Diffusion. CVPR 2026: DeepMind's D4RT wins best paper, advancing diffusion/representation learning.