DeepSeek V4 MoE LLM Release
Key Questions
What are the main technical innovations in DeepSeek V4?
DeepSeek V4 is a 1.6T parameter MoE model with 49B active parameters, featuring hybrid attention mechanisms that achieve over 90% KV cache reduction and 9.5x cache efficiency. It introduces dMoE block-level routing for diffusion models and supports chiplet-based scaling.
How does ProbMoE improve upon traditional MoE routing?
ProbMoE replaces hard top-k routing with probabilistic inference over expert subsets, addressing limitations of deterministic selection as noted in community discussions. This approach enables more flexible and potentially accurate expert activation in large models.
What does Noam Shazeer's departure from Google to OpenAI indicate?
Noam Shazeer, co-author of the Transformer and MoE architect, moving to OpenAI signals a notable talent shift in the AI industry. It highlights ongoing competition for expertise in scaling efficient architectures like MoE.
1.6T params (49B active), hybrid attn/mHC/HCA/CSA 90%+ KV cut and 9.5x cache reduction; Gemma 4 27B MoE 256K multimodal; chiplet MoE; SANA-WM hybrid linear diffusion transformer. New: dMoE block-level routing for diffusion MoE (69.5→14.6 experts), MiniMax M3 with MSA sparse attention (1M context, 9x prefill, 15x decoding, open-source), StateKV linear-time video prefill, NITP next implicit token prediction (5.7% MMLU-Pro gain, ICML 2026), Draft-OPD on-policy distillation for speculative decoding (5x acceleration), Scaling of PEFT for million personal models. Also Confidence-Adaptive SwiGLU, LongAttnComp, NVIDIA Nemotron 3 Ultra (hybrid Mamba-Transformer, LatentMoE, MOPD). Apple's third-gen foundation models introduce sparse architecture with Instruction-Following Pruning and PT-MoE. Google's DiffusionGemma (26B MoE) uses text diffusion for 4x faster generation. New: Redesign MoE routers with manifold power iteration. New: ProbMoE replaces hard top-k routing with probabilistic inference over expert subsets. Community coverage includes video analysis of DiffusionGemma. Noam Shazeer (Transformer co-author, MoE architect) leaves Google for OpenAI, signaling talent shift.