Mamba series & SSM-hybrid resurgence
Key Questions
What is SDM and how does it improve linear RNNs?
SDM (Sparse Delta Memory) extends Gated DeltaNet with explicit sparse memory, achieving higher capacity under isoFLOP for efficient long-context recall.
What new techniques were introduced for KV cache management?
Techniques include KV Packet, Entropy-Guided KV, KVLink, KV Summarization, and Prefill-as-a-Service, alongside Nemotron 3 Super 120B MoE and JetSpec.
What does the comparative study of linear attention architectures cover?
It examines DeltaNet, Gated DeltaNet, Kimi Delta and similar models with cross-layer routing (CLVR), providing insights on training throughput and loss.
SDM (Sparse Delta Memory) extends Gated DeltaNet with explicit sparse memory, achieving higher capacity under isoFLOP — key signal for efficient long-context recall. Also KV Packet, Entropy-Guided KV, KVLink, KV Summarization, Prefill-as-a-Service, Nemotron 3 Super 120B MoE, JetSpec, HOLA. New resource: comparative study of linear attention architectures (DeltaNet, Gated DeltaNet, Kimi Delta, etc.) with cross-layer routing (CLVR) — practical insights for training throughput and loss.