Core ML Architecture and Scaling Advances
Key Questions
What is xHC and how does it improve scaling?
xHC (Expanded Hyper-Connections) overcomes scaling ceilings with sparse stream updates, delivering a 4-point improvement at 18B scale.
What insights does the new reasoning paper provide on scaling laws?
Pavel Izmailov's paper examines scaling laws from pretraining to post-training, yielding key architectural insights for LLMs.
How does Distilled Reinforcement Learning benefit LLM post-training?
It combines RL and distillation to improve credit assignment and close the teacher-student gap, producing strong cross-family results.
xHC (Expanded Hyper-Connections) overcomes scaling ceiling with sparse stream updates, achieving 4-point improvement at 18B scale. A new reasoning paper from Pavel Izmailov studies scaling laws from pretraining to post-training. VideoRAE achieves SOTA video generation by repurposing frozen foundation models. Distilled Reinforcement Learning for LLM Post-training combines RL and distillation to improve credit assignment and teacher-student gap, showing strong cross-family results. These represent significant architectural and scaling insights for LLMs and generative models.