Core ML Advances: Mechanistic Interpretability and Model Efficiency
Key Questions
What is the Knowing-Using Gap in LLM finetuning?
The gap measures differences between what models know and apply after finetuning. Self-patching techniques recover 58-75% of performance in these scenarios.
What new quantization method was introduced?
KronQ enables effective 2-bit quantization for models. It aims to improve efficiency without major capability loss.
How does Partition-Prompt-Aggregate expose LLM limitations?
It reveals that LLMs fail to follow the law of total probability in reasoning tasks. This highlights gaps in probabilistic understanding.
What tools support better agent observability?
AgentDebugX provides an open-source toolkit for failure observability, attribution, and recovery in LLM agents. It aids debugging of complex agent behaviors.
What optimization stack was proposed for RLVR?
ISO offers an RLVR-native optimization stack to advance reasoning capabilities. It targets verifiable reward-based reinforcement learning.
What benchmark evaluates factual completeness?
GAMUT introduces two-level meta-rubrics for assessing open-ended generation. It measures factual completeness in model outputs.
What application of mechanistic interpretability was shown?
It was applied to materials science for reading and steering model representations of mechanisms. This uses open-weight language models for domain-specific insights.
What other new papers address RL and training issues?
Papers include Stale but Stable for async RL trust regions, H^2SD for hybrid hindsight self-distillation, and studies on on-policy distillation pathologies. Nvidia Vera Rubin focuses on performance per watt.
Knowing-Using Gap in LLM finetuning; self-patching recovers 58-75%. KronQ for 2-bit quantization. Self-guided test-time training for long-context LLMs. Partition-Prompt-Aggregate reveals LLMs fail law of total probability. DeepLoop formalizes tied-depth residual scaling. Demystifying On-Policy Distillation identifies pathologies. Google Cloud reduces model upgrade times. Understanding Reasoning from Pretraining to Post-Training. RAGU, DSWorld, Cura 1T, RecGPT-V3. Nvidia Vera Rubin. New papers: Stale but Stable (staleness-adaptive trust regions for async RL), ISO (RLVR-native optimization stack), AgentDebugX (toolkit for agent failure observability), H^2SD (hybrid hindsight self-distillation), GAMUT (benchmark for factual completeness), ConsiSpace (video spatial reasoning). New: Mechanistic interpretability applied to materials science – reading and steering representations of materials science mechanisms in an open-weight language model.