Core ML methods & efficiency
Key Questions
What performance does KronQ achieve for quantized LLMs?
KronQ uses Kronecker-factored Hessian quantization to reach 2-bit LLaMA-3-70B with 7.93 perplexity, avoiding the divergence seen in GPTQ. It offers efficient compression for large models.
How does ShortOPD recover pruned LLMs?
ShortOPD applies short-to-long on-policy distillation to recover pruned models for free-form generation. It achieves 9x recovery over baselines while using 71% fewer tokens, aiding compressed model deployment.
What advantage does visual pretraining offer over text-only methods?
Scalable Visual Pretraining on documents outperforms text-only pretraining on identical corpora for language intelligence tasks. It demonstrates the value of cross-modal approaches in core ML efficiency.
New highlight covering practical ML techniques for industry. KronQ — Kronecker-factored Hessian quantization achieves 2-bit LLaMA-3-70B with 7.93 perplexity vs GPTQ divergence. Video generation models (GenCeption) as general-purpose vision learners, SOTA on depth/normals/segmentation with far less data. Scalable Visual Pretraining for Language Intelligence — visual pretraining on documents outperforms text-only on same corpora. Self-Guided Test-Time Training (S-TTT) for long-context LLMs, up to 15% improvement with low overhead. ReChannel repurposes text-to-image DiT for dense prediction (matting, depth, segmentation) with 33K parameters. Knowing-Using Gap in LLM fine-tuning: self-patching recovers 58-75% of generalization failure. New from today's reading: ShortOPD recovers pruned LLMs for free-form generation via short-to-long on-policy distillation, achieving 9x recovery over baseline with 71% fewer tokens — practical for compressed model deployment.