Deep Learning Theory Maturing
Key Questions
What key topics are covered in maturing deep learning theory?
Topics include phase transitions in loss landscapes, contrastive Gaussian representations, and a 'Scientific Theory of DL'. Additional areas are Trees to Flows unifying decision trees and diffusion models.
What is Trees to Flows?
Trees to Flows unifies decision trees and diffusion models. It is a paper available for discussion, reinforcing theoretical foundations.
What are SAEs and their role in LLMs?
Sparse Autoencoders (SAEs) map low-dimensional manifolds in LLMs, as discussed in 'How SAEs Map Concept Manifolds in LLMs'. This aids in understanding model representations.
What is Orbit-Space Particle Flow Matching?
It is a generative modeling technique featured in a paper for discussion. It contributes to advancing diffusion model theory.
What low-power AI development is mentioned?
Flourish enables sparse async processing at 20W, as in the work by Thomas Reardon, creator of Internet Explorer, aiming to teach AI to think efficiently.
What seminars or talks are highlighted?
Stony Brook theoretical ML, Stanford JEPA to causal LOWER seminar reposted by @ylecun, and 'The Geometry of Smarter AI' podcast. Luca Ambrogioni's talk covers physics and information theory of generative diffusion.
What is the status of deep learning theory maturation?
The status is 'developing', with reinforcing foundations through works like ComboStoc on diffusion complexity and Geometric PIML.
How do Transformers factor into the theory?
Discussions cover Transformers' succinctness and expressivity. This ties into broader themes like Orbit-Space Particle Flow Matching.
Phase transitions/loss landscapes; contrastive Gaussian reps; 'Scientific Theory of DL'; Trees to Flows unifies trees/diffusion; Flourish sparse async 20W; Geometric PIML; SAEs low-dim manifolds; Transformers succinctness expressivity; Orbit-Space Particle Flow Matching; ComboStoc diffusion complexity. Stony Brook theo ML; Stanford JEPA to causal LOWER. Reinforcing foundations.