Reproducibility gap in AI-assisted mathematics
Key Questions
What performance did Leanstral 1.5 achieve on math benchmarks?
Open-source Leanstral 1.5 reached 99.8% on miniF2F and solved 5 out of 6 Putnam problems at a cost of $0.20 per problem.
What new benchmarks reveal gaps in AI-assisted mathematics?
MA-ProofBench, ComBench, DynaMath 2026, and MIRA-Math expose reproducibility and verification gaps in current AI math systems.
How does the Math-Graph dataset support formal mathematics?
Math-Graph bridges arXiv papers with Lean 4 formalizations, enabling better theorem dependency tracking and dataset validation.
What issues were found in existing Lean benchmarks?
A critical audit identified dataset defects in Lean benchmarks that affect reliability of reported AI performance metrics.
What framework did Terence Tao propose for AI math agents?
Tao proposed a calibration framework after AI agents ported his Java applets and identified bugs in the original code.
OpenAI's Astra solved ten open problems with Lean 4 certificates, providing a strong example of reproducible AI-assisted proof. Open-source Leanstral 1.5 achieves 99.8% miniF2F, 5/6 Putnam at $0.20/problem. New benchmarks (MA-ProofBench, ComBench, DynaMath 2026, MIRA-Math) reveal gaps. FrontierMath expands Open Problems benchmark to ~50 unsolved problems; AI solves 3 (6%), solving rate growing from near zero to 30-40% in 18 months. Critical audit of Lean benchmarks shows dataset defects. Math-Graph dataset bridges arXiv and Lean 4. AI agents ported Terence Tao's Java applets, finding bugs; Tao proposes calibration framework. Benchmarking Agents for Proving Theorems in Quantum (Lean 4) shows +15.9 points improvement. Fields Medalist Terence Tao warns at ICM 2026 about AI-induced 'proof indigestion'. URSA multimodal process reward model outperforms GPT-4o in math reasoning. Best Datasets for Math 2026 roundup (FormalMATH, CriticLeanBench from 2077AI) provides practical reference.