Reproducibility gap in AI-assisted mathematics
Key Questions
What issues were identified in the FrontierMath audit for AI math models?
The audit revealed approximately 33% fatal errors in AI-generated solutions. This highlights ongoing reproducibility challenges in AI-assisted mathematics despite high reported performance on benchmarks.
What is Leanstral 1.5 and how does it advance formal proof verification?
Leanstral 1.5 is an open-source model from Mistral that achieves 99.8% on miniF2F, solves 5 out of 6 Putnam problems, and supports software verification tasks at low cost. It democratizes access to formal verification tools in Lean 4.
How does the KANDy tool from Clarkson researchers support mathematical discovery?
KANDy uses Kolmogorov-Arnold Networks to recover governing equations from data, including chaotic PDEs and topological structures like the Hopf Fibration. The open-source release aims to improve reproducibility in equation discovery from complex systems.
What new benchmarks address multimodal and dynamic mathematical reasoning?
DynaMath Benchmark 2026 evaluates dynamic multimodal reasoning, where only Qwen3.6-27B has scored 85.6% so far. Additional benchmarks like ASyMOB, TheoremBench, and Benchmarking Multimodal Mathematical Reasoning expand evaluation coverage.
What concerns exist about dataset defects in Lean theorem proving evaluations?
Research on 'Dataset Defects and Evaluation Failures in Lean Theorem Proving' points to faults in formal benchmarking datasets that undermine reliability. These issues contribute to the broader reproducibility gap in AI-assisted mathematics.
Neuro-symbolic PDE solvers, WaveLiT, GeoMathCode, non-Markovian diffusion sampling, Meta AI ATLAS (46k+ Lean 4 declarations, 92.7% proved), Gemini 2.5 Pro on AIME 2025 and FrontierMath (under audit), Terence Tao on AI proof-checkers, LeJEPA formal proof, Rust verification in Lean 4, CPOG proof framework, LEAP agentic prover, MPI MIS benchmark (100 questions, 2 unsolved), generalization theory for multi-task operator learning, unified framework for deep representation learning, ERBench for equation discovery, TheoremBench, Seephys Pro, ComBench, MA-ProofBench, multi-step FOL deduction. FrontierMath audit reveals ~33% fatal errors. First Proof v2.0 results released. New AI reasoning: ELM and SuCo. TorchLean formalizes neural networks in Lean 4. New: ASyMOB dataset (35,368 problems) and Diffusion-Proof recipe. New: Benchmarking Multimodal Mathematical Reasoning adds to evaluation stack. Also: Certified World Models paper provides provable guarantees via equivariance, a significant applied math contribution with AI relevance. New: Open Source Leanstral 1.5 tops formal proof verification — 99.8% on miniF2F, 5/6 Putnam problems, and software verification case study (buffer overflow detection) at $0.20 per problem; democratizes formal verification. New: From Natural Language to Certified Geometry Proofs — converts NL geometry problems into certified proofs, bridging human-readable math and machine-checkable proofs. New: DynaMath Benchmark 2026 — dynamic multimodal math reasoning benchmark; only Qwen3.6-27B scored (85.6%) so far; adds to evaluation stack. New: Clarkson Researchers develop KANDy, an AI tool using Kolmogorov-Arnold Networks to discover governing equations from data, tested on chaotic PDEs and recovering Hopf Fibration topology; open-source release enhances reproducibility. New: A critical audit of Lean theorem proving benchmarks reveals dataset defects and evaluation failures, challenging the reliability of reported success rates and underscoring the need for rigorous meta-evaluation.