LLM reasoning & evaluation bottlenecks
Key Questions
What does the new economic benchmark evaluate?
The benchmark covers 1500+ tasks across 55 occupations, providing a broader evaluation than narrow academic tests for LLM capabilities.
What new benchmarks assess life sciences and clinical AI?
LifeSciBench includes 750 tasks for life science research with expert rubrics, while BRIDGE offers a multilingual clinical benchmark from Mass General Brigham.
How does inference compute affect LLM evaluation?
Inference compute shapes frontier LLM evaluation by revealing plateauing trends across benchmarks, challenging simplistic model comparisons.
New economic benchmark covering 1500+ tasks across 55 occupations challenges narrow academic tests. MLPerf Mobile v6.0 introduces GenAI benchmarks for on-device LLM inference (Llama 3.1/3.2). Ethan Mollick pushes back on Nature headline framing AI math test failure. On the Limits of LLM Adaptability (2/3 zero-shot errors resist prompt correction). VSTAT benchmark. FrontierCode benchmark. Gemini 3.5 Pro with Deep Think reasoning. Hedge-Bench 1.0. New: LifeSciBench (750-task life science benchmark, best model ~36%). New: BRIDGE multilingual clinical benchmark from Mass General Brigham. New: Inference compute shapes evaluation — plateauing discussion challenges simplistic model comparisons.