LLM Benchmark Watch

Key multi-angle insights on advancing research agents:

• NanoKnow benchmark tests LLM parametric knowledge in retrieval contexts
• Fast Thinking...

Nvidia's GPU advances slash AI bottlenecks for agents:

• Vera Rubin (H2 2026): 10x perf/watt vs Blackwell, 10x cheaper inference tokens, 4x fewer GPUs...

Practical takeaways from months of Opus agent work:

• Culmination of experiments with Opus 4.5 agents and beyond
• Many hard-won learnings uncovered
• New optimization trick shows promise
• Bonus provocation for Rust engineers

Enterprise shift to SLMs: AT&T cut AI costs 90% by swapping large models for small ones, boosting speed, latency, and 3x token processing.

• Agentic...

Anthropic's Claude faces deployment hurdles on two fronts—ethical refusals and technical gaps.

• Pentagon Rejection: Refuses unrestricted military...

RL post-training debates intensify: Key question raised—does LLM RL post-training need to be on-policy?

• Open collaboration sought: Researchers...

New technique for real-world LLM deployments:

• Compositional steering tokens embed natural language behaviors into input tokens via...

RO-FIN-LLM introduces a benchmark using LLM-as-a-Judge and human evaluation.

Key focus:

• Assesses models' capacity for complex algorithmic logic and...

New competitive eval for LLMs:

• Unsaturable LLM Benchmark developed to rate skill, reliability, and metacognition
• Targets internal reasoning...

Google's February 2026 Gemini Drop packs major capability boosts:

• Lyria 3 (beta): Generates 30-sec soundtracks from text or images
• Gemini 3.1...

OmniGAIA bridges gaps in multi-modal LLM reasoning across video, audio, and language:

• 360-task benchmark demands complex tool execution and...

Key upgrade: Enhanced reasoning for science, research, engineering, processing messy/incomplete data for actionable insights.

• Real-world apps:...

Breakthrough module gives LLMs real memory for the first time, enabling instant pattern retrieval over repeated recomputation.

• Efficiency surge:...

Real-world LLM impact: Netflix post-trains Llama 3.1-8B models using SFT and DPO to pick the best artwork for each user, tackling visual preference...

Over half of common AI benchmarks are contaminated, casting serious doubt on headline model rankings and marketing claims per new NE2NE study.

Open-Source LLM Builders Summit Highlights

• 🔥 Olmo 3: Olmo 3 advances the state-of-the-art for fully open models, presented at the 2nd...

Key projects from the 2nd Open-Source LLM Builders Summit highlight open-source momentum:

• Olmo 3 (Ai2) advances SOTA for fully open models
  -...

Breakthrough for PDF/scanned doc workloads: Matches accuracy of models 1000× larger while finetuning/deploying on a single 24GB GPU—handles up to 400k...

Qwen 3 pushes multilingual reasoning, coding, and instruction-following to new levels.

• Efficiency focus: Smarter architectures and training over...

ARLArena launches as a unified framework and testbed tackling instability in Agentic RL with LLMs:

• Identifies sparse rewards and non-stationary...