Observability & evals boom — dynamic benches, meta-harnesses, runtime graders
Key Questions
What new evaluation frameworks support agent tool use and observability?
Frameworks include MCP-Use for tool-use quality, DiscoBench for clarification-aware search, PACE for low-cost proxy benchmarks, AgenticDataBench across 15 verticals, and AgenticSTS for bounded-memory testing.
How are production eval patterns evolving for agents?
Patterns include shadow mode CI to detect prompt regression, tool misuse, and schema drift, plus trace-driven development using deterministic metrics combined with LLM judges. OTel-based spans and tools like Honeycomb Agent Timeline and Arize Phoenix enable real-time tracing.
What benchmarks address induction reasoning and OOD failures?
Muse Spark 1.1 targets induction reasoning and outperforms models like Opus and Gemini on finite model theory tasks. Coverage Illusion paper and MemSyco-Bench examine sycophancy and out-of-distribution issues in agent memory.
New eval frameworks: MCP-Use (deepeval for tool-use quality), DiscoBench (clarification-aware search), PACE (proxy benchmarks at <1% cost), AgenticDataBench (15 verticals), AgenticSTS (bounded-memory, 0% frontier LLM win rate), Harness Engineering (tool contracts/validators). New benchmarks: CEO-Bench, PowerAgentBench-SS, AutoGen-TraceKit, Cross-lingual BrowseComp-Plus, Muse Spark 1.1 (induction reasoning, ICML '26). Observability: Honeycomb Agent Timeline, OTel-based spans, Agent Control Plane, Arize Phoenix (real-time tracing, evals, OTel). Production eval patterns: shadow mode CI (prompt regression, tool misuse, schema drift), trace-driven development, scoring agents with deterministic metrics + LLM judge. Coverage Illusion paper challenges HyDE. MemSyco-Bench for sycophancy in agent memory. Nouha Dziri on OOD failures and jagged intelligence. New practical walkthrough: tracing local LLM agent with Strands + Ollama + OTel (model choice dominates tool-calling reliability, two-pass structured output for small models).