Benchmark Reliability Becomes a Deployment Bottleneck

Trial by Trace reports that 31.1% of 3,420 runs completed tasks while failing at least one harness criterion; InteractiveWorld and Trajectory2Gym expose additional long-horizon and training-trace weaknesses. TauIndianBankBench finds outcome-only scoring can mask skipped steps or unrequested changes in 6.5–12.0% of sessions. Momentic’s QA-agent comparisons add practical cost and precision tradeoffs, but vendor claims and transfer of newer evaluations still need independent validation.