Agent safety, evals & runtime / self-evolution
Key Questions
What percentage of production agents currently pass security evaluations?
Only 11% of production agents pass the security bar according to recent benchmarks. This highlights a significant gap in agent safety readiness.
What is OWASP Top 10 for Agentic Applications 2026?
It is an updated security framework outlining the top risks specific to agentic AI systems. It provides guidance for enterprises building secure multi-agent applications.
How effective is Vera at testing agent safety?
Vera's systematic testing infrastructure achieved a 93.9% attack success rate across four major frameworks using a combinatorial taxonomy. It supports evidence-grounded verification of safety issues.
What does AgentGym2 reveal about current frontier models?
AgentGym2 shows GPT-5 and Claude Sonnet 4.5 fail at roughly 44% and 37% on tasks requiring tool discovery, noise robustness, and end-to-end procedures. This quantifies the production readiness gap.
What is Reward-Swap Policy Optimization (RSPO) designed to address?
RSPO swaps dense process rewards and sparse outcome rewards in multi-turn agents to reduce misalignment. It maintains trajectory diversity and shows consistent gains on WebShop and ALFWorld.
What bottlenecks does Lilian Weng identify for self-improving AI agents?
Seven key bottlenecks are outlined, including weak evaluators and reward hacking. The essay grounds self-improvement in practical harness engineering and scaffolding.
How does SWARM support multi-agent safety in production?
SWARM is an open-source framework providing soft probabilistic labels, adverse selection metrics, and governance levers. It offers practical tools for runtime safety in multi-agent deployments.
What new benchmark evaluates contextual authorization in LLM agents?
The Contextual Authorization Benchmark tests agents on permission-aware decision making in dynamic environments. It helps measure trustworthiness beyond static security checks.
Only 11% of production agents pass security bar; SABER, Workflow-GYM, iOSWorld benchmarks. OWASP Top 10 for Agentic Applications 2026. AI Judges on Trial survey. Contextual Authorization Benchmark. Securing the AI Agent framework. First real-world agentic ransomware (JadePuffer) — LLM autonomously runs attacks in <31s. RedCoder (multi-turn red teaming for code security). Vera: systematic safety testing infrastructure with combinatorial taxonomy, 93.9% attack success rate on four major frameworks, evidence-grounded verification. Epistemic state representations in LLMs — low-dimensional confidence/self-vs-world geometry, confabulation detector (0.89-0.92 AUC), steering/patching results. LLM-as-a-Verifier: probabilistic logit scoring for verification, SOTA 78.2% SWE-Bench, 86.5% Terminal-Bench, training-free. Governance Frameworks for Multi-Agent Systems: observability, control, trust — practical runtime governance guidance. SWARM open-source multi-agent safety framework with soft probabilistic labels, adverse selection metrics, and governance levers — practical tool for multi-agent deployment safety. AgentGym2 benchmark reveals even GPT-5 and Claude Sonnet 4.5 fail at ~44% and ~37% on tasks requiring tool discovery, noise robustness, end-to-end procedures — quantifies production readiness gap. Reward-Swap Policy Optimization for multi-turn agents swaps dense process rewards and sparse outcome rewards to avoid misalignment while maintaining trajectory diversity, tested on WebShop/ALFWorld with consistent gains across GRPO/PPO/GiGPO. Lilian Weng's harness engineering essay grounds self-improvement in scaffolding, identifying seven bottlenecks (weak evaluators, reward hacking) — actionable framing for practitioners.