Agent safety, evals & runtime / self-evolution

Only 11% of production agents pass security bar; SABER, Workflow-GYM, iOSWorld benchmarks. OWASP Top 10 for Agentic Applications 2026. AI Judges on Trial survey. Contextual Authorization Benchmark. Securing the AI Agent framework. First real-world agentic ransomware (JadePuffer) — LLM autonomously runs attacks in <31s. RedCoder (multi-turn red teaming for code security). Vera: systematic safety testing infrastructure with combinatorial taxonomy, 93.9% attack success rate on four major frameworks, evidence-grounded verification. Epistemic state representations in LLMs — low-dimensional confidence/self-vs-world geometry, confabulation detector (0.89-0.92 AUC), steering/patching results. LLM-as-a-Verifier: probabilistic logit scoring for verification, SOTA 78.2% SWE-Bench, 86.5% Terminal-Bench, training-free. Governance Frameworks for Multi-Agent Systems: observability, control, trust — practical runtime governance guidance. SWARM open-source multi-agent safety framework with soft probabilistic labels, adverse selection metrics, and governance levers. AgentGym2 benchmark reveals even GPT-5 and Claude Sonnet 4.5 fail at ~44% and ~37% on tasks requiring tool discovery, noise robustness, end-to-end procedures — quantifies production readiness gap. Reward-Swap Policy Optimization for multi-turn agents swaps dense process rewards and sparse outcome rewards to avoid misalignment while maintaining trajectory diversity, tested on WebShop/ALFWorld with consistent gains across GRPO/PPO/GiGPO. Safety Game — inference-time alignment for black-box LLMs; two-player zero-sum formulation for safety-helpfulness trade-off; no model internals needed, practical for third-party safety enforcement. IGRPO — information gain-based rollout policy optimization for tree-structured search in LLM agents; principled rollout budget allocation; consistent gains on 7 QA benchmarks. Codenotary AgentMon 3 — adaptive runtime security policies that learn from agent behavior; 5M daily interactions secured, 80% policy maintenance reduction; immutable audit trail, resilience to prompt obfuscation. Strike48 survey — 84% of enterprises believe agents should handle L1 tasks, but only 36% have deployed; trust gap remains the bottleneck, not capability. PROBE benchmark — measures proactive problem solving in LLM agents; even GPT-5 and Claude Opus-4.1 only achieve 40% end-to-end success; highlights fundamental limitation in current agentic systems. New: Shared API keys — 69% of enterprises share credentials, single compromise cascade, identity scoping and runtime enforcement are immediate priorities. New: HalluSquatting — new attack vector exploiting AI hallucinations to create botnets, 85-100% success rates, tested against Cursor, Copilot, Gemini CLI; reinforces that agentic capabilities introduce new attack surfaces. New: UniClawBench — capability-driven benchmark for proactive agents using live Docker containers and step-by-step checkpoints; five-capability taxonomy (Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, Cross-Platform Coordination); closed-loop evaluation with multiple agents. New: Evaluation gap survey — 50% of enterprises have had customer-facing failures after passing internal evals; repeatability as metric, autonomy by risk recommendations. New: Sixtyfour's rigorous eval stack for research agents — grades every build against expert-curated questions, only ships improvements that improve score; provenance and compounding knowledge graph. New: SWARM RL training lessons — concrete insights from SWARM's own run: reward noise, non-monotonic improvement, async coordination costs, distribution reporting; translates into adaptive evaluation windows, async governance tiers, and proxy recalibration. New: Cross-family fact-checking (e.g., Claude monitor + GPT-4.1 fact-checker) cuts adversarial CoT violations by 45% vs 6% for same-family — cheap, practical robustness lever (DeepMind). New: Long-Horizon-Terminal-Bench — 46 long-horizon terminal tasks with dense rewards, 9.9M tokens per run, strongest model only 15.2% pass@1; addresses sparse reward gap in agent evaluation. New: TRACE — capability-targeted agentic training system from Stanford; contrastive capability analysis + targeted synthetic envs + MoE composition; 27B model beats GPT-5.2-Codex on SWE-bench. New: Cynative — open-source security research agent with read-only enforcement by default, runtime-resolved action classification from cloud provider APIs, verifier refutes findings without writing. New: SingGuard-NSFA — open-source security guardrail from Ant Group; 185 threat scenarios, 100K test samples, 0.8B model rivaling 8B, 50ms latency; addresses prompt injection and tool misuse. New: Duke University hardware safety gate for LLM-driven trapped-ion control — token-based authorization + MCP server pattern; metacognitive limitations (knowing when to reframe) more critical than domain knowledge. New: PhD thesis on RL-based security analysis (SafeAudit framework) — systematic evaluation of tool-call safety benchmarks; relevant to agent vulnerability discovery. New: OpenAI GPT-Red — self-play RL red-teaming model that scales prompt injection discovery; adversarially trains GPT-5.6 Sol achieving 6x fewer failures; discovered Fake CoT attacks. New: Lineation.ai Zero Trust runtime security control plane for autonomous agents — MCP-level enforcement, immutable reasoning audit trail. New: User Permissions for AI Agents survey — taxonomy of specification, derivation, enforcement across 21 proposals and 5 commercial agents; no proposal achieves all three goals (low overhead, formal policies, deterministic enforcement). New: AgentCompass — unified evaluation infrastructure decoupling benchmark, harness, environment; fault-tolerant runtime and trajectory analysis for reward-hacking detection. New: From Controlled to the Wild: Evaluation of Pentesting Agents — protocol shifting from task completion to validated vulnerability discovery using LLM-based semantic matching and bipartite resolution; open-source code/ground truth.