LLM Benchmark Watch

ChatGPT Work launched as persistent agent. Claude Code token waste: 33k tokens before reading prompt vs OpenCode's 7k, fueling tokenmaxxing narrative. New benchmarks: AgenticDataBench, Vera-Bench, AgenticSTS, KernelCraft, Android Bench, etc. First fully agentic AI ransomware attack (JadePuffer). Pentagon confirms Grok used for military targeting. Efficient Model Evaluation paper proposes sequential testing to cut compute up to 80%. LLM leaderboards shifting focus to enterprise TCO and latency. Meta's behavioral state decay paper introduces separate memory agent. Second Brain for AI v2 persistent memory layer launched. Comprehensive architecture-first guide to agentic AI frameworks published. Microsoft routing some Copilot prompts to in-house MAI models to cut costs. Internet Court uses AI juries to resolve agent disputes. Cynative open-source security agent enforces read-only by default for frontier model agents in cloud environments. Self-guided test-time training paper improves long-context reasoning up to 15% on LongBench-v2/Pro. Benchmark methodology critique: LLM-generated code shows little execution-time difference from canonical solutions. Mechanistic study on fine-tuned knowledge generalization identifies 'Knowing-Using Gap' with self-patching intervention recovering 58-75% of gap. Grok CLI uploaded entire home directory to GCS — another major security incident. AISec emerges as new discipline; AI supply chain security highlighted. Task-cost vs token-cost analysis shows GLM 5.2 matching Opus 4.8 at lower cost, Sonnet 5 more expensive per task despite cheaper tokens. UX roundup confirms agents expand tasks and shift human roles. 50% of enterprises shipped agents that passed evals but failed in production — stark evaluation gap. Lineation.ai launches Zero Trust runtime security control plane for autonomous agents at MCP layer. GPT-Red automated red-teaming beats humans on prompt injection, setting new safety eval standard. EU Android mandate could reshape agent distribution. SpaceXAI open-sources Grok Build (Rust agent harness). LLM-generated GPU kernels benchmark shows only 70% roofline on production operators, correctness inflated by PyTorch fallbacks. AgentCompass unified evaluation infrastructure released. Statistical self-consistency paper identifies 'macro fallacy' as new eval gap. Entrust and ITU launch agent trust program for identity frameworks. @omarsar0 shares overview of self-improving agentic systems. New benchmark: Multimodal LLMs for scientific visualization literacy — Gemini leads, open-source lags human baseline, performance varies by visualization type (good at illustrations/search, poor at quantitative estimation/flow). AgentFootprint benchmark for storage footprint of LLM agents reveals huge dispersion across frameworks and that framework residue can dominate output artifacts. Methodological critique questions reliability of Item Response Theory in AI benchmarks due to regime mismatch — simulation study across 18,000 conditions. Google lowers agent barrier with background execution, remote MCP, and custom function mixing in Gemini API's Managed Agents. Pearl LLM Leaderboard report challenges public leaderboard reliability: newer models don't always outperform older ones, vertical variance is large, and more reasoning effort rarely helps. GhostWriter attack on AI agent long-term memory: 98% injection rate, 60% activation; AM-Sentry mitigation proposed. Connector security risks: PromptArmor data on rapid tool changes and data leakage to third-party AI services — reinforces need for Zero Trust runtime controls. Recursive Harness Self-Improvement (RHI) paper: low-reasoning-effort agents can surpass high-reasoning-effort ones with 60% cost reduction via better context management. DSWorld world model for data science agents: 14x RL training speedup, 3-6x inference gains, cuts trial-and-error. From Human-Centric to Agentic Code Review study: agent-involved reviews speed up decisions but don't improve quality — challenges faster=better assumption. Revolut agent deployment stats: 2x GenAI vs classical ML, 8x cost premium for frontier models, silent model failure left them blind, 8x faster voice assistant across 25k calls/month — concrete enterprise ROI signal. 1Password and Anthropic bring secure credential access to Claude — zero-exposure architecture, per-task biometric validation, directly addresses agent credential leakage vulnerability. Red Hat publishes practical guide on computer use with open-weight models locally using quantization — aligns with cost-efficiency and privacy trends. Goodput vs throughput article highlights real-world LLM serving optimization — reinforces cost-per-task thinking. Writer's AI harness cuts token spend 38% without accuracy loss. Contrastive analysis (comparing cohorts vs baselines) proposed as better evaluation methodology for agent conversations.