Autoresearch & deterministic evals
Key Questions
What progress has been made in self-evolving multi-agent systems?
MLEvolve achieves SOTA on MLE-Bench under half the budget, while EvoDS shows 28.9% improvement. Frameworks like AEVO and Role-Agent enable agentic self-evolution with gains up to 4%.
How do agent harnesses improve performance on coding and research tasks?
HarnessForge delivers 12% gains, and Retrospective Harness Optimization boosts SWE-Bench Pro success from 59% to 78%. Self-Harness and HarnessBridge explore learnable bidirectional control for better workflows.
What risks arise from using multiple coding agents?
A Stanford study finds two coding agents perform 50% worse than one due to coordination issues. Meta-Agent Challenge highlights reward hacking risks in multi-agent setups.
What benchmarks test realistic agent capabilities?
Hedge-Bench shows frontier LLMs under 16% success on hedge fund tasks, and AutoLab features 36 long-horizon auto research tasks where persistence outperforms initial intelligence.
How does EP250 reduce costs for agent workflows?
EP250 compiles agent workflows into model weights via fine-tuning, achieving near-frontier quality at 100x less cost by replacing external orchestration.
What open-source frameworks support agentic self-evolution?
Orchard achieves SOTA with smaller models by addressing sandbox management and credit assignment. EvoTrainer co-evolves LLM policies and training harnesses for autonomous RL.
Do AGENTS.md files help agent performance?
Research shows AGENTS.md files do not improve success rates and may reduce efficiency in agent workflows.
What new skill evolution methods are emerging for agents?
OpenClaw-Skill uses tree-based skill libraries, while Orchestra-o1 reaches 72.8% on OmniGAIA through multi-agent orchestration. PreAct reduces redundant reasoning to cut costs.
Self-evolving multi-agents + DeepSeek-V4 + RubricEM. MLEvolve SOTA on MLE-Bench under half budget. EvoDS 28.9% improvement. Stanford study finds two coding agents 50% worse than one. Meta-Agent Challenge shows reward hacking risks. SkillOpt (Microsoft) +23.5 point gain on GPT-5.5. HarnessForge 12% gains over isolated updates. AGENTS.md files don't improve success and may hurt efficiency. AEVO meta-editing framework for agentic self-evolution. Hedge-Bench reveals frontier LLMs <16% on realistic hedge fund tasks. Self-Harness concept. Role-Agent dual-role evolution (4% gain). SGDR state-grounded dynamic retrieval for web agents. Retrospective Harness Optimization self-supervised 59% to 78% on SWE-Bench Pro. EEVEE router-prompt co-evolution. SearchSwarm delegation intelligence. AutoLab benchmark (Jun 2026) — 36 long-horizon auto research tasks; key finding: persistence beats initial intelligence. Claude-Opus-4.6 leads. EvoTrainer co-evolves LLM policies and training harnesses for autonomous agentic RL. Arbor tackles long-horizon research with hypothesis-tree refinement framework, open-source. Q-Evolve uses Implicit Q-Learning for self-evolving agents under delayed feedback. Orchard open-source framework achieves SOTA with smaller models, practical for agentic workflows. Harness Engineering case study with Codex agents demonstrates zero-manual-code paradigm. New: EvoArena benchmark for memory evolution in dynamic environments adds to evaluation landscape. New: Self-Evolving Multi-Agent Systems via Decentralized Memory paper addresses shared memory bottleneck with reuse/exploration pools. New: Orchestra-o1 multi-agent orchestration framework achieves 72.8% on OmniGAIA. New: Role-Agent dual-role evolution paper (4% gain) confirmed. New: OpenClaw-Skill introduces tree-based skill library construction for agents, moving beyond flat greedy distillation. New: EP250 paper proposes 'subterranean agent' paradigm — compiling agent workflows into model weights via fine-tuning, replacing external orchestration, achieving near-frontier quality at 100x less cost. New: PreAct reduces redundant reasoning for computer-use agents, cutting cost and latency. New: HarnessBridge, Retrospective Harness Optimization, Bayesian-Agent, FORT-Searcher papers on agent harnesses and skill evolution. New: Practical tip from @diptanu: using snapshots/forks for rollout starting worlds can significantly reduce infrastructure cost for coding agents.