Meta-Harness: Automating Harness Optimization for LLMs Outperforms Manual Tuning
The Meta-Harness system, developed by experts from Stanford, MIT, and KRAFTON, automates the creation of harnesses for language models. A harness defines context processing: data extraction, structuring, and feeding to the model. Meta-Harness uses the Claude Code coding agent based on Opus 4.6 to analyze errors and generate improved code versions. On text classification tasks, accuracy improved by 7.7 percentage points while using 4x less context compared to manual setups. The project code is publicly available.
Full Access to Optimization History
A key feature of Meta-Harness is giving the agent access to the entire history via the file system. This includes source code from previous harnesses, execution logs, and reasoning traces. The volume of diagnostic information reaches 10 million tokens per iteration—1,000 times more than systems like OpenEvolve and TTT-Discover, which are limited to 26k tokens.
The agent independently selects relevant data using standard tools like grep and cat. This enables deep analysis without compressing feedback.
Advantages of the approach:
- Scalability: Handles large data volumes without losing details.
- Autonomy: The agent decides which files to examine.
- Auditability: Generated code is easy to verify.
Results on Benchmarks
Testing was conducted on three task suites:
- Text classification: Meta-Harness achieved results in 4 iterations instead of 40 for competitors—a 10x speedup.
- Math Olympiad problems (200 IMO-level tasks): The automatic example retrieval strategy boosted accuracy by 4.7 pp on average across five models not involved in optimization.
- TerminalBench-2 (agentic coding): The harness for Claude Haiku 4.5 delivered 37.6% success, surpassing Goose (35.5%) and topping the leaderboard.
| Benchmark | Manual Tuning | Meta-Harness | Improvement |
|--------------------|-------------------|--------------|----------------------|
| Text Classification| Baseline | +7.7 pp | 4x less context |
| IMO Math | Baseline | +4.7 pp | - |
| TerminalBench-2 | 35.5% (Goose) | 37.6% | 1st place |
Transferability and Practical Value
Harnesses discovered by Meta-Harness transfer to new models and tasks. The logic is implemented in readable code, simplifying audits and refinements. Development became feasible thanks to advances in coding agents in early 2026.
The authors highlight the shift from manual harness engineering to an automated process. This reduces reliance on engineer expertise and speeds up iterations.
Key Takeaways
- Meta-Harness uses 10 million tokens of diagnostics per iteration for deep analysis.
- 10x faster optimization on text tasks.
- First place on TerminalBench-2 with 37.6% success.
- Full transferability of harnesses to other models.
- Open-source code for reproduction and extension.
The approach shows how meta-optimization based on advanced agents is transforming LLM app development.
— Editorial Team
No comments yet.