Open-Source Benchmarks for AI Coding Agents

The benchmark that defined the format: give an agent a real codebase and a real GitHub issue, ask it to generate a patch, and verify the result in a reproducible Docker harness (120GB storage, 16GB RAM, 8 CPU cores recommended). Accepted as an oral presentation at ICLR 2024. Three variants followed as the field matured: SWE-bench Lite, a smaller and cheaper subset for fast iteration; SWE-bench Verified (2024), 500 problems human-confirmed as actually solvable, addressing early concerns about unsolvable or mislabeled issues; and SWE-bench Multimodal (2025), extending evaluation into visual software domains.

Scale AI's extension of the SWE-bench format toward long-horizon tasks — issues that demand sustained reasoning across a codebase rather than a localized fix. Ships as a HuggingFace dataset (ScaleAI/SWE-bench_Pro) with Docker images per instance for reproducible evaluation, and maintains both a public and a commercial leaderboard.

Built to test something pass/fail correctness can't: whether an agent codes like a senior engineer, not just a correct one. Instructions are deliberately under-specified natural-language messages — 31% shorter than SWE-bench Pro's — forcing the agent to investigate rather than follow a spec. Tasks average 11 files touched and hundreds of steps, sourced from real production PRs. A validation agent adds "taste scoring": it writes behavioral tests adapted to whatever solution the agent produced, evaluating code quality alongside correctness. Across 50 public and 50 private tasks spanning repos like PostHog, Electric and Gitea, top frontier models solve only around 35% — and responses run 36K–117K output tokens, reflecting how much investigation the tasks actually demand.

SWE-bench's biggest structural gap was that it was almost entirely Python. Multi-SWE-bench closes it: 1,632 high-quality instances curated by 68 expert annotators from 2,456 candidates, spanning seven languages — Java, TypeScript, JavaScript, Go, Rust, C and C++. The accompanying paper (accepted to NeurIPS 2025's Datasets and Benchmarks track) evaluated nine models, including GPT-4o, Claude 3.5 Sonnet and DeepSeek-V3, across three agent frameworks (Agentless, SWE-agent, OpenHands). A companion Multi-SWE-RL community effort adds 4,723 further instances for reinforcement-learning research.

Coding agents don't just edit files — they run commands. Terminal-Bench, built on the Harbor framework, tests agents purely through a terminal: building a Linux kernel from source with QEMU, standing up a git server, cracking an archive, training a FastText model, reshaping a dataset. Tasks span software engineering, ML, security and data science rather than issue resolution alone. Now on version 2.1 with a public leaderboard, Terminal-Bench 3 has shipped as a frontier-difficulty benchmark, and a Terminal-Bench-Science variant is in development for scientific-computing tasks.

A direct answer to benchmark saturation: 113 tasks written entirely from scratch across 91 repositories in five languages, specifically to be contamination-free — no model could have seen these solutions during training. 25 models were evaluated using hand-written verifiers that check actual runtime behavior rather than matching implementation details. Prompts run shorter than comparable benchmarks but demand roughly 5.5x more code and 2x more output tokens to solve. Claude Opus 5 currently leads at a 74% pass rate, with several other frontier models within 4–5 percentage points — evidence the field is closing in on saturating even a benchmark designed to resist it.

Real specs change mid-project — single-shot benchmarks can't see how an agent handles that. SlopCodeBench has an agent implement a spec, then keeps extending the spec across further iterations, measuring how the agent adapts existing code rather than just how well it writes new code. That surfaces failure modes single-shot evaluation structurally can't: path dependence, quality decay across iterations, and the trade-offs an agent makes under evolving requirements. It's explicitly framed as a community-driven evaluation primitive rather than a finished leaderboard — Docker-based execution, LLM-assisted grading, MIT-licensed, with problems maintained in a companion repository.

The hardest task format on this list: agents get only a compiled binary and its documentation — no source, no method signatures, no architectural hints — and must independently design and implement a complete codebase reproducing its behavior. 200 tasks range from small utilities (jq, ripgrep) to enormous projects (a PHP compiler, FFmpeg, SQLite), verified against more than 248,000 behavioral tests generated through agent-driven fuzzing, run sandboxed with no internet access to rule out lookup. It shows: Claude Opus 5 (xhigh) leads with only 4.5% of instances fully resolved, though 37.0% land "almost resolved" at ≥95% of tests passing — and most models score below 1% on full resolution. This is currently the least-saturated benchmark of the group by a wide margin.

A different question again: not "can it solve a hard issue" but "can it follow instructions and edit code correctly, without a human stepping in, across languages and at a defined cost." The polyglot leaderboard runs 225 Exercism exercises across C++, Go, Java, JavaScript, Python and Rust, scoring success rate, cost per run, edit-format accuracy and response well-formedness together — the only benchmark here that treats cost as a first-class evaluation axis alongside correctness.