
SWE-bench
The benchmark coding agents are measured on: real GitHub issues, judged by each repository's own tests
What is SWE-bench?
A task gives the model a codebase and an issue and asks for a patch; the patch is accepted only if the tests that were failing now pass and the ones that were passing still do. Using the projects' own test suites as the judge is what makes the result mean something concrete rather than resemble a rubric score. What it measures is correspondingly narrow: producing a fix for a problem someone else already found, filed and described — not deciding what to build, not reviewing, not navigating a codebase without an issue to anchor on.
What can you do with SWE-bench?
- Let the repository do the grading — A submission is scored by running the project's existing tests: the ones the issue broke must pass afterwards and the ones that already worked must not regress, so the criterion belongs to the codebase rather than to the benchmark author.
- Run the evaluation reproducibly — The harness is fully containerised with Docker so a run on your machine matches a run on someone else's, and evaluation can also be executed in the cloud through Modal when a full pass would take too long locally.
- Pick the variant that matches the claim — Verified is a 500-problem subset that software engineers confirmed are actually solvable, Lite is a smaller cut for quick iteration, Multimodal covers visual software domains and Multilingual goes beyond the original Python repositories.
- Compare against published results — A public leaderboard collects submissions, which is the only reason numbers reported by different agents on different weeks can be placed next to each other at all.
- Evaluate against a split nobody can train on — For the Multimodal variant the test split is deliberately kept private and submissions are scored through a cloud tool, which is how that variant stays meaningful as models absorb more public data.
Before you choose SWE-bench
- The score measures writing a patch that satisfies existing tests for an issue someone else already diagnosed; design decisions, code review and working without a filed issue are outside what it can say anything about.
- Task instances are built from public repository history, which is also the material the models under test were trained on, so absolute scores should be read with that overlap in mind.
Star history
21 Aug to 30 Oct · +442
Frequently asked questions
Is SWE-bench free for commercial use?
SWE-bench is released under the MIT licence — OSI-approved open source, which permits commercial use.
How can SWE-bench be deployed?
SWE-bench is available as Runs locally / Self-hosted.