← Back to all projects

Terminal-Bench

Measures agents in the environment terminal agents actually inhabit: a shell inside a container

OfficialApache-2.0
Stars
992
Forks
611
Open issues
515
Last commit
30 Oct 2026

What is Terminal-Bench?

Rather than a fixed set that saturates and stops distinguishing anything, this is a continuous benchmark: tasks are added and fixed over time and the dataset is published as tagged releases, run by a separate harness called Harbor. That design is the strength and the complication — a number without a dataset version attached does not mean much, version 1.0 lives in its own repository that the original URL now redirects to, and the documented first step is running the reference solutions repeatedly to confirm your own sandbox is sound before trusting any result.

What can you do with Terminal-Bench?

  • Run an agent against the tasks with one command — Harbor downloads the dataset and executes the run; flags select which agent and which model are under test, so comparing two models is a change of argument rather than a change of setup.
  • Verify your sandbox before trusting a number — The documented first step is running the reference solutions several times over to confirm every task behaves correctly in your environment — a flaky oracle means the scores you produce are measuring your infrastructure.
  • Run it at real concurrency in the cloud — Modal is supported as an execution environment with hundreds of concurrent runs, which is what makes a full pass over the task set finish in a sitting rather than overnight.
  • Track a frontier that keeps moving — Tasks are added and corrected continuously and published as tagged dataset releases, so the benchmark keeps discriminating as agents improve instead of going flat at the top.
  • Fix a task you disagree with — Task bugs are reported as issues and improvements or new tasks arrive as pull requests, with a publicly visible roadmap — the task set is editable rather than handed down.

Before you choose Terminal-Bench

  • The dataset is versioned separately from the harness and keeps changing by design, so a Terminal-Bench score is only comparable to another when both state which dataset release produced it.
  • Version 1.0 sits in a separate repository and the original project URL now redirects there rather than here, so older published results need checking against which of the two produced them.

Star history

21 Aug to 30 Oct · +468

524992

Frequently asked questions

Is Terminal-Bench free for commercial use?

Terminal-Bench is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.

How can Terminal-Bench be deployed?

Terminal-Bench is available as Runs locally / Self-hosted.

Terminal-Bench
Ask AI
GitHub