← Back to all projects

AgentBench

Measures a model as an agent by putting it in a shell, a database and a shopping site, not in a question-and-answer set

Apache-2.0
Stars
3.8k
Forks
293
Open issues
80
Last commit
8 Feb 2026

What is AgentBench?

A model that answers well can still be poor at operating something, and a question-and-answer benchmark will not show that. AgentBench puts a model into environments that push back: a shell it has to work in, a database it has to query, a knowledge graph, a text world and a shopping site where the task only counts if the right item ends up ordered. Every task is scored on whether it was completed, not on whether the reasoning read well. The current release runs those environments as containers driven by function calling, which makes it reproducible on your own hardware — with the practical caveats the maintainers list, including one environment that needs about 16GB of memory and another that leaks until its worker is restarted.

What can you do with AgentBench?

  • Score completion, not eloquence — A task counts when the database returned the right rows or the right item was ordered, which is a harder bar than a plausible-looking answer.
  • Cover several kinds of operating at once — Shell work, database queries, knowledge-graph navigation, a text world and web shopping each fail differently, so the set exposes different weaknesses.
  • Reproduce it on your own machine — The environments ship as containers brought up with a single compose command, so a comparison does not depend on a hosted leaderboard.
  • Test the function-calling path you actually use — The current version drives models through function calling rather than parsed free text, so what is measured is the interface production code uses.
  • Scale the workers to the run — Each environment runs as its own worker that can be replicated, so a long evaluation can be spread across more of the machine.

Before you choose AgentBench

  • The maintainers document real operational rough edges: one environment needs roughly 16GB of memory, another leaks memory and disk until its worker is restarted, and a third needs a separately downloaded dataset.
  • The current function-calling release covers a different task set from the original versions, so published numbers are only comparable within the same version — check which one a score refers to.

Star history

27 Aug to 30 Oct · +105

3.7k3.8k

Frequently asked questions

Is AgentBench free for commercial use?

AgentBench is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.

How can AgentBench be deployed?

AgentBench is available as Runs locally / Self-hosted.