← Back to all projects

Inspect AI

The framework for writing and running evaluations, from the UK AI Security Institute — not a benchmark

OfficialMIT
Stars
3.1k
Forks
880
Open issues
489
Last commit
30 Oct 2026

What is Inspect AI?

Where the other entries in this category are each one measurement, this is the machinery for building measurements: a dataset supplies inputs and targets, a solver produces a response (a single model call or a full multi-turn agent), a scorer decides whether it was right, and a task binds the three together. More than two hundred pre-built evaluations come with it, model-written code runs inside a sandbox, and a log viewer shows what actually happened on each sample. What it cannot do is tell you what is worth measuring.

What can you do with Inspect AI?

  • Assemble an evaluation from four parts — Datasets carry inputs and targets, solvers turn an input into a response, scorers judge it by text comparison, model grading or a scheme you write, and a task binds them — so swapping the scorer does not mean rewriting the evaluation.
  • Start from two hundred existing evaluations — A large library of pre-built evaluations ships with the framework, which means the common measurements are available immediately and the custom work is confined to what is genuinely specific to you.
  • Evaluate agents, not just completions — Built-in agents and multi-agent primitives are supported, along with bash, python, text-editing, web search, browsing and computer tools, plus custom and MCP tools — so the thing under test can be a full agent.
  • Run untrusted model code safely — Sandboxing covers Docker, Kubernetes, Modal, Proxmox and Vagrant with an extension API for anything else, which is a prerequisite once an evaluation involves executing what the model wrote.
  • Look at what actually happened — A web-based log viewer shows individual samples and runs, turning a surprising aggregate score into something you can trace back to specific transcripts rather than only report.

Before you choose Inspect AI

  • It is a framework, not a verdict: it runs the evaluation you design, so deciding what to measure and what counts as passing stays with you and is the part that is actually hard.

Star history

21 Aug to 30 Oct · +545

2.6k3.1k

Frequently asked questions

Is Inspect AI free for commercial use?

Inspect AI is released under the MIT licence — OSI-approved open source, which permits commercial use.

How can Inspect AI be deployed?

Inspect AI is available as Runs locally / Self-hosted.