← Back to all projects

promptfoo

Declarative evals and automated red-teaming from one CLI, running entirely on your own machine

OfficialMIT
Stars
26.6k
Forks
2.6k
Open issues
897
Last commit
30 Oct 2026

What is promptfoo?

Test cases are YAML: a prompt, some inputs, and assertions about the output. That is enough to produce a side-by-side matrix comparing several prompts or models on the same inputs, and to fail a build when a change makes things worse. The same tool also attacks the application, generating adversarial inputs to surface vulnerabilities and compliance risks. It works from the outside — prompt in, answer out — so when the question is which step inside a running agent went wrong, this is not the instrument for it.

What can you do with promptfoo?

  • Describe the test, not the harness — A YAML file with prompts, inputs and assertions is the whole configuration — no notebook to maintain and no test framework to wire up first, which also keeps it usable from a codebase in any language.
  • Compare options as a table — Results render as a matrix across prompts and inputs in a web viewer and on the command line, so the choice between two prompts or two models is settled by a grid rather than by impressions.
  • Fail the build on a regression — It runs as a CLI, as a library or inside CI pipelines such as GitHub Actions, which is what turns evaluation from an occasional exercise into a gate.
  • Attack your own application first — Automated red-teaming generates adversarial inputs to find vulnerabilities and compliance risks, covering the security half that evaluation tools in this category usually leave out.
  • Keep the data on your machine — Evaluations run locally and talk to the model provider directly, so test inputs and outputs are not routed through a third-party service to be scored.

Before you choose promptfoo

  • It exercises the application from the outside, so it does not replace tracing: when the failure is inside a multi-step agent, this tells you the answer got worse but not which step caused it.

Star history

21 Aug to 30 Oct · +2.2k

24.4k26.6k

Frequently asked questions

Is promptfoo free for commercial use?

promptfoo is released under the MIT licence — OSI-approved open source, which permits commercial use.

How can promptfoo be deployed?

promptfoo is available as Runs locally / Self-hosted.