DeepEval
Evaluation written as unit tests, so a worse prompt or a swapped model fails CI like broken code
What is DeepEval?
The shape is deliberately pytest's: assertions inside test functions, a runner that discovers them, parameterised cases — except the assertion is about whether an answer was faithful to its context or whether an agent finished the task. That framing is the point, because it puts quality regressions in front of the same gate that catches syntax errors. The thing to plan for is that most of these metrics are scored by a model, so the suite has a bill and a variance that ordinary tests do not.
What can you do with DeepEval?
- Run evals the way you run tests — Assertions sit inside ordinary test functions, a single command discovers and runs the files, and parameterised cases work as they do in pytest — so there is no separate evaluation harness to learn or maintain.
- Write the criterion as a sentence — The G-Eval metric takes an evaluation criterion in plain language and scores against it, which is how a domain rule no standard metric covers still becomes an enforceable test.
- Score the steps, not only the answer — Component-level evaluation scores individual spans — a retrieval, a tool call, a reasoning step — so a failing test points at which part of the pipeline regressed instead of just reporting a worse final answer.
- Judge an agent by whether it finished — A task-completion metric asks whether the objective was actually achieved, which is the question a per-response quality score structurally cannot answer.
- Test conversations, not single turns — Conversational metrics evaluate multi-turn interactions for qualities like coherence across the exchange, covering failures that only appear after several turns.
Before you choose DeepEval
- Most metrics are scored by a model, so the suite carries an API bill and run-to-run variance that ordinary unit tests do not — pinning the judge model matters as much as pinning the one being tested.
- This is the open half of Confident AI: local runs are complete on their own, but shared reporting, regression history and production monitoring sit in the commercial platform behind it.
Star history
21 Aug to 30 Oct · +1.4k
Frequently asked questions
Is DeepEval free for commercial use?
DeepEval is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.
How can DeepEval be deployed?
DeepEval is available as Runs locally / Self-hosted.