
Ragas
Scores retrieval on two questions: did it find the right passages, and did the answer stay inside them
What is Ragas?
The failure most retrieval systems have is invisible from the outside — the answer reads well and is not supported by anything that was retrieved. Ragas measures that directly. It splits the pipeline in two and scores each half: whether the retrieved passages actually contained what was needed, and whether the generated answer stayed within them. It can also build a starting test set out of your own documents, so the first evaluation does not wait for someone to hand-write a hundred questions. Nearly every metric is itself a model call, which is the thing to plan around: a full run costs money, and two runs on the same data will not produce identical numbers.
What can you do with Ragas?
- Separate a retrieval failure from a generation failure — Retrieval and answer are scored apart, so you learn whether to fix the index or the prompt instead of guessing at both.
- Catch answers with nothing behind them — The faithfulness score checks each claim in the answer against the passages that were retrieved — the failure that reads perfectly well.
- Start from a generated test set — Questions can be produced from your own documents, so a first evaluation is possible before anyone has written test cases by hand.
- Judge what a number cannot — A free-text criterion — is this reply rude, does it give medical advice — is evaluated by a model as a pass or fail you can track over time.
- Score tool use as well as answers — Metrics for agent behaviour look at whether the right tools were called and whether the run achieved what was asked, not only at the final text.
Before you choose Ragas
- Most metrics are model calls, so an evaluation run has a bill and a variance — the same inputs scored twice will not give identical numbers, which matters when you are comparing small changes.
- The repository has been quiet since early 2026 and moved to a new owner, so confirm the maintenance situation before adopting it as the evaluation gate in a release process.
Star history
27 Aug to 30 Oct · +698
Frequently asked questions
Is Ragas free for commercial use?
Ragas is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.
How can Ragas be deployed?
Ragas is available as Runs locally / Self-hosted.