
MLflow
The experiment tracker teams already run, extended to hold agent traces, judges and prompt versions too
What is MLflow?
MLflow's usefulness for agents comes from where it already is: many organisations run it for model training, and the GenAI half puts agent traces, evaluation runs and prompt versions in that same experiment view. Instrumentation is automatic for a long list of agent libraries — one line and every prompt, retrieval, tool call and response is recorded, in OpenTelemetry's GenAI conventions so the traces are readable by other tools too. From there the recorded runs become evaluation sets, scored by built-in judges or ones you write. The cost is that MLflow is a platform, not a library: even for traces alone you are running a tracking server with a database and an artifact store behind it.
What can you do with MLflow?
- Record a run without instrumenting it — Automatic instrumentation covers a long list of agent frameworks and model providers, so the prompts, retrievals and tool calls appear from one line of setup.
- Traces other tools can read — The recorded spans follow OpenTelemetry's conventions for generative AI, so they can be exported into an existing observability stack instead of staying in a silo.
- Turn recorded runs into an evaluation set — Real traffic becomes the test cases, which is the only reliable way to find out whether a change helped on the inputs users actually send.
- Score with judges, built-in or your own — Ready-made judges cover the common failures such as hallucination and irrelevance; a custom scorer encodes what "good" means for your own task.
- Version prompts with the runs that used them — Prompt templates are stored, compared and traced back to the runs they produced, so a regression can be tied to the edit that caused it.
Before you choose MLflow
- MLflow is a platform, not a client library. Even if all you want is traces, you take on a tracking server with a backing database and an artifact store, and their upgrades.
- The documentation covers the open project and Databricks' managed version together, so check which of the monitoring and governance features apply to a self-hosted install before planning around them.
Star history
27 Aug to 30 Oct · +912
Frequently asked questions
Is MLflow free for commercial use?
MLflow is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.
How can MLflow be deployed?
MLflow is available as Self-hosted / Runs locally / Managed cloud.