OpenLLMetry — Observability & Evals

What is OpenLLMetry?

Every other option in this category is a product with its own backend, which means adopting one adds a system to run and a place where half your telemetry lives apart from the rest. This is the instrumentation layer instead: it emits standard OpenTelemetry spans for model and vector-database calls, and those go to Datadog, Grafana, New Relic, Splunk, Honeycomb or wherever your existing traces go. The trade is that a general-purpose backend shows spans and latencies, not the prompt-versioning and evaluation views a dedicated platform is built around.

What can you do with OpenLLMetry?

Send LLM traces where the other traces go — Because the output is standard OpenTelemetry, more than twenty-five destinations are supported, including Datadog, Grafana, New Relic, Splunk and Honeycomb, alongside Traceloop's own dashboard.
Instrument without restructuring — Tracing attaches non-intrusively, whether the application uses a supported framework or calls a foundation-model API directly, so adoption does not start with a refactor.
Attribute the slow hop correctly — Vector-database calls are reported alongside model calls, which is what lets a slow answer be traced to retrieval rather than to the model, or the other way around.
Change observability vendor without re-instrumenting — The spans are vendor-neutral by construction, so switching backend is an exporter configuration rather than a second pass through the codebase.
Report by hand where automation stops — Calls to models and vector databases can be reported manually for code paths no automatic instrumentation covers, so coverage is not limited to the supported integration list.

Before you choose OpenLLMetry

A general-purpose observability backend renders spans and latencies but not the LLM-specific surfaces — prompt versioning, evaluation scores, dataset curation — the dedicated platforms are built around.

Star history

21 Aug to 28 Aug · +18

7.4k7.4k

Frequently asked questions

Is OpenLLMetry free for commercial use?

OpenLLMetry is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.

How can OpenLLMetry be deployed?

OpenLLMetry is available as Self-hosted / Runs locally.

Documentation

Reproduced from the traceloop/openllmetry README, published under Apache-2.0. Read the original ↗

🎉 New: Our semantic conventions are now part of OpenTelemetry! Join the discussion and help us shape the future of LLM observability.

Looking for the JS/TS version? Check out OpenLLMetry-JS.

OpenLLMetry is a set of extensions built on top of OpenTelemetry that gives you complete observability over your LLM application. Because it uses OpenTelemetry under the hood, it can be connected to your existing observability solutions - Datadog, Honeycomb, and others.

It’s built and maintained by Traceloop under the Apache 2.0 license.

The repo contains standard OpenTelemetry instrumentations for LLM providers and Vector DBs, as well as a Traceloop SDK that makes it easy to get started with OpenLLMetry, while still outputting standard OpenTelemetry data that can be connected to your observability stack. If you already have OpenTelemetry instrumented, you can just add any of our instrumentations directly.

🚀 Getting Started

The easiest way to get started is to use our SDK. For a complete guide, go to our docs.

Install the SDK:

pip install traceloop-sdk

Then, to start instrumenting your code, just add this line to your code:

from traceloop.sdk import Traceloop

Traceloop.init()

That’s it. You’re now tracing your code with OpenLLMetry! If you’re running this locally, you may want to disable batch sending, so you can see the traces immediately:

Traceloop.init(disable_batch=True)

⏫ Supported (and tested) destinations

See our docs for instructions on connecting to each one.

🪗 What do we instrument?

OpenLLMetry can instrument everything that OpenTelemetry already instruments - so things like your DB, API calls, and more. On top of that, we built a set of custom extensions that instrument things like your calls to OpenAI or Anthropic, or your Vector DB like Chroma, Pinecone, Qdrant or Weaviate.

Vector DBs

✅ Chroma
✅ LanceDB
✅ Marqo
✅ Milvus
✅ Pinecone
✅ Qdrant
✅ Weaviate

Frameworks

✅ Agno
✅ AWS Strands (built-in OTEL support)
✅ CrewAI
✅ Haystack
✅ LangChain
✅ Langflow
✅ LangGraph
✅ LiteLLM
✅ LlamaIndex
✅ OpenAI Agents

Protocol

✅ MCP

🔎 Telemetry

We no longer log or collect any telemetry in the SDK or in the instrumentations. Make sure to bump to v0.49.2 and above.

Why we collect telemetry

The primary purpose is to detect exceptions within instrumentations. Since LLM providers frequently update their APIs, this helps us quickly identify and fix any breaking changes.
We only collect anonymous data, with no personally identifiable information. You can view exactly what data we collect in our Privacy documentation.
Telemetry is only collected in the SDK. If you use the instrumentations directly without the SDK, no telemetry is collected.

🌱 Contributing

Whether big or small, we love contributions ❤️ Check out our guide to see how to get started.

Not sure where to get started? You can:

Book a free pairing session with one of our teammates!
Join our Slack, and ask us any questions there.

💚 Community & Support

Slack (For live discussion with the community and the Traceloop team)
GitHub Discussions (For help with building and deeper conversations about features)
GitHub Issues (For any bugs and errors you encounter using OpenLLMetry)
Twitter (Get news fast)

🙏 Special Thanks

To @patrickdebois, who suggested the great name we’re now using for this repo!