Key facts.
- LangChain provides tracing and evaluation through LangSmith, a standalone platform that works with or without LangChain, which means observability is paired on, not built in by default. source
- CrewAI ships its own observability dashboard in the enterprise tier and otherwise exports OpenTelemetry to vendors like Langfuse and Arize. source
- The common failure mode is not the framework. It is the absence of observability, human oversight, and cost discipline built in from the first pull request. source
- OpenTelemetry's GenAI conventions exist precisely so this added instrumentation stays portable across backends. source
Why don't frameworks just include observability?
Because observability is a product in its own right, and the framework vendors know it. Tracing, evaluation, drift detection, and cost analysis are deep enough to support whole companies, so they get sold as platforms (LangSmith, Langfuse, Arize) rather than bundled as defaults. The framework gives you the orchestration that makes a demo work. The observability that makes production survivable is a separate purchase or a separate setup, and a lot of teams do not notice the gap until they are debugging a live failure with nothing to read.
This is not a knock on the frameworks. It is a planning fact. If you assume observability comes with the orchestration, you will discover at the worst moment that it did not.
What should you wire in from the first commit?
Emit traces early, using OpenTelemetry's GenAI conventions so they stay portable. Add an evaluation layer that judges output quality, not just whether the call returned. Baseline behavior and watch for drift. Track token cost per step so a runaway loop shows up as a line on a chart. None of this is hard at the start of a project. All of it is painful to retrofit once the agent is serving traffic and the failures are already happening.
| Capability | Typically in the framework? | Where it usually lives |
|---|---|---|
| Orchestration, tools, memory | Yes | Core framework |
| Tracing | Not by default | Separate platform / OTel export |
| Evaluation, drift | Rarely | Add-on or enterprise tier |
| Cost telemetry | Rarely | Wired in by the team |
This gap is the case for a Pattern Intelligence Layer that does not depend on which framework you picked. Reliability at the pattern level means tracing, evaluation, and drift detection live around the agent as portable, always-on capabilities, so observability is a property of how you run agents, not a feature you hoped your framework included. Pick any orchestrator. Keep the observability constant.
Frequently asked questions
Does picking the right framework solve observability?
No. The frameworks deliberately leave tracing and evaluation to separate platforms or paid tiers. Observability is something you add, regardless of framework.
How do I avoid lock-in to one observability vendor?
Emit OpenTelemetry GenAI-convention spans. They are portable across Datadog, Honeycomb, Langfuse, Arize, and others, so you can switch backends.
When is the cheapest time to add it?
The first commit. Tracing and evaluation are light to add early and brutal to retrofit once the agent is live and failing.