In July 2025, an autonomous coding agent executed a DROP DATABASE command on a production system — during a scheduled code freeze. Every infrastructure metric was green. CPU, memory, latency, error rates: all normal. The agent had been reasoning over stale retrieval results, and no existing monitoring system caught it. This is the failure mode that’s driving one of the most significant shifts in observability infrastructure in years.
The monitoring blind spot AI agents create
Traditional APM tools were built for deterministic systems. When a service throws an exception or a database query times out, Prometheus and Datadog capture it. AI agents introduce a different class of failure: context decay (acting on outdated retrieval), orchestration drift (multi-agent chains diverging from intent), and autonomous side effects (actions that are technically valid but operationally wrong). None of these surface as errors on a dashboard.
The data confirms the gap. VentureBeat reported that only 21% of organizations have runtime visibility into what their agents are actually doing — and 88% had AI agent security incidents in the last 12 months. Meanwhile, 43% of AI-generated code changes require manual debugging in production after passing QA and staging, according to a 2026 Lightrun survey of 200 senior SRE and DevOps leaders at large enterprises.
OpenTelemetry’s answer: GenAI semantic conventions
OpenTelemetry’s response to this gap is the GenAI semantic conventions specification. As of mid-2026, gen_ai.client spans are stable — covering model name, input/output token usage, and finish reasons — while gen_ai.agent spans remain experimental. The direction is clear: standardized telemetry for LLM calls and agent interactions, exportable to any OTel-compatible backend.
The tooling is already following. In February 2026, New Relic launched an AI agent platform with centralized OTel fleet management and MCP support — APM agents now ingest OTel data streams alongside native telemetry. In March, Datadog released a remote MCP server giving AI coding agents (Claude Code, Cursor, Codex, GitHub Copilot) live access to production logs, metrics, and traces. Datadog’s CPO described it as “the next stage of AI-native development — moving from AI copilots to AI operating on live production systems.”
The cost problem is accelerating the shift
There’s a second driver. Elastic’s 2026 Observability Survey found that 97% of organizations experienced unexpected observability costs or overages, with 67% reporting these surprises regularly. Median enterprise spend sits at $1.95M per year. 67% of IT leaders say they’re likely to switch observability vendors within 1–2 years — compared to the traditional 5–7 year cycle — according to LogicMonitor’s 2026 Observability Trends report.
OpenTelemetry makes switching viable. When instrumentation is vendor-neutral, migrating from Datadog to Elastic or Dash0 or New Relic doesn’t require re-instrumenting every service. Instrument once with OTel; change backends without friction. That’s the underlying reason 84% of organizations are either consolidating or evaluating consolidation of their observability stack.
What teams are actually instrumenting in 2026
The practical shift for engineering teams comes down to three layers that weren’t part of observability thinking two years ago:
- LLM call telemetry — token usage, model latency, finish reasons, prompt/response pairs (where retention policy allows). The gen_ai.client spans give you a standard schema that works across OpenAI, Anthropic, and any OTel-compatible provider.
- Agent trace correlation — connecting agent decision steps to the downstream API calls, database writes, and external actions they trigger. Without this, a multi-agent failure looks like a random infrastructure spike rather than a reasoning error in step 3 of 7.
- Human-in-the-loop checkpoints — instrumenting where and when agents pause for approval, and what happens when they proceed autonomously. This is the audit trail that security and compliance teams need, and it’s currently missing in most deployments.
Conclusion
The DoorDash case makes the return on investment concrete: Resolve AI’s multi-agent observability system reduced time to root cause by 87%, saving an estimated 1,000 engineering hours per year. DoorDash’s target is resolving production incidents within 10 minutes — a goal that requires agents that can perceive production state in real time, not engineers staring at dashboards. OpenTelemetry is how that perception gets standardized. Teams that build their observability on OTel now are the ones who will be able to plug in whatever AI-native tooling emerges next year without starting over.