NthLayer
The AI reliability gap almost nobody is measuring
In March 2026, every observability tool we reviewed monitored the mechanics of AI: latency, tokens, cost. None of them measured whether AI agents were making good decisions. By November, several vendors scored production responses with AI judges, but nothing we found checked an agent’s decisions against what later turned out to be true.
· Updated
- March review
- 9 established observability vendors, 14 AI-specific startups and 6 governance platforms, plus open standards, analyst reports and conference talks.
- November update
- A targeted re-check of announcements, documentation and standards. The March product sets were not re-reviewed in full.
- Quotes
- Checked against their sources in November 2026.
The numbers
- 89% of organisations surveyed had some form of observability for their agents LangChain, Dec 2025 [1]
- #1 barrier to production was still quality LangChain, Dec 2025 [1]
- 69% of agentic AI decisions were verified by a human Dynatrace, Jan 2026 [2]
- 40%+ of agentic AI projects forecast to be cancelled by the end of 2027 Gartner forecast, Jun 2025 [3]
- 85% of IT leaders were comfortable with AI investigating incidents Grafana Labs, Oct 2026 (n=300) [17]
- 50% were comfortable letting it deploy changes to production Grafana Labs, Oct 2026 (n=300) [17]
November 2026 update
The March finding holds, in a narrower form. Between March and November the market moved from mechanics to production judge scores: several large vendors now grade live responses with AI evaluators, which measures perceived quality at the moment of response. Nothing we found checks an agent’s decisions against outcomes confirmed later. In the products we reviewed, human override or reversal rates, confidence calibration, high-confidence failure rates and SLOs on decision quality are not documented features, and no open standard defines them.
Vendors now score production responses
Datadog launched Bits Evals in preview at DASH in June, alongside Patterns (also in preview), which clusters agent interactions and shows evaluation scores beside cost and latency. Its case for the product is this page’s thesis: production traces “do not always indicate whether the interaction was successful from the user’s perspective” [18]. Splunk Agent Observability, “powered by Galileo”, became available in September; Splunk says teams can “observe 100% of traffic without the need for sampling” [19]. Dynatrace’s evaluation preview scores real interactions with an LLM judge, with online evaluations that run “after deployment against sampled production or user traffic” [20].
An AI SRE product now reports outcomes
Microsoft’s Azure SRE Agent documentation (28 October 2026) describes a dashboard that sorts each incident into agent mitigated, assisted by agent, mitigated by user or pending user action, with a success rate for each response plan. After each thread completes, an automated evaluation “scores the outcome on a scale from 1 to 5”, reported as the Intent Met score [21]. It is the closest thing to production outcome measurement we found, and its limits matter. The categories record who mitigated an incident, not whether the agent’s diagnosis was right. The score comes from an automated evaluation of the thread, not from the cause confirmed later. The documentation reports no override or reversal rate.
Elsewhere the measuring is left to the customer. AWS’s DevOps Agent launch post cites “94% root cause accuracy” reported by preview customers and partners, gives no method, and suggests customers track accuracy rates themselves [22].
Independent benchmarks are far lower
ITBench-AA, launched by Artificial Analysis and IBM in May 2026, tests models on Kubernetes root-cause tasks. At launch every frontier model scored below 50%; the leaderboard’s top score was 56.2% when we checked [27]. On OpenRCA 2.0, 11 frontier LLMs recovered the exact root-cause set in 20.7% of cases on average. They named at least one correct root-cause service in 76.0% of cases, but grounded it in a verified causal path in only 61.5% [28]. These are offline benchmarks, not production telemetry, but they show why a confident answer needs checking against what turns out to be true.
The failure mode is not silence, it is a plausible cause delivered with the same tone as a correct one.
Assessing AI SRE agents, he adds that most teams that trial auto-remediation end up “gating it behind approval”.
Governance moved toward runtime
In March 2026 Traceloop, whose platform ran quality checks such as faithfulness and relevance on real traffic, announced it was joining ServiceNow, with its technology becoming part of ServiceNow’s AI Control Tower [23]. Governance is no longer only a gate before execution. What Traceloop brings is quality checks on live traffic, not confirmed outcomes.
The startups were bought
Much of the March startup category now belongs to larger companies. Cisco announced its intent to acquire Galileo on 9 April 2026 and completed the deal in May [24]. Dynatrace completed its acquisition of Arize on 1 October 2026 [25]. Promptfoo agreed to be acquired by OpenAI in March [26], and Traceloop is joining ServiceNow [23]. Evaluation is merging into the big observability and governance platforms. In what we checked, it still means judge scores and annotations, not outcomes.
Standards added agent structure, not judgement
In OpenTelemetry semantic conventions v1.42.0 (12 June 2026), all gen_ai.* conventions were deprecated in the core repository and moved to a dedicated semantic-conventions-genai repository, which had no tagged release when we checked. Its evaluation event, gen_ai.evaluation.result, still carries a name, a score or label and an explanation, as it did in v1.38.0 [30] [12].
Arize’s OpenInference conventions added annotations in July 2026 that record whether a human, an LLM or code judged a span, trace or session. That is the nearest thing to a standard human-judgement signal, but it is still a generic score or label, with no notion of an override or of an outcome confirmed later. Its DECISION span kind, added in October, describes calls to decision-model APIs (models that choose among fixed answers), not the quality of an agent’s decisions [31].
The SLO tooling is unchanged on this point. OpenSLO’s 2026 commits were documentation and website work, the 2026 releases of Sloth and Pyrra add nothing for AI decision quality, and Google SLO Generator has not released since February 2024 [14] [15]. NIST’s AI 800-4 report (March 2026) says monitoring “best practices, validated methodologies, and common terminology are still nascent and scattered across the field” [32].
What this update did not cover
Absence here means not found in the announcements, documentation and blogs we checked in November; it is not the result of an exhaustive audit. We did not re-review the March product sets or the six governance platforms named below, could not retrieve ServiceNow’s own announcements, and did not check per-customer dashboards in other AI SRE products or several evaluation startups. Any of these could hold a counter-example.
The sections that follow are the March 2026 findings, kept as a dated snapshot and corrected where this check showed them to be wrong or too broad.
The thesis
By March 2026 the observability industry had converged on a single strategy for AI: making AI better at investigating incidents. Every vendor we reviewed had an MCP integration. Every vendor had a chatbot. Every vendor had a demo where the AI found root cause in thirty seconds.
Almost nobody was asking the other question: are the AI agents themselves making good decisions?
This research examined 9 established observability vendors, 14 AI-specific startups, 6 governance platforms, the major open-source standards, analyst reports from Gartner and Forrester, and conference talks from SREcon, QCon and KubeCon. The finding was consistent: every tool we reviewed monitored the mechanics of AI (latency, tokens, cost, error rates), while decision quality (reversal rates, confidence calibration, high-confidence failure rates) remained unmeasured in production.
The gap was not subtle, and people building and operating these systems were saying so in public.
By November 2026 the broad form of that finding no longer held: several vendors scored production responses with LLM judges (see the November 2026 update). Reversal rates, calibration and high-confidence failure rates remained unmeasured in the products we reviewed.
What practitioners and analysts said
Obviously, we’re going to monitor all the same things: Do we return results successfully? What’s the latency? None of that stuff changes, but it no longer says that your system is working right.
In his account, what shows whether the system is working right is the user’s feedback. His session abstract makes the related point that AI releases rarely give a clean “working” or “broken” signal.
Every single time, I would ask Claude, what happened here with this graph? Claude will say, request volume increased. This is a capacity problem. You just need to add more servers.
The real cause, he explained, was a lost KV cache. Adding the correction to CLAUDE.md covers that one situation, he said; the model “will get wrong correlation versus causation in the other 99 situations.”
You can’t understand [an AI] model in a vacuum, [and] it’s complexity that is driving a lot of this. … [With] the surge of generative AI, complexity was increasing fast to begin with. It’s now been cranked up even faster.
A model can’t be understood in isolation, she said, and generative AI has sped up the growth of complexity. As a result, TechTarget reports, she said “the wheels are coming off” the older observability model built on separate data pillars.
The implementation of guardrails, security filters, human oversight, or even security observability are not sufficient to ensure consistently appropriate agent use.
Speaking about security attacks on AI agents, Gartner’s chief of research said guardrails, filters, human oversight and security observability are not enough on their own to ensure appropriate agent use.
We move into a world where determinism with respect to management of a system has gone away, and we are into probabilistic management.
A co-author of the Google SRE book says managing systems is moving from deterministic to probabilistic.
More than two-thirds of agentic AI decisions are currently verified by a human.
The report puts that figure at 69%. It also lists “Limited visibility into how agents make or execute decisions” (41%, report PDF, p. 13) among the technical barriers to production. According to the press release, only 13% of organisations “use fully autonomous agents”.
What the vendors offered
A cross-vendor analysis showed near-total convergence. Every vendor we reviewed tracked the same AI metrics: latency, token usage, costs, error rates, and prompt/response traces. Several added hallucination detection and toxicity scanning. None of those we reviewed measured decision quality as an SLO with error budgets.
The table records March 2026. By November, Datadog, Splunk and Dynatrace had added LLM-judge scoring of production traffic, which moves established vendors towards “Yes” for quality scoring and to a partial answer on post-execution quality. No category had added override or reversal rates or confidence calibration as documented features. See the November 2026 update.
| Capability | Established observability vendors 9 reviewed | AI-specific startups 14 reviewed | Governance platforms 6 reviewed |
|---|---|---|---|
| Latency, token and cost monitoring | Yes ✓all 9 | Yes ✓all 14 | Partial ~some |
| Prompt and response tracing | Yes ✓most | Yes ✓all 14 | No ✗ |
| Quality scoring (hallucination, relevance) | Partial ~a few | Yes ✓most | No ✗ |
| Decision quality as an SLO with an error budget | No ✗none | No ✗none | No ✗none |
| Pre-execution policy gates | No ✗ | No ✗ | Yes ✓all 6 |
| Post-execution decision quality measurement | No ✗none | Partial ~via evaluations only | No ✗none |
How we rated
Each column is a category, not a single product: 9 established observability vendors, 14 AI-specific startups and 6 governance platforms. Ratings come from each product’s public documentation as of March 2026; we did not test the products. A capability counts only if it is documented as shipping.
- Yes ✓
- Documented by all or most products in the category. The note says which.
- Partial ~
- Documented by some or a few products in the category (fewer than half), or available only in a limited form. “Via evaluations only” means quality is scored in evaluation runs rather than tracked as the outcome of each production decision.
- No ✗
- Documented by none of the products we reviewed in that category.
Two conflations
“AI for Ops” vs. “Ops for AI”
Thoughtworks articulated a distinction that, in our review, the market was consistently failing to make: AI for Ops is about optimising IT operations; Ops for AI is about making AI systems trustworthy, safe, reliable and cost-effective. Collapsing the two into a single definition, they write, “erases the nuance — and risks applying the wrong playbook to the wrong problem.” [9]
Every established vendor we reviewed was primarily investing in AI for Ops: using AI to help humans investigate incidents faster. Almost nobody was building Ops for AI: measuring whether AI systems themselves are reliable.
Governance vs. reliability
In March 2026 the AI agent governance platforms we reviewed (Microsoft Agent Governance Toolkit, CortexHub, AControlLayer, Kore.ai, Acuvity, Aegis) focused on pre-execution authorisation: is this agent allowed to do this? None of them systematically measured post-execution decision quality: did this agent make a good decision? By November, governance had begun moving into runtime, with Traceloop’s production quality checks going into ServiceNow’s AI Control Tower [23]. We did not re-check the six platforms above.
JetBrains put it clearly: “Governance frameworks define permissions, approval gates, and high-level constraints. They rarely account for how an agent interprets context, chains tools, or explores edge cases while attempting to complete a task. The result is a structural gap.” [10]
The gap between governance and reliability is the space between the bouncer checking your ID at the door and the auditor reviewing whether the decisions made inside were sound. The industry had the bouncer. It didn’t have the auditor. By November it had someone grading the answers as they were given, but still no one checking later whether they held up.
No open standard we found supported judgment SLOs
This section describes March 2026; the November 2026 update covers what has changed since, and none of it adds judgment SLOs.
At the time of review, OpenTelemetry had merged evaluation event conventions (v1.38.0, October 2025) with attributes for generic evaluation scores [12]. The agentic systems proposal (Issue #2664) covered tasks, actions, agents, teams, artifacts and memory, but no decision quality semantics [13]: no gen_ai.decision.* or gen_ai.override.* attributes.
OpenSLO v1.0 supported traditional ratio-based SLIs. It had no concept of judgment quality, observation periods, confidence-weighted metrics or human override signals [14].
All three major SLO generators (Sloth, Pyrra, Google SLO Generator) produced Prometheus rules exclusively for availability and latency SLOs. None supported AI decision quality metrics [15].
OpenSRM (the open specification NthLayer is built on) was the only open specification we found, as of March 2026, that models AI decision quality as an SLO, with constructs for reversal rate, high-confidence failure rate above a confidence_threshold, calibration as a threshold, and measurement windows [16].
Compound failure and the Jevons paradox
The Jevons paradox is the observation that when something becomes cheaper to use, people use more of it, so total consumption rises instead of falling. Applied to software: when code is cheaper to write, we write more of it, and complexity goes up. By 2026 that argument had data behind it.
- 362 documented AI incidents in 2025, up from 233 in 2024 (a 55% rise, our calculation) Stanford AI Index 2026 [11]
- ~1 in 3 attempts still failed for AI agents on structured benchmarks, after OSWorld task success rose from 12% to about 66% Stanford AI Index 2026 [11]
- ~20% success rate for a 10-step AI agent at 85% per-step accuracy 0.8510 ≈ 0.197 — compound probability, illustrative
The compound failure math is unforgiving. An AI agent with 85% per-step accuracy succeeds only about 20% of the time on a 10-step task. As organisations run dozens or hundreds of agents that interact with each other and with traditional services, individual agent monitoring cannot detect compound failures that emerge from interactions between agents.
Independent benchmarks published since March give the illustration real numbers: on OpenRCA 2.0, frontier LLM agents recovered the exact root-cause set in 20.7% of cases [28].
In our case specifically, it’s easier to write software, so we write much more of it, so the complexity goes up and not down, which means things break in more interesting ways, which means more incidents, which means more on-call.
He was applying the Jevons paradox by way of a 2023 post from OpenAI researcher roon, whose follow-up held that gains in developer tooling would be cancelled out by ever-growing complexity. Agents might break that cycle, Palcuie said, but “that’s a big if.”
The structural gap
The research found a market that was simultaneously booming and structurally incomplete. Billions were flowing into observability tooling, and every major vendor had shipped an AI monitoring product. Yet the fundamental question, is this AI agent making good decisions?, was answered by no shipping product we reviewed in March. By November, products scored responses as they were made and one AI SRE product reported who mitigated each incident, but none we found checked decisions against outcomes confirmed later.
Three findings stood out:
- An engineer working on AI reliability at Anthropic said in a public talk that Claude repeatedly misread a lost cache as a capacity problem, and that a correction covers only that case: it “will get wrong correlation versus causation in the other 99 situations.” We found no shipping product that measured how often that kind of judgment turns out wrong.
- The LangChain survey, with 89% reporting observability while quality remained the #1 barrier, suggests the gap is not an observability gap in the traditional sense. It is a decision quality measurement gap.
- Discussing security attacks on AI agents, Gartner stated that “guardrails, security filters, human oversight, or even security observability are not sufficient to ensure consistently appropriate agent use.” In our reading, closing that gap takes something other than more of the same controls.
The gaps followed from architectural choices. Established vendors built around metrics, logs and traces for infrastructure. Startups built around SDK instrumentation for individual model evaluation. Governance platforms built around pre-execution policy gates. The 2026 acquisitions folded several of the startups’ evaluation tools into the first and third groups; in what we checked, what they measure is still a judge score, not an outcome.
Where NthLayer fits
NthLayer is our answer to the gap this research describes: measuring whether decisions hold up, in the monitoring you already run.
- Decisions no test can check are recorded as verdicts. Each verdict is written once, when the decision is made. Its outcome settles once, later: confirmed when a positive outcome is posted, overridden when a human overrides it, expired after 7 days with no signal. The human verification in Dynatrace’s numbers is exactly this kind of signal.
- Judgment SLOs compile into the Prometheus and Grafana you run. Targets, measurement windows and breach actions are declared in an OpenSRM manifest. Today the reversal rate is overrides ÷ decisions, computed from Prometheus counters.
- The flagship case: measuring a third-party AI SRE. Every triage, suppression or remediation it makes becomes a verdict, the on-call’s overrides settle them, and its reversal rate is held to a target.
- NthLayer executes nothing. Its respond worker only proposes; a human approves and acts. When a judgment SLO breaches,
reduce_autonomyis the breach action declared in the manifest, and your platform dispatches it. - External to the agent. NthLayer doesn’t instrument the agent’s internals or ask the agent to grade itself. The manifest declares what reliable means; the measurement comes from outside the agent. That is the difference from a score assigned automatically when a thread ends: a verdict settles on what happened afterwards.
Sources
- LangChain, State of Agent Engineering (survey of 1,340 respondents, fielded Nov–Dec 2025). langchain.com/state-of-agent-engineering
- Dynatrace, The pulse of agentic AI: Building trust in autonomous operations from pilot to production (survey of 919 senior leaders and decision makers), January 2026. Press release, 22 Jan 2026 · Report (PDF)
- Gartner, “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027”, press release, 25 June 2025. gartner.com
- Brendan Burns, “Safe Evaluation and Rollout of AI Models”, plenary talk, SREcon25 Americas, 25 March 2025. Quoted remarks as reported in Beth Pariseau, “SREs map uncharted territory with LLMOps”, TechTarget, 26 March 2025. techtarget.com · Session abstract
- Alex Palcuie, “Can Claude Fix Itself? Using LLMs for Incident Response”, QCon London, 17 March 2026. Session page · Video and transcript (InfoQ), the source of the quoted wording · also reported in Tim Anderson, “Fixing Claude with Claude: Anthropic reports on AI SRE”, The Register, 19 March 2026. theregister.com
- Charity Majors, interviewed by Beth Pariseau, “Charity Majors on AI observability and the future of SRE”, TechTarget IT Ops Query podcast, 20 March 2025. techtarget.com
- Gartner (Daryl Plummer), “Gartner Unveils Top Predictions for IT Organizations and Users in 2025 and Beyond”, press release, Gartner IT Symposium/Xpo, 22 October 2024. gartner.com
- Niall Murphy, SREcon25 Americas, March 2025, as reported in Beth Pariseau, “SREs map uncharted territory with LLMOps”, TechTarget, 26 March 2025. techtarget.com
- Anton Baturan and Zichuan Xiong, “AI for Ops vs. Ops for AI”, Thoughtworks Insights, 27 November 2025. thoughtworks.com
- Orit Golowinski, “Why Your AI Governance Is Holding You Back, and You Don’t Even Know It”, The JetBrains AI Blog, 19 March 2026. blog.jetbrains.com
- Stanford HAI, The 2026 AI Index Report, top takeaways (“Documented AI incidents rose to 362, up from 233 in 2024”; OSWorld task success from 12% to ~66%). The 55% rise is our calculation, not Stanford’s figure. Replaces the 2025 edition cited in March. hai.stanford.edu
- OpenTelemetry semantic conventions v1.38.0 release notes (introduces the GenAI Evaluation Event), 29 October 2025. github.com/open-telemetry/semantic-conventions
- “Semantic Conventions for Generative AI Agentic Systems (gen_ai.*)”, opened 20 August 2025 as semantic-conventions Issue #2664, since moved to semantic-conventions-genai Issue #35. github.com/open-telemetry/semantic-conventions-genai
- OpenSLO specification, v1. github.com/OpenSLO/OpenSLO · checked November 2026: commit history (2026 commits are documentation and website work, September 2026)
- SLO generators reviewed: Sloth github.com/slok/sloth · Pyrra github.com/prometheus-community/pyrra · Google SLO Generator github.com/google/slo-generator. Checked November 2026: latest releases Sloth v0.16.0 (4 April 2026), Pyrra v0.10.2 (18 September 2026), SLO Generator v2.6.0 (19 February 2024).
- OpenSRM core specification (v2 draft), the open specification NthLayer is built on. Disclosure: NthLayer’s author maintains it; it is cited to describe what it models, not as evidence about the market. github.com/rsionnach/opensrm
- Grafana Labs, “Survey: AI Adoption Is Outpacing Trust as Enterprises Push Toward Autonomous Operations”, press release on The 2026 State of Observability Report (ETR survey of 300 IT and engineering decision-makers, June 2026), 19 October 2026. grafana.com
- Datadog, “Improve AI agent quality with Bits Evals”, 9 June 2026. datadoghq.com · “DASH 2026: Guide to Datadog’s newest announcements”, 9 June 2026. datadoghq.com
- Dave Hickman, “Splunk Observability at .conf26: Optimize AI at Scale”, Splunk blog, 16 September 2026. splunk.com
- Dynatrace, “Evaluate LLM and agent quality in Dynatrace AI Observability”, Dynatrace blog, 11 June 2026. dynatrace.com
- Microsoft Learn, “Track Incident Value in Azure SRE Agent”, 28 October 2026. learn.microsoft.com · “Review agent insights in Azure SRE Agent”, 28 October 2026. learn.microsoft.com
- AWS, “Announcing General Availability of AWS DevOps Agent”, AWS Cloud Operations Blog, 31 March 2026. aws.amazon.com
- Nir Gazit, “Traceloop is joining ServiceNow”, Traceloop blog, March 2026. traceloop.com
- Kamal Hathi, “Making AI Trustworthy and Observable in Real-Time: Cisco Announces Intent to Acquire Galileo”, Cisco Blogs, 9 April 2026 (updated 22 May 2026: acquisition completed). blogs.cisco.com
- Steve Tack, “Dynatrace completes acquisition of Arize to advance full-lifecycle AI observability”, Dynatrace blog, 1 October 2026. dynatrace.com
- Ian Webster and Michael D’Angelo, “Promptfoo is joining OpenAI”, Promptfoo blog, 9 March 2026. promptfoo.dev
- Artificial Analysis, “ITBench-AA: Agentic benchmark to test models on real-world SRE tasks”, 27 May 2026. artificialanalysis.ai · Leaderboard, checked 2 November 2026
- Aoyang Fang et al., “OpenRCA 2.0: From Outcome Labels to Causal Process Supervision”, arXiv:2606.27154, 25 June 2026 (revised 30 June 2026). arxiv.org
- Adam Frank, “Do AI SRE agents actually work? An honest capability assessment”, Rootly blog, 15 September 2026. rootly.com
- OpenTelemetry semantic conventions v1.42.0 release notes (moves
gen_ai.*to a dedicated repository), 12 June 2026. github.com/open-telemetry/semantic-conventions · semantic-conventions-genai repository (no tagged release, checked 2 November 2026) · evaluation event definition - OpenInference specification: “Annotations and Evaluations” (added 31 July 2026) github.com/Arize-ai/openinference · “Decision Spans” (added 1 October 2026) github.com/Arize-ai/openinference
- Anita Rao et al., NIST AI 800-4, “Challenges to the monitoring of deployed AI systems”, Center for AI Standards and Innovation, NIST, 6 March 2026. nist.gov
Sources 17–32 were added in the November 2026 update, and source 11 replaced the 2025 Stanford report; the March PagerDuty 2024 figure was removed as out of date. Links were checked when this list was compiled. Some publishers block automated requests, so a link that fails in a script may still open in a browser.