NthLayer

Your dashboards say it’s up. NthLayer tells you whether it was right.

Agents, rules and people make calls that no test can check: a triage, an approval, a release. NthLayer records each one, waits for the outcome, and turns how often they hold up into SLOs and alerts in the Prometheus and Grafana you already run.

pip install nthlayer-generate
See how it measures an AI SRE →

Verdicts · recent · example data AI SRE reversal rate, 7d: 9.0% target ≤ 5%
DecidedDecided byDecisionConfidenceOutcome
Today 02:14 ai-sreAI SRE Marked the fraud-detect p99 latency alert as noise and suppressed it 0.92 overriddenby on-call, in Slack
Today 02:16 nthlayer-respondremediation Proposed rollback of fraud-detect to model v2.2 (needs human approval) 0.74 pendingoutcome pending; not counted in the AI SRE’s rate
Yesterday release-gaterule engine Approved deploy payments v4.12 — confirmed
Oct 26 refund-approvermodel Approved refund for order #88213 0.67 overriddenby a reviewer, in Jira
Oct 21 a.murphyhuman approver Approved schema migration orders_v7 — expiredno outcome within 7 days
Each row is a verdict: NthLayer’s record of one decision. The decision is written once; its outcome is settled once, later.

Back pressure for decisions that have no compiler.

Pipelines scale by turning judgment into verification. A type checker, a test suite or a simulator says no, and the loop moves on.

Whatever can’t be verified that way is still a decision: a release, a migration, a destructive API call, an alert triage. Most systems never measure whether those decisions hold up. NthLayer does, and shows how much of your system still runs on judgment.

Starting with your AI SRE

In operations, the evidence shows up on its own. The alert was real or it wasn’t. The service recovered or it didn’t. And the on-call who sees it is already the reviewer.

So every decision an AI SRE makes (to triage, suppress or remediate) is recorded, then settled: confirmed when a positive outcome is posted, overridden when someone on the team rejects it where they already work. When its decisions stop holding up, its judgment SLO (a target for how often they hold up) breaches, and the manifest’s declared breach action, reduce_autonomy, tells your AI SRE’s platform to rein it in.

Walk through the full scenario →

  1. The AI SRE marks the fraud-detect p99 latency alert as noise and suppresses it. Confidence 0.92.

    verdict recorded

  2. Error-budget burn rate on fraud-detect passes 10×. NthLayer’s workers group the breach into a situation and open a case.

    evidence arrives

  3. The on-call rejects the suppression in Slack. The override adapter binds it to the 02:14 verdict by decision ID, and its outcome settles as overridden. A confident decision that’s overridden counts as a high-confidence failure.

    high_confidence_failure above target

  4. Over the week, overrides push the AI SRE’s reversal rate to 9.0% against a 5% target. The judgment SLO breaches, its alert fires in Alertmanager, and the manifest’s declared breach action, reduce_autonomy, is dispatched by your AI SRE’s platform. NthLayer executes nothing.

    reduce_autonomy declared

Illustrative scenario.

What it measures

Classical observability tells you a service was up, fast and cheap. These tell you whether its decisions were right.

  • Reversal rate

    How often is a decision overturned? Today: overrides ÷ decisions, from Prometheus counters exposed by your AI SRE (via its shim) and the override adapter. The spec also counts reconsideration by the service and downstream feedback.

  • High-confidence failure rate

    How often is a high-confidence decision wrong? Those are the decisions that skip the review that would have caught them.

  • Calibration

    When the service says it is 80% confident, is it right 80% of the time?

Override rate
How often does a human step in on any decision, right or wrong, and is that rising or falling?
Feedback latency
How long until you can know whether a decision was right? That sets how quickly the other signals can catch a regression.
Segments, stability, audit backlog
Does quality hold across customer tiers (segments) and model updates (stability), and is anyone still sampling decisions for review (audit backlog)?

Values shown are example data from the walkthrough scenario.

How it works

Declare what “right” means, compile it into the monitoring you already run, then record decisions against it.

NthLayer works in seven stages. Define (the OpenSRM manifest) and Generate (nthlayer-generate) happen at build time and are stable. Observe, Measure, Correlate and Learn run at runtime. Respond also runs at runtime but only proposes: it records what it would do and opens a case, and a human acts on your systems.

Declare

In the manifest, any service that makes decisions has the type ai-gate, even when the decider is a rule engine or a human approver.

The manifest follows the open OpenSRM spec. It carries classical SLOs for whether the service is up, and judgment SLOs for whether its decisions are right.

service.yaml (excerpt) 
spec:
  service: { name: ai-sre, type: ai-gate }
  judgment_slo:
    - service: ai-sre
      judgment_type: reversal_rate
      measurement: { window: 7d, source: lineage }
      target: { maximum_reversal_rate: 0.05 }
    - service: ai-sre
      judgment_type: high_confidence_failure
      measurement: { confidence_threshold: 0.9, window: 30d }
      target: { maximum_failure_rate: 0.02 }
See a full manifest →

Compile

nthlayer apply turns the manifest into Prometheus recording rules, dashboards, docs, Backstage entities and two kinds of alert: breach, and not-evaluable. A not-evaluable alert fires when a bound can’t be measured at all, so it is reported rather than treated as healthy. nthlayer apply is deterministic and needs no runtime.

The same step runs in CI through the GitHub Action, so a judgment SLO without a data source fails the build before it fails in production.

$ nthlayer init
$ nthlayer apply service.yaml
.github/workflows/reliability.yaml 
- uses: rsionnach/nthlayer@v1
  with:
    manifest: service.yaml

Run

The runtime is three tiers that talk only over HTTP, plus an adapter for human overrides. Every decision becomes a content-addressed record of what was decided, on what assumptions and with what confidence. The decision is written once; its outcome is settled once, later.

Decisions arrive as an HTTP POST to core, as a plain verdict or a CloudEvents envelope. Overrides go to the adapter, which binds each one to its verdict by decision ID.

  • Tier 1nthlayer-coreVerdict store, case management and the API.
  • Tier 2nthlayer-workersFive runtime modules: observe and measure your SLOs and judgment SLOs against Prometheus, correlate related events into situations, respond by proposing actions for a human to approve and run, and learn by resolving outcomes and proposing manifest changes for human review.
  • Tier 3nthlayer-benchTerminal UI for the moments that need a human. Reads as prose, not graphs.
  • Adapternthlayer-override-adapterIngests human overrides from Slack, Jira, a review UI or anything that can send a webhook, and binds each one to its verdict by decision ID.
POST /verdicts (example) 
{
  "id": "ai-sre-0412-a1",
  "service": "ai-sre",
  "created_at": "2026-10-28T02:14:00Z",
  "producer": { "system": "ai-sre" },
  "subject": { "type": "triage", "ref": "fraud-detect" },
  "judgment": {
    "action": "custom",
    "confidence": 0.92,
    "reasoning": "suppress: matches a known flapping pattern"
  },
  "outcome": { "status": "pending" },
  "metadata": { "custom": { "vendor_action": "suppress_alert", "alert": "fraud-detect p99 latency" } }
}

Where NthLayer sits

Beside the loop, not inside it. NthLayer measures your AI SRE the same way no matter who built it, from records you own.

Can a machine say no before it ships?

Yes

  • Compilers
  • Type checkers
  • Test suites
  • Simulators
  • Offline evals

Use them. They catch it before production.

No: everything else

Your AI SRE, rule engines and human approvers post decisions to NthLayer over HTTP. Human overrides from Slack, Jira or a webhook feed outcomes back into NthLayer. NthLayer sends recording rules and alerts to Prometheus, Alertmanager and Grafana, the stack you already run, and reads SLO series back from Prometheus.

What NthLayer needs from your AI SRE

  1. A stable decision ID for every decision. It becomes the verdict’s id, so it must be unique across NthLayer; your shim (the small adapter between your AI SRE and NthLayer) prefixes it (for example ai-sre-…).
  2. The decision as one of NthLayer’s actions: approve, reject, flag, escalate, defer or custom. The vendor’s own action (“restart the container”) goes in the reasoning or metadata.
  3. A confidence between 0 and 1. Without it, high-confidence failure and calibration can’t be measured.
  4. The decision ID wherever a human can reject it. The vendor never calls NthLayer: Slack, Jira or a review tool can post as they are, through a webhook with a field mapping. An override binds only while the verdict is pending, so it has to arrive before the outcome does.
  5. What happened next, as a verdict pointing back at the decision. This is how NthLayer learns a decision was right: with no downstream verdict it can never be confirmed, and it expires after seven days. The alert that triggered the decision goes in as context, not as a parent.
  6. Your shim’s part, not the vendor’s: post each verdict to core with the ai-gate’s service name, a subject type (triage, recommendation or remediation), the producer, a time-zoned created_at and a pending outcome, and emit the decision counters with a confidence bucket so the high-confidence failure rate can be computed.
  • Not your AI SRE’s replacement.It doesn’t run your agents. Every decision, theirs or your vendor’s, is recorded and judged the same way. Its own response agents only propose; a human approves and runs.
  • Beyond operations.The same record works for any decision no test can check: refunds, cleanups, schema changes, releases.

Status

All open source. Each repo keeps its own changelog, and an integration test across the three runtime tiers runs nightly.

Get started

Start with the compiler. It needs nothing deployed.

pip install nthlayer-generate 
$ nthlayer init
$ nthlayer apply service.yaml
Run the full runtime instead 
$ pip install nthlayer   # generate + core + workers + bench
$ nthlayer-core serve
$ nthlayer-bench --core-url http://localhost:8000