What to expect

Observability Unified connects the evidence engineers and AI agents need to debug production software. Engineers can follow a user action into its backend trace and related logs, replay, or CPU profile in the dashboard's Connected Rail. Agents can retrieve evidence from the same graph through the MCP server. This page shows what the dashboard surfaces once instrumentation is in place.

The Connected Rail

Every detail page in the dashboard mounts a right-side rail with four sections:

• Up — the parent entity (trace ← span, session ← usage event, etc.)
• Across — sibling signals sharing the same identity key (other spans in the same trace, logs from the same session)
• Down — derived data (pprof profile for a trace, off-CPU profile for a span)
• Related — non-identity-based neighbors (the click that caused this trace, alerts firing on this service)

When a section has no neighbors, the rail renders an informative-absence message explaining why — never a silent empty section. The platform's contract is that "no data" should always tell you what's missing and how to populate it.

Scenario A — alert → trace → flame graph → cohort → session → replay

The headline product test. From a paged alert:

Six clicks across the entire platform. The platform's claim is that every neighbor at every step is on the rail.

Illustrative scenario. The "bound Analysis narrative" in step 1 and the "(243)" other-traces count in step 3 describe a production-shaped incident, not what the seed produces. The sample seed creates one profile trace, and its alert rules fire without a bound analysis (analyses are generated for real alerts that fire against your own traffic). Expect the same click path with smaller numbers.

Interaction ID to CPU

The browser SDK mints a single interaction_id for a frontend action and injects it as x-obs-interaction on outbound requests. Backend SDKs copy that value onto the active span as obs.interaction.id, and correlated logs, AI calls, or MCP tool context inherit it from the span context.

CPU and off-CPU profiles are joined through traces rather than storing interaction_id directly on every sample. If profiling is enabled and samples are labeled with trace IDs, the dashboard can follow:

That is the accurate version of "one ID from frontend to CPU": one interaction ID anchors the user action, and the trace it caused carries the investigation into profiling data.

Scenario B — AI cost spike → user → session → trace

A different entry point exercising the same identity skeleton:

1. AI dashboard shows a cost spike (SPANS OVER TIME chart peaks). The Sessions view ranks the heavy spender at the top by cost.
2. Click the 👤 user-id chip on the heavy spender's row → user detail page.
3. User detail page shows the user's Identity card + a Connected Rail with "Latest session", "Recent traces", "Recent AI calls". The rail surfaces the count-collapsed link for a session with N traces / M AI calls.
4. Click "Latest session" → Replay tab scoped to that session, showing the session's interactions linked to their traces.
5. Click an interaction → trace waterfall for the trace that click caused. Connected Rail's "Click that caused this trace" closes the loop back to the originating click.

The seed (pnpm seed) plants a "Heavy Spender (seed)" user with 8–9 high-cost claude-3-5-haiku calls so this walkthrough is reproducible without writing real AI traffic.

Scenario B2 — agent action graph → MCP investigation

Use the seeded recommendation agent run to investigate a failed budget evaluation. In a seeded local runtime, follow these steps with the default read-only MCP tools.

1. Call recent_traces with status error and query budget_guard_failed. Open the recommendation agent trace with get_trace.
2. Locate the eval.recommendation_budget_guard span. Call connected_signals with kind span and ID <traceId>:<spanId> to find its linked eval and action.
3. Call get_action with the eval action ID and use its rootActionId to call get_agent_run. The action has status error and the run has status success. The evidence shows a failed budget check within a completed run.
4. Cite the trace, failed span, and eval action. Share the dashboard trace or run link so an engineer can inspect the same evidence.

In this seed the span rail links to a replay session and an AI call. It reports no correlated logs or profile for the failed eval span.

Scenario C — futex contention via off-CPU flame graph

Validates the kernel-level layer:

1. Trace shows an unexplained pause inside a span (no child spans, on-CPU profile shows little activity).
2. Rail's "Down → 🔥 off-CPU profile" leads to an icicle flame graph that surfaces futex_wait_queue ↑ pthread_mutex_lock ↑ inventory_pool::checkout taking 84% of off-CPU time.
3. Root cause: a single pool-wide mutex serializing every checkout.

This Scenario C off-CPU path currently runs only against the docker-compose demo with Beyla feeding pprof. The dashboard code paths are live; the synthetic seed doesn't generate off-CPU pprof blobs for this path.

Scenario D — deploy regression → cohort lift → counterfactual verification → guarded rollback

Validates the AutoSRE causal ontology (RFC 0013) end-to-end across both human dashboard surfaces and MCP agent tools:

1. Incident triage bundle (intent: "triage_incident"): From a firing checkout alert, the collector separates the deepest failing span (bundle.autosre.faultLocus, an object with nodeId, spanId, relationshipCertainty, and isBlackoutSuspected; locating a span proves nothing causal, so the proof tier stays unverified_locus until a change is correlated) from ranked bundle.autosre.candidateChanges, contentionCandidates, and priorFindings (including chg_scenario_d_deploy_241 and fnd_scenario_d).
2. Traffic-normalized cohort lift: Calling get_change_blast_radius compares per-request error rates between v2.4.1 and v2.4.0 across minimum sample sizes (N >= 50 per cohort), confirming cohortVerdict: "isolated_to_new_version" (relationshipCertainty: "inferred", causalProofTier: "cohort_rate_isolated") and avoiding the traffic-share fallacy.
3. Code-path corroboration: Stack traces on the failing span (src/checkout/charge.ts) intersect touchedPaths on v2.4.1, adding an "exact" structural link while keeping causal proof separate from link provenance.
4. Counterfactual verification: Recording a finding and attaching a paired baseline-failing run on v2.4.1 plus one candidate-passing run on v2.4.0 (the seeder creates one passing run; the counterfactual route accepts requiredTrials >= 1) upgrades the finding to causalProofTier: "counterfactual_verified".
5. Pre-execution remediation gate: Proposing a rollback (propose_remediation_action) is decided by the finding's causalProofTier and your requestedAutonomy. For rollback_deploy and toggle_feature_flag alike: cohort_rate_isolated or counterfactual_verified earns an autonomous grant (a single-use grantToken with status: "grant_issued"), corroborated_candidate parks the proposal in the Autonomous Review approval queue (pending_human_approval), and unverified_locus is blocked_by_policy, which is recorded as proposal status: "rejected". A 3-per-hour circuit breaker per finding fingerprint and action kind applies on top, and the target-state compare-and-swap (expectedCurrentState: { version: "2.4.1" }) is checked when the runner claims the grant. Proposals move through pending_human_approval, grant_issued, claimed_executing, completed, failed_execution, execution_timed_out, expired, and rejected. See AutoSRE for the full matrix and the approval-to-execution handoff.

Both pnpm seed and pnpm e2e:autosre plant this complete Scenario D dataset automatically.

Per-tab walkthrough

When you should expect informative absence

The rail is honest about what's missing. You'll see explicit "—" messages when:

• No interaction_id on a span — the trace wasn't caused by a browser click (cron, queue consumer, retry). The "Originating click" section explains this.
• No pprof profile — the producing service hasn't wired startProfiler() or an eBPF agent. The Down section explains how to populate.
• No rrweb replay — the session had no real browser to capture chunks. The Replay tab tells you to visit /playground and click "Start replay" to capture one.
• Alert/analysis topic links — alerts and analyses don't carry identity columns; they relate by topic, not identity. The rail's Related section explains this is by design.

These are part of the design — empty data should always be explained, never silent.

Production deployment caveats

• The migration runner has a --remote mode; first-run on a partially-migrated production DB needs manual backfill (see Installation).
• The every-minute analyses cron uses a 90s claim/lease to prevent overlap on long-running LLM narrative passes (RFC 0002 Stage 4 follow-up).
• The pprof receiver returns 422 on decode failure (corrupted blobs surface to the agent instead of landing silently in R2).
• The connected-routes endpoint returns 400 on unknown entity kinds (catches client-side URL building bugs).

Recent reliability & AutoSRE behavior

The October 2026 updates (RFC 0013) and earlier May 2026 hardening added several user- and agent-visible guarantees:

• Orthogonal epistemic badges on the Connected Rail: Every causal link can display both link provenance (exact link, inferred link, temporal heuristic) and causal proof tier (fault locus, corroborated candidate, cohort isolated, counterfactual verified).
• Traffic-normalized blast radius & noisy-neighbor checks: GET /internal/changes/:id/blast-radius normalizes error rates per request across service.version / vcs.commit.id / feature_flag.variant cohorts and flags shared downstream contention (verdicts shared_degradation_new_version_aggressor / shared_degradation_external_cause).
• Pre-execution remediation gate & recovery watchdog: High-risk production actions require either cohort/counterfactual proof or 1-click human approval in Autonomous Review, enforce CAS on expectedCurrentState, and track recoveryStatus separately from causalVerificationStatus.
• Dashboard loader & stream stability: Telemetry and AI dashboards abort stale loaders on tab/filter switches, replay chunks load progressively, and live-tail streams enforce project isolation end-to-end.