Production operations

This page covers production deployment patterns for the standalone Node collector backed by Postgres and S3-compatible object storage. The Cloudflare Workers path remains supported, but the Node collector is the clearest fit for container platforms and private infrastructure.

1. Network and reverse proxy

The recommended layout places the collector behind TLS, keeps OTLP ingest routes reachable, and protects dashboard/internal query routes with your normal auth boundary.

Nginx

server {
    listen 443 ssl http2;
    server_name obs.my-app.com;

    ssl_certificate /etc/letsencrypt/live/obs.my-app.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/obs.my-app.com/privkey.pem;

    gzip on;
    gzip_types application/json application/x-protobuf text/plain text/css;

    location /v1/ {
        proxy_pass http://127.0.0.1:8790;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        client_max_body_size 10m;
    }

    location /internal/ {
        proxy_pass http://127.0.0.1:8790;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location / {
        proxy_pass http://127.0.0.1:5173;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        try_files $uri $uri/ /index.html;
    }
}

Set TRUST_PROXY=true on the collector (and only when it is reachable solely through this proxy). Without it the dashboard login rate limit and client-IP handling key on the proxy's address instead of the real client, so one noisy client can lock everyone out of login.

CORS: the all-in-one image allows localhost and 127.0.0.1 on ports 5173, 5174, and 3000 by default. The standalone Node collector has no default; set ALLOWED_ORIGINS (comma-separated) to your dashboard and app origins, for example https://obs.my-app.com.

Caddy

obs.my-app.com {
    reverse_proxy /v1/* 127.0.0.1:8790 {
        header_up Host {host}
        header_up X-Real-IP {remote}
    }

    reverse_proxy /internal/* 127.0.0.1:8790 {
        header_up Host {host}
        header_up X-Real-IP {remote}
    }

    reverse_proxy /* 127.0.0.1:5173 {
        header_up Host {host}
        header_up X-Real-IP {remote}
    }
}

Set TRUST_PROXY=true on the collector (and only when it is reachable solely through this proxy). Without it the dashboard login rate limit and client-IP handling key on the proxy's address instead of the real client, so one noisy client can lock everyone out of login.

2. Postgres tuning

Each collector instance uses a persistent client-side pool. Start with the default PG_POOL_MAX=10 for low to moderate traffic (1–50 spans/second). For high-throughput environments (>500 spans/second), scale to 30 or 50, making sure the total across replicas stays below the database server's max_connections in postgresql.conf.

Set a statement timeout so expensive dashboard or analysis queries cannot lock the database indefinitely:

PG_STATEMENT_TIMEOUT=30000

3. Object storage retention

Replay chunks (.json) and pprof profiles (.pprof) live in S3-compatible object storage. Match bucket lifecycle retention to the database retention window (RETENTION_HOURS=72). For a 72-hour hot-debugging window, save the following three-day expiration rule as lifecycle-policy.json:

{
  "Rules": [
    {
      "ID": "AutoDeleteOldTelemetryBlobs",
      "Status": "Enabled",
      "Filter": {
        "Prefix": ""
      },
      "Expiration": {
        "Days": 3
      }
    }
  ]
}

Apply it with:

aws s3api put-bucket-lifecycle-configuration \
  --bucket obs-unified-storage-bucket \
  --lifecycle-configuration file://lifecycle-policy.json

4. Kubernetes orchestration manifests

Run collectors as stateless replicas against managed Postgres and S3-compatible storage.

Every collector process ships an in-process scheduler for the three recurring jobs the Cloudflare Worker gets from Cron Triggers (due analyses every minute, alert evaluation every 5 minutes, retention cleanup hourly). It is on by default, so a plain multi-replica Deployment would run each job once per replica. The manifests below keep exactly one owner: serving replicas set MAINTENANCE_ENABLED: "false" via the shared ConfigMap, and a separate single-replica obs-maintenance Deployment re-enables it. See the collector-node README for the MAINTENANCE_* settings.

ConfigMap and Secrets (obs-config.yaml)

apiVersion: v1
kind: ConfigMap
metadata:
  name: obs-config
  namespace: observability
data:
  BLOB_STORE: "s3"
  S3_REGION: "us-east-1"
  S3_BUCKET: "my-production-obs-bucket"
  PG_POOL_MAX: "30"
  PORT: "8790"
  RETENTION_HOURS: "72"
  # Serving replicas must not run background maintenance; the dedicated
  # obs-maintenance Deployment below overrides this to "true".
  MAINTENANCE_ENABLED: "false"
  # Behind the reverse proxy above: login rate limit and client IP use X-Forwarded-For.
  TRUST_PROXY: "true"
---
apiVersion: v1
kind: Secret
metadata:
  name: obs-secrets
  namespace: observability
type: Opaque
data:
  # Base64 encoded connection strings, AWS keys, and passwords
  DATABASE_URL: cG9zdGdyZXM6Ly91c2VyOnBhc3NAcGctaG9zdDo1NDMyL29ic191bmlmaWVk  # postgres://user:pass@pg-host:5432/obs_unified
  S3_ACCESS_KEY_ID: QUtJQVhYWFhYWFhYWFhYWFhYWFg=                           # AKIAXXXXXXXXXXXXXXX
  S3_SECRET_ACCESS_KEY: c2VjcmV0LWtleS12YWx1ZS1nb2VzLWhlcmU=               # secret-key-value-goes-here
  INGEST_KEY: bXktc2VjdXJlLXdyaXRlLWtleQ==                                 # my-secure-write-key
  DASHBOARD_PASSWORD: bXktc3VwZXItc2VjdXJlLXBhc3N3b3Jk                    # my-super-secure-password
  # Programmatic read/admin bearer token for /internal/* (MCP server, CLI).
  # Generate a fresh random value per environment and base64-encode it:
  #   openssl rand -hex 32 | tr -d '\n' | base64
  DASHBOARD_TOKEN: <base64 of your generated token>

DASHBOARD_TOKEN is the one secret shared with programmatic clients, so keep it distinct from DASHBOARD_PASSWORD and INGEST_KEY:

• The MCP server (@obsunified/mcp-server) sends OBS_DASHBOARD_TOKEN as Authorization: Bearer; it must equal this DASHBOARD_TOKEN exactly.
• The CLI (@obsunified/cli) sends OBS_ADMIN_TOKEN the same way for keys mint, keys list, and doctor; it must also equal this DASHBOARD_TOKEN.
• Rotate all three together. The collector compares the token in constant time and only on /internal/* and GET /auth/check; ingest keys never grant read access, and the token plays no part in ingest or background maintenance.

Both Deployments below mount the same obs-secrets, so the serving replicas and the single obs-maintenance replica accept the same token. See Identity & External Access Recipes for the full header-forwarding and proxy patterns.

Serving Collector Deployment (obs-deployment.yaml)

Handles ingest and dashboard traffic only. MAINTENANCE_ENABLED is inherited as "false" from the ConfigMap, so scaling replicas up or down never changes how many times a job runs.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: obs-collector
  namespace: observability
  labels:
    app: obs-collector
spec:
  replicas: 2
  selector:
    matchLabels:
      app: obs-collector
  template:
    metadata:
      labels:
        app: obs-collector
    spec:
      containers:
      - name: collector
        image: ghcr.io/dev-sawan-ruparel/obs-unified:2.4.0  # pin a tested version; avoid :latest
        ports:
        - containerPort: 8790
        envFrom:
        - configMapRef:
            name: obs-config
        - secretRef:
            name: obs-secrets
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "250m"
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /health
            port: 8790
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8790
          initialDelaySeconds: 15
          periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: obs-collector-service
  namespace: observability
spec:
  selector:
    app: obs-collector
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8790
  type: ClusterIP

Maintenance Deployment (obs-maintenance.yaml)

The single owner of the scheduled jobs. Same image, ConfigMap, and Secret as the serving Deployment; the only differences are replicas: 1, the explicit MAINTENANCE_ENABLED: "true" override (a container env entry takes precedence over envFrom), a Recreate strategy so a rollout never has two maintenance pods alive at once, and a distinct app label so obs-collector-service never routes traffic to it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: obs-maintenance
  namespace: observability
  labels:
    app: obs-maintenance
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: obs-maintenance
  template:
    metadata:
      labels:
        app: obs-maintenance
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: maintenance
        image: ghcr.io/dev-sawan-ruparel/obs-unified:2.4.0  # pin a tested version; avoid :latest
        ports:
        - containerPort: 8790
        envFrom:
        - configMapRef:
            name: obs-config
        - secretRef:
            name: obs-secrets
        env:
        - name: MAINTENANCE_ENABLED
          value: "true"
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "250m"
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /health
            port: 8790
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8790
          initialDelaySeconds: 15
          periodSeconds: 20

Verify the split after applying both manifests:

kubectl -n observability logs deploy/obs-maintenance | grep '\[maintenance\] scheduler started'
kubectl -n observability logs deploy/obs-collector   | grep 'maintenance scheduler disabled'

On SIGTERM the maintenance pod stops arming new ticks, waits up to 10 s for an in-flight job, then closes its Postgres pool, which fits inside the 30 s grace period above. The serving Deployment's /health endpoint is unaffected by the scheduler, so the probes are identical.

Never run two or more replicas of the same image without MAINTENANCE_ENABLED=false on all but one of them. The all-in-one image and the Node collector default to MAINTENANCE_ENABLED=true, so every replica would otherwise run each scheduled job.

If you prefer a scheduler you already operate, set MAINTENANCE_ENABLED: "false" everywhere, omit obs-maintenance, and run the jobs from your own cron; do not leave the default (true) on a multi-replica serving Deployment.

5. Upgrades and database migrations

During software updates, execute database migrations before recycling rolling container deployments to prevent schema mismatch errors inside the collector instances.

Pin the image to the version you are rolling out (:2.4.0 here; use your target release) rather than :latest. Run the migrations securely on your database using a temporary migrations job container:

docker run --rm \
  -e DATABASE_URL="postgres://user:pass@pg-host:5432/obs_unified" \
  ghcr.io/dev-sawan-ruparel/obs-unified:2.4.0 \
  node /repo/apps/collector-node/scripts/migrate-pg.mjs

6. Operational checks

Before opening ingest traffic:

1. Run database migrations for the target collector storage.
2. Confirm /health responds through the reverse proxy.
3. Verify CORS from the browser origin with pnpm dlx @obsunified/cli doctor.
4. Send one synthetic trace, one log, and one usage event.
5. Confirm dashboard login and internal query routes are not publicly exposed without your intended auth.

7. Backup and restore

Applies to the all-in-one image, which keeps Postgres 15 (cluster 15/main, database obs_unified, role obs) in /var/lib/postgresql and file blobs in /data/blobs. Managed Postgres plus S3 deployments should use the provider's snapshot and bucket-versioning tools instead.

Back up (container named obs; the embedded Postgres only accepts password auth over TCP, so pass -h 127.0.0.1):

docker exec -e PGPASSWORD=obs obs \
  pg_dump -h 127.0.0.1 -U obs --clean --if-exists obs_unified > backup.sql

docker run --rm -v obs-unified-blobs:/data -v "$PWD":/backup alpine \
  tar czf /backup/blobs.tgz -C /data .

--clean --if-exists makes the dump replace the schema the new container creates on first boot. /data only holds replay recordings and pprof profiles; skip the tarball if you do not need them.

Restore into empty volumes (a new container on fresh volumes, same image tag or newer):

# 1. Restore blobs into the empty blob volume before the first start
docker run --rm -v obs-unified-blobs:/data -v "$PWD":/backup alpine \
  tar xzf /backup/blobs.tgz -C /data

# 2. Start the container once so Postgres initializes and migrations run
docker run -d --name obs -p 5173:5173 -p 8790:8790 \
  -e INGEST_KEY -e DASHBOARD_PASSWORD -e DASHBOARD_TOKEN \
  -v obs-unified-db:/var/lib/postgresql -v obs-unified-blobs:/data \
  ghcr.io/dev-sawan-ruparel/obs-unified:<version>

# 3. Load the dump, then restart so the collector drops cached state
docker exec -i -e PGPASSWORD=obs obs psql -q -h 127.0.0.1 -U obs obs_unified < backup.sql
docker restart obs

Ingest keys live in the database, so they come back with the dump (a key you revoked stays revoked). INGEST_KEY, DASHBOARD_PASSWORD, and DASHBOARD_TOKEN come from the new container's environment; changing DASHBOARD_PASSWORD invalidates existing browser sessions.

8. Alerts and webhooks

Alert rules are evaluated by the maintenance scheduler every 5 minutes by default (MAINTENANCE_ALERTS_INTERVAL_SECONDS on the Node collector; see the maintenance split in the Kubernetes section). Channels are webhooks or email (below). Webhook channels are admin-supplied URLs that the collector POSTs to from inside its own network, so they are validated:

• Only plain http(s) URLs to public hosts are accepted. Loopback, RFC 1918 ranges, link-local (cloud metadata) addresses, and embedded credentials are rejected. Hostnames are checked as written, so a name that resolves to a private address is not caught at validation time.
• To post to an internal receiver (for example an in-cluster Alertmanager or a local webhook sink), set ALERT_WEBHOOK_ALLOW_PRIVATE=true on the collector. The Workers collector honours it today; the Node/Docker image requires image 2.2.2 or newer, which forwards the flag (earlier Node images ignored it even though the rejection message mentions it).
• Webhook URLs are redacted in logs and span attributes (only the origin and first path segment are kept), since Slack and PagerDuty style hooks carry secrets in the path.
• The seeded sample data (OBS_LOCAL_SEED=true) creates alert rules that point at a placeholder hooks.example.com webhook URL. Since collector 2.2.2 the seeder creates them disabled (enabled: false), so they do not fire. Enable them only after pointing them at a real receiver.

Managing rules over HTTP

Rules belong to a project and are managed through the collector's /internal/alerts/* routes with the dashboard token (Authorization: Bearer $DASHBOARD_TOKEN). The project is chosen with X-Project-Id (omitted: default, the project the INGEST_KEY bootstrap key belongs to).

AlertRuleInput is { name, signal, query, threshold, comparison, windowMins, channels, enabled?, pendingSecs?, analysisId? }. Rule names are not unique, so a script that manages rules idempotently lists the rules, matches on name, and then PATCHes or POSTs.

• signal: spans, logs, usage, ai (count records in the window) or metrics (see below).
• comparison: >, >=, <, <=, ==, != (exact numeric comparison).
• windowMins: integer minutes, at least 1 (at most 1440 for metrics rules).
• pendingSecs (default 0, at most 86400): how long the condition must hold on consecutive evaluations before the rule fires, like Prometheus for. The first matching evaluation records pendingSince; the hold continues only while every evaluation matches. It starts over after an evaluation that does not match, after an update that changes how or whether the rule evaluates (signal, query, threshold, comparison, windowMins, pendingSecs, enabled, analysisId; a rename does not), after a gap between evaluations longer than two evaluation intervals (missed ticks, a stopped collector), and on the Node collector after a restart. Resolution is immediate. Evaluations happen on the maintenance cadence, so a rule fires on the first evaluation at least pendingSecs after the hold started: with the default 5-minute cadence pendingSecs: 30 means "two evaluations in a row". Set MAINTENANCE_ALERTS_INTERVAL_SECONDS (Node collector) to a value well below your shortest pendingSecs; the Workers collector evaluates every 5 minutes.
• Listed rules carry their health: notificationDelivered (whether the channels accepted the notice for currentState; false means a transition is still being retried, null means the rule never changed state) and lastEvaluation ({ evaluatedAt, outcome, value, error }, null when no evaluation is kept; error is the failure reason for an error outcome, one line, at most 500 characters, with URLs replaced by <url>). For example "notificationDelivered": false, "lastEvaluation": { "evaluatedAt": "2026-10-31T10:05:00.000Z", "outcome": "error", "value": 0, "error": "metrics query matches 1200 series (limit 1000); narrow it with attributes or serviceName" }.
• Each evaluation is recorded in /internal/alerts/evaluations with an outcome: value (compared as usual), no_data (a metrics rule with noData: "keep" found no fresh series; state unchanged) or error (the evaluation failed or timed out; state unchanged; the reason is in the collector log as [alerts-evaluator] rule evaluation failed).

Metric rules (signal: "metrics")

Metric rules read OTLP metrics ingested on /v1/metrics. The query:

How the value is computed:

• Points are taken from the window by the collector's receive time, ordered by the producer's timestamp. Each matching series is reduced on its own and the per-series results are summed (for avg, min, max and last too, so narrow those to one series with attributes when you want a single series' value).
• Staleness: for last, min, max, avg and sum, a series whose newest point is older than staleSecs is left out, so a source that was replaced (a new host.name, a new pod) stops adding its last value after 5 minutes. increase and rate keep every series with points in the window.
• No data: when no series takes part (in the numerator or the denominator), the value is 0. With the default noData: "zero" it is compared as 0: a gauge rule with == 0 fires when the gauge is not reported, and a firing > rule resolves when its source disappears. With noData: "keep" the state stays as it is until data returns. age_seconds always has a value, so noData does not apply to it; use an age_seconds rule to alert on a source that stopped.
• increase: for cumulative series, the sum of positive differences between consecutive points; a drop, or a new start timestamp, is a counter reset and the new value counts in full. The first point in the window is decided by the point's OTLP start timestamp: inside the window, the counter began there and its first value counts from 0; before the window, the first point is the baseline, even for a series Observability Unified sees for the first time (a long-running counter reaching a fresh database is not a burst). The cost of that rule: the first errors of a new attribute set on a long-running process (whose start timestamp is the process start) are a baseline, so increase sees them from the second point on. Without a start timestamp, a series first seen inside the window counts its first value from 0, and one with earlier points uses it as the baseline. No extrapolation. Delta series: the sum of the values. Gauges are treated like cumulative series.
• rate: increase divided by the window in seconds.
• age_seconds: seconds since the newest point of any matching series, regardless of the window. When no matching series has any stored point (never reported, or aged out of retention), it is the seconds since the rule was created, a finite number that grows until the threshold is crossed. So "never reported" fires threshold seconds after the rule was created, the same as "stopped reporting".
• The value stored in alert_evaluations and sent to channels is always a finite number.
• Cost bounds: a query that matches more than 1000 series, or whose selector would load more than 50,000 points in its window, fails the evaluation (recorded with outcome error, state unchanged) instead of returning a value computed from part of the data; POST /internal/alerts/rules/:id/test answers 422 with the reason. Narrow the query or shorten windowMins.

Examples:

{ "name": "postgres down", "signal": "metrics",
  "query": { "metric": "vbox_postgres_up", "aggregation": "last" },
  "comparison": "==", "threshold": 0, "windowMins": 1, "pendingSecs": 30,
  "channels": [{ "type": "webhook", "url": "https://hooks.example.com/alerts" }] }

{ "name": "call error ratio", "signal": "metrics",
  "query": { "metric": ["vbox_inbound_failed_total", "vbox_outbound_failed_total"],
             "aggregation": "increase",
             "denominator": { "metric": ["vbox_inbound_calls_total", "vbox_outbound_calls_total"] },
             "minDenominator": 5 },
  "comparison": ">", "threshold": 0.05, "windowMins": 5,
  "channels": [{ "type": "webhook", "url": "https://hooks.example.com/alerts" }] }

{ "name": "vbox stopped reporting", "signal": "metrics",
  "query": { "metric": "vbox_heartbeat", "serviceName": "vbox", "aggregation": "age_seconds" },
  "comparison": ">", "threshold": 60, "windowMins": 1,
  "channels": [{ "type": "webhook", "url": "https://hooks.example.com/alerts" }] }

Email channels

{ "type": "email", "to": ["oncall@example.com"] } sends a plain-text message on the same transitions as a webhook (firing and resolved), naming the rule, signal, value, threshold, comparison, state, evaluation time and, for analysis-bound rules, the narrative. Up to 20 plain addresses per channel. The SMTP server and credentials are collector settings, never part of the rule or the API:

• Only the Node collector (and the all-in-one image) can send email. The Cloudflare Workers collector cannot open SMTP connections, so it refuses email channels, as does a Node collector without ALERT_SMTP_HOST: creating or updating such a rule returns 400 with a message naming these variables.
• An incomplete SMTP configuration (for example a host without ALERT_SMTP_FROM, or a username without a password) stops the collector at startup.
• A send counts as delivered only when the server accepted every recipient. A failed send is logged ([alerts-evaluator] email delivery failed), recorded as an undelivered evaluation, and retried on the next evaluation, like a failed webhook.

Delivery limits that apply to every channel type:

• Only the latest transition is owed. If channels cannot be reached (SMTP down, webhook failing) while a rule fires and the rule resolves before they recover, the FIRING notice is never sent; once they recover, the RESOLVED notice is. The evaluations list still shows the firing period.
• A rule's channels are retried together. If one of several channels accepted a notice and another failed, the retry sends it to all of them again, so the channel that accepted it receives a duplicate.
• There is no resolve delay. A value that crosses the threshold back and forth on consecutive evaluations sends FIRING and RESOLVED each time; pendingSecs damps firing, not resolving.

curl -sS -X POST "$OBS_URL/internal/alerts/rules" \
  -H "Authorization: Bearer $DASHBOARD_TOKEN" -H "Content-Type: application/json" \
  -d '{ "name": "call error ratio", "signal": "metrics",
        "query": { "metric": "vbox_calls_failed_total", "aggregation": "increase",
                   "denominator": { "metric": "vbox_calls_total" }, "minDenominator": 5 },
        "comparison": ">", "threshold": 0.05, "windowMins": 5, "pendingSecs": 60,
        "channels": [{ "type": "email", "to": ["oncall@example.com"] }] }'

9. Monitoring the collector

• Health: GET /health is the only probe endpoint. There is no /metrics and no /ready route, so use /health for both readiness and liveness (as in the Kubernetes manifests above).
• Docker: the image ships a HEALTHCHECK (every 15s) that needs both the collector /health on :8790 and the dashboard on :5173 to answer. docker ps shows healthy or unhealthy.
• Logs: written to stdout/stderr as [component] message followed by an optional fields object, for example [maintenance] scheduler started { ... }. Startup lines use the [Observability Unified] prefix. Fields are printed with Node's console, so treat them as readable key/value output rather than strict JSON when building log parsers.

10. Identity and external access

Observability Unified does not contain native per-user RBAC or SSO. To deploy behind Cloudflare Access or oauth2-proxy with proper path boundaries, secrets management, header forwarding, and MCP service tokens, see Identity and external access.