AtlasBurn
DocsPlatformIntegrationsOperationsReference

Auto-Recovery & Manual Override

When a guardrail trips, you have two paths back online

Every kill flag carries a TTL so the platform heals itself, and the dashboard provides a one-click manual override for when you need traffic restored immediately.

Auto-Recovery (TTL expiry)

When a guardrail layer triggers, the ingestion server writes a flag to the global GUARDRAIL_FLAGS Cloudflare KV namespace: 2 hours for SUSPENDED, 30 minutes for THROTTLED. Budget breaches flag the whole organization; loops flag only the feature that looped (see What gets blocked).

  • Every edge worker reads the flag on every request — no polling, no propagation delay.
  • When the TTL expires, the flag lapses and traffic resumes everywhere, at the edge and in the SDK gate.
  • No engineering action is required for the system to come back online.

Auto-recovery is a safety net, not a strategy

If a flag fires twice in 24 hours, fix the root cause. The TTL is designed to absorb a single incident, not to mask a misbehaving agent in production.

Manual Override (instant resume)

When you have already diagnosed the issue and want traffic back immediately, open Safety Guardrails in the dashboard. There are two buttons:

  • Resume on a paused feature clears only that feature. The rest of the project is untouched.
  • Resume all traffic clears the organization flag and every paused feature at once.

Both call the enforcement endpoint, which removes the flags from Cloudflare KV:

Manual override under the hood
http
POST /api/guardrails/enforce
Authorization: Bearer <firebase-id-token>
Content-Type: application/json

{
  "apiKey": "abn_xxxxxx",
  "action": "ACTIVE",
  "featureId": "support-chat"   // optional: omit to resume everything
}

Because the worker reads KV on every request, the next call after the resume sees no flag and forwards normally. Global propagation is sub-second.

Override via toggle configuration

For programmatic recovery (CI hooks, on-call runbooks), the same effect is achieved by flipping the org's enforcement toggle in the dashboard. The toggle calls the same /api/guardrails/enforce endpoint and clears KV identically.

Audit trail

Every kill and manual override is logged with the timestamp, the triggering layer, the observed value against its limit and, for loops, the feature that was paused. Manual resumes record who clicked them.

Next steps