Runbook / ops

Anvil is built to be operated, not just run. This page summarizes the operational model — modes, startup discipline, failure handling, and recovery. Concrete risk limits, thresholds, and credential handling are deployment-specific and live in the operator's private configuration, not in public docs.

Operating modes

A runner always declares its mode explicitly; the mode is journaled in the run metadata so a recorded run is unambiguous.

Every mode runs the same reducer over the same journal format. That is what makes a PAPER or BACKTEST result a faithful predictor of LIVE behavior.

Pre-flight discipline

Before a LIVE start, the operator confirms — at minimum — that infrastructure and time sync are healthy, configuration validates against its schema, connectivity to each venue is established, and the journal's last run closed cleanly (or was reconciled). Anvil refuses to advance into normal operation until startup reconciliation succeeds.

Stop states

Risk controls express themselves as stop states of escalating severity — from a soft throttle that backs off order activity, through scoped hard stops that halt new risk while letting existing exposure be managed, up to a full flatten-and-halt. Each transition is journaled. The specific triggers and thresholds are configured per deployment; the platform guarantees that a stop state, once entered, cannot be silently bypassed by strategy code.

Incident playbooks

The runbook carries playbooks for the failure modes a live trading system actually hits, each with a detect → contain → recover shape:

• Market-data or websocket disconnects
• Exchange rate-limiting and backoff
• Order-rejection spikes
• Reconciliation divergence (journal vs. venue)
• Duplicate or unexpected fills
• Loss-velocity events

The common thread: contain first (enter the appropriate stop state), then use the journal to establish exactly what happened before resuming.

Recovery

Because the journal is the source of truth and the reducer is pure, recovery is just replay:

1. Load the latest checkpoint snapshot (snapshots bound replay time).
2. Replay the journal tail from the snapshot to rebuild exact state.
3. Run startup reconciliation against the venue.
4. Resume, or remain stopped if reconciliation fails.

There are no partial writes to untangle and no manual state reconstruction — the state that comes back is hash-identical to the state that was lost.

The operator console

anvil-tui is a read-only, out-of-process terminal console. It renders live runner health and metrics by reading the monitoring endpoints — it has no ability to mutate runtime state, place orders, or change configuration. Observation, never intervention.

# Point the console at a running runner's metrics endpoint.
anvil-tui --endpoint http://127.0.0.1:9090

If the runner is unreachable the console shows a degraded state rather than guessing; it never fabricates data.

Observability

Anvil exposes Prometheus metrics and health endpoints and emits structured JSON traces across the runtime via the tracing stack, so the operator console, dashboards, and alerting all read from the same signal source.