Nervos Sidecar Recovery Runbook

Use this runbook when nervos-smt-v2 is scheduled or active and the sidecar is unhealthy, missing, unresponsive, wrong-version, or unable to load checkpoints.

After activation, sidecar failure is a consensus-safety blocker. NOOSChain must fail closed rather than compute roots with another engine.

First Checks

Run:

npm run noos -- state-root status --json
npm run noos -- state-root preflight --json
npm run noos -- production monitor-report --json

In the Admin GUI, check Dashboard and Membership > Consensus for:

• readiness;
• failed readiness checks;
• supervisor state;
• sidecar running;
• sidecar engine/protocol/checkpoint version;
• latest verified checkpoint;
• last error.

Sidecar Process Died

Symptoms:

• stateRootEngine.supervisor.state = degraded or stopped;
• stateRootEngine.supervisor.sidecar.running = false;
• monitor-report emits state_root_sidecar_supervisor_unhealthy.

Actions:

1. Check the sidecar binary path and permissions.
2. Check process logs for the supervisor last error.
3. Confirm the configured command exists on this host.
4. Run noos state-root preflight --json.
5. Restart the node or sidecar supervisor according to the deployment model.
6. Re-run production preflight before admitting validator traffic.

If the chain is post-activation and block execution failed, run replay/sync verification before marking any incident resolved.

Sidecar Is Unresponsive

Symptoms:

• request timeout in supervisor lastError;
• no recent lastHealthyAt;
• sidecar process may still have a PID.

Actions:

1. Stop or restart the supervised process through the node/deployment manager.
2. Confirm NOOS_STATE_ROOT_SIDECAR_TIMEOUT_MS is realistic for the host.
3. Inspect CPU, memory, disk, and checkpoint directory latency.
4. Run noos state-root preflight --json.
5. If repeated, keep the validator out of production traffic and collect a support bundle/logs.

Do not increase timeouts to hide deterministic execution stalls without benchmark evidence.

Wrong Version

Symptoms:

• readiness check sidecarVersion or sidecarSupervisorVersion fails;
• monitor-report emits state_root_sidecar_version_mismatch;
• reported engine/protocol/checkpoint format does not match the frozen contract.

Actions:

1. Deploy the sidecar binary built from the same NOOSChain release.
2. Confirm NOOS_STATE_ROOT_SIDECAR_PROTOCOL_VERSION=1.
3. Restart the supervisor/node.
4. Run state-root preflight.
5. Run production preflight.

Never allow a wrong-version sidecar to execute post-activation blocks.

Missing Or Corrupt Checkpoint

Symptoms:

• sidecar checkpoint load fails;
• checkpoint file hash does not match DB metadata;
• checkpoint root changes after load.

Actions:

1. Treat verified checkpoint mismatch as unsafe local cache.
2. Restore the checkpoint directory from a backup taken with the database, if available.
3. If no checkpoint backup is available, allow the node to rebuild the sidecar from materialized state_leaves only after confirming database integrity.
4. Run chain and replay verification.
5. Let the next successful Nervos block export a new verified checkpoint.

Checkpoint files are local operational cache. They are not a replacement for block roots, snapshot verification, or replay.

Snapshot Restore Path

After importing a snapshot:

1. Verify the snapshot archive.
2. Import into an empty database or approved restore flow.
3. Confirm the checkpoint records the snapshot engine metadata.
4. If the snapshot is post-activation, ensure the sidecar is ready before the first new block.
5. Run replay/chain verification.

Commands:

npm run noos -- snapshots verify --file snapshot.noosnap.tar.gz
npm run noos -- snapshots import --file snapshot.noosnap.tar.gz --mode empty_database_only --yes
npm run noos -- state-root preflight
npm run noos -- chain verify
npm run noos -- chain replay-verify

Incident Handling

If block execution fails after activation because the Nervos sidecar is missing, wrong-version, corrupt, or unresponsive:

1. stop validator traffic;
2. keep the incident active;
3. repair the sidecar/checkpoint problem;
4. replay or resync the node;
5. verify chain and replay;
6. only then mark the incident resolved.

Marking an incident resolved only updates incident status. It does not repair a sidecar, checkpoint, state root, or validator membership problem.