Backup, Restore, And Recovery

This runbook helps an operator choose the right recovery path, prove a backup, restore a node, and rejoin production without introducing stale or unsafe state.

It is operational guidance. It does not change consensus rules, replace deterministic chain/replay verification, explain snapshot file internals, or replace incident response.

Use the focused pages for details this runbook intentionally does not repeat:

Need	Page
Snapshot formats, export/import behavior, and tuning	Snapshots
General snapshot concept and recipes	General Snapshots
Production admission criteria	Production Readiness
Alerting during and after recovery	Monitoring And Alerting
Incident workflow	Incident Response
Fork or conflict recovery	Conflict Handling

Safety Model

NOOSChain recovery has two different state classes:

State class	Examples	Recovery concern
Consensus database state	Blocks, transactions, domain rows, state-root leaves, checkpoints, governance state, snapshot metadata, contract state.	Must verify deterministically before serving traffic.
Local/runtime operational state	Trusted peers, peer scores, local payload availability, node process state, sidecar process state, HashiCorp Raft data directory.	Must match the node role before rejoin.

Hard rules:

• Stop the node before replacing its database.
• Stop the HashiCorp Raft sidecar before changing validator database state.
• Restore into an empty database/schema unless you are following the explicit operator-restore procedure.
• Verify the restored chain before letting the node serve production traffic.
• Do not mark an incident resolved until the cause is fixed and verification is green.
• Do not rejoin a validator as a voter with mismatched, empty, or unknown Raft sidecar data.

Choose A Recovery Path

Situation	Use this path
Prove backups on a healthy node	Routine Backup Proof
Replace or rebuild an observer	Observer Restore
Restore a standalone deployment	Single-Node Restore
Validator host is recoverable and Raft data is intact	Validator Restore With Raft Data Intact
Validator database and Raft data are lost or untrusted	Validator Total Data Loss
Emergency overwrite of a non-empty local database	Operator Restore Into A Non-Empty Database
Smart-contract state was present	Smart-Contract Post-Restore Checks

When in doubt, restore as an observer first. A restored observer can sync, verify, and backfill without risking validator quorum or finalized history.

Routine Backup Proof

Run this on each production node on the same cadence as your backup policy:

npm run noos -- production backup-drill --output .\backup-drill

The drill exports a consensus-only archive, verifies it, imports it into isolated schemas, runs chain verification, proves checkpoint bootstrap, writes backup-drill-report.json, and drops the generated schemas. It does not replace the running node database.

Treat a failed backup drill as a production blocker. A backup artifact is not credible until it has been imported and verified.

For snapshot commands and format details, see Snapshots. For the production gate that consumes this proof, see Production Readiness.

Backup Artifact Checklist

Keep these artifacts together for each backup point:

• packaged consensus snapshot archive, usually *.noosnap.tar.gz;
• backup-drill-report.json from a successful drill;
• release bundle version: TypeScript node, Admin GUI, HashiCorp Raft sidecar, state-root sidecar, migrations, and docs/runbook revision;
• node .env inventory, with secrets stored in the secret manager rather than inside the backup archive;
• TLS/mTLS certificate inventory and CA chain;
• for validators, HashiCorp Raft sidecar data-dir backup when your recovery model restores the same validator identity in place.

Snapshot archives must not contain private keys, operator tokens, DEKs, or plaintext payloads. consensus_state_only archives also omit local ciphertext bytes.

Observer Restore

Use this when replacing a sync-only observer or bringing up a new read/sync node from backup.

1. Stop the observer process.

2. Create an empty database/schema for the restored observer.

3. Apply migrations from the same release:

   npm run migrate

4. Import the verified snapshot:

   npm run noos -- snapshots import --file .\node-backup.noosnap.tar.gz --mode empty_database_only --yes

5. Verify the restored database while the node is still offline:

   npm run verify:chain
   npm run verify:replay

6. Start the node as an observer, not as a validator.

7. Check sync and payload state:

   npm run noos -- node sync
   npm run noos -- node payloads

8. Run observer admission checks:

   npm run noos -- production preflight --profile observer

If expected payloads are missing, run backfill and inspect peer scoring before marking the restore complete. Missing payloads can be expected for buckets this node is not allowed or configured to hold.

Validator Restore With Raft Data Intact

Use this only when the validator host is recoverable and its HashiCorp Raft data directory is intact, but the TypeScript database needs restore or verification.

1. Remove traffic from the validator at the load balancer or process supervisor.

2. Stop the TypeScript node.

3. Stop the HashiCorp Raft sidecar.

4. Back up the current database and Raft data directory before changing anything.

5. Restore the database into an empty replacement schema or database.

6. Import the verified snapshot:

   npm run noos -- snapshots import --file .\validator-backup.noosnap.tar.gz --mode empty_database_only --yes

7. Verify the restored database offline:

   npm run verify:chain
   npm run verify:replay

8. Confirm the restored chain head is not ahead of the rest of the cluster.

9. Start the HashiCorp Raft sidecar with the same data directory.

10. Start the TypeScript node.

11. Wait for catch-up and sidecar health:

    npm run noos -- raft-go doctor --json
    npm run noos -- raft status --json
    npm run noos -- node sync

12. Run validator admission checks:

    npm run noos -- production preflight --profile validator

Do not force the node back into service if sidecar status, membership, chain verification, replay verification, or state-root readiness is red.

Validator Total Data Loss

Use this when both the TypeScript database and the HashiCorp Raft data directory are lost or untrusted.

The conservative path is replacement, not blind in-place resurrection:

1. Keep the failed validator offline.

2. Use a healthy validator to inspect membership:

   npm run noos -- governance operator-membership plan --json
   npm run noos -- raft membership --json

3. If quorum is healthy without the failed node, prepare an operator-approved membership plan to remove the failed voter or add a replacement before removal, depending on the target validator set.

4. Execute membership changes only after operator approval:

   npm run noos -- governance operator-membership execute --plan-file approved-plan.json --yes

5. Build the replacement node from a verified snapshot as an observer first.

6. Let it sync and backfill until chain head and state root match healthy validators.

7. Run chain and replay verification on the replacement.

8. Add it to runtime Raft membership only through the approved membership workflow.

9. Run production preflight before admitting validator traffic.

Do not reuse a lost validator identity with an empty Raft data directory unless the cluster membership workflow and operator runbook explicitly account for that replacement. HashiCorp Raft membership is runtime state; validator governance state alone does not safely reconstruct Raft log/stable state.

Single-Node Restore

Use this for standalone deployments where NOOS_CONSENSUS_MODE=single_node.

1. Stop the node.

2. Restore into an empty database/schema.

3. Import the verified snapshot:

   npm run noos -- snapshots import --file .\single-node-backup.noosnap.tar.gz --mode empty_database_only --yes

4. Verify before restart:

   npm run verify:chain
   npm run verify:replay
   npm run noos -- production preflight --profile single-node

5. Start the node only after verification passes.

Single-node restores have no quorum to protect against stale local history. Treat the verified backup artifact as the authority for that deployment.

Operator Restore Into A Non-Empty Database

The normal restore path is empty_database_only. Use operator restore only for controlled disaster recovery where you intentionally replace existing local database contents:

$env:NOOSCHAIN_ALLOW_OPERATOR_RESTORE="true"
npm run noos -- snapshots import --file .\backup.noosnap.tar.gz --operator-restore --yes

Operator restore is dangerous. It can overwrite local chain tables. Before using it:

• stop the node and sidecars;
• take a filesystem/database backup of the current state;
• confirm the snapshot archive verifies;
• confirm the target role and recovery scenario;
• ensure no process is writing to the database;
• run chain and replay verification before restart.

Smart-Contract Post-Restore Checks

Smart-contract consensus state is restored through the normal snapshot path. Do not copy contract tables by hand.

After a restore, verify local runtime readiness before routing contract calls:

npm run noos -- contracts runtime preflight --json
npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json
npm run noos -- contracts alerts <CONTRACT_ID> --json
npm run noos -- production monitor-report --json

If readiness is red after restore, treat it as a local runtime, bucket, permission, or payload-availability problem first. The restored consensus rows may be valid while this node is still not ready to execute a specific contract.

Use Smart Contract Runtime Operations for runtime recovery. Use Packages And Provenance only when post-incident audit needs the original package evidence.

Rejoin Checklist

Before a restored node rejoins production traffic, confirm:

• npm run verify:chain passes against the restored database;
• npm run verify:replay passes or the approved checkpoint bootstrap proof is green;
• npm run noos -- production preflight --profile <role> passes;
• npm run noos -- production monitor-report --json has no critical alert for this role;
• no active incident remains for the restored node;
• state-root readiness is green;
• in Raft mode, leader, quorum, membership, and sidecar health are green;
• payload availability is complete or missing only payloads that policy says this node should not have;
• trusted peers and peer scoring do not block every healthy sync/backfill peer.

Incident Recovery

If restore is triggered by an incident, collect diagnostics first:

npm run noos -- incidents doctor --json
npm run noos -- support bundle --output noos-support-bundle.json

Then:

1. Fix the cause: database loss, sidecar data-dir loss, payload availability, peer scoring, membership mismatch, TLS/mTLS failure, or state-root readiness.

2. Restore and verify using the role-specific procedure above.

3. Mark the incident resolved only after verification is green:

   npm run noos -- incidents resolve <incidentId> --reason "restored and verified" --yes

Marking an incident as resolved is bookkeeping. It does not repair state, rewrite finalized history, unban peers, restart sidecars, or change runtime membership. Use Incident Response and Conflict Handling for the full incident runbooks.

Not Covered

This runbook does not cover:

• snapshot archive internals or performance tuning;
• automatic Raft voter replacement;
• external secret-manager backups;
• host volume snapshots;
• cloud-specific database restore tooling;
• smart-contract package/provenance review;
• soak, chaos, or E2E recovery tests.