Incident Response

Incident response is the operator workflow for diagnosing, repairing, verifying, and closing an active NOOSChain incident.

The lifecycle is:

1. Detect the symptom.
2. Stabilize the node or environment.
3. Collect evidence.
4. Diagnose the cause.
5. Repair the cause.
6. Verify recovery.
7. Resolve the incident record.
8. Review and improve the runbook.

noos incidents resolve is bookkeeping. It marks the incident row resolved; it does not rewrite finalized history, repair peer state, restart sidecars, fetch missing payloads, change runtime membership, or restore a database.

Related Pages

Need	Page
What should page an operator	Monitoring And Alerting
Finalized conflict or divergence	Conflict Handling
Restore or node replacement	Backup, Restore, And Recovery
Rejoin gate after repair	Production Readiness
CLI command details	Operator CLI
Smart-contract runtime incidents	Smart Contract Runtime Operations

Severity

Use severity to decide how quickly to page and whether traffic should be held.

Severity	Examples	Default posture
Critical	Finalized conflict, unsafe validator state, state-root sidecar failure at/after activation, database loss, restore required, contract runtime unsafe for production calls.	Page immediately. Hold validator participation or production traffic until understood.
Warning	Sync/backfill issue, expected-but-missing payloads, degraded peer set, repeated contract failures, backup metadata missing.	Ticket or page if repeated, worsening, or near launch/activation.
Info	Long-running verification, maintenance state, expected observer catch-up.	Record for context. No page by default.

Local policy can be stricter, especially for validators.

Immediate Triage

Start with a broad, read-only picture:

npm run noos -- production monitor-report --json
npm run noos -- incidents active
npm run noos -- incidents doctor --json
npm run noos -- node health --json
npm run noos -- node sync --json
npm run noos -- support bundle --output noos-support-bundle.json

incidents doctor collects a focused diagnostic view from observability, incidents, conflicts, consensus, peers, and payloads. It returns checks and a fixPlan with recommended next commands.

Add targeted commands based on the symptom:

Symptom	Add commands
Finalized conflict or replay divergence	npm run noos -- chain verify, npm run noos -- chain replay-verify, npm run noos -- conflicts list
Raft or consensus issue	npm run noos -- raft status --json, npm run noos -- raft-go doctor --json, npm run noos -- governance operator-membership plan --json
State-root issue	npm run noos -- state-root status --json, npm run noos -- state-root preflight --json
Payload availability issue	npm run noos -- node payloads, npm run noos -- node reconcile-payload-availability --dry-run
Snapshot or restore issue	npm run noos -- snapshots list, npm run noos -- snapshots verify --file <snapshot>
Smart-contract runtime issue	npm run noos -- contracts alerts --json, npm run noos -- contracts runtime preflight --json

Keep outputs with timestamps. They become the incident record.

Stabilize First

Before repair, prevent the problem from getting worse.

• For validator incidents, hold or drain validator traffic when consensus, state-root, replay, or runtime safety is uncertain.
• For finalized conflicts, do not force sync until the authoritative history is known.
• For restore incidents, stop the node before replacing the database.
• For validator database changes, stop the HashiCorp Raft sidecar first.
• For contract runtime incidents, stop routing calls to unsafe contracts before changing runtime or package state.
• For missing payloads, distinguish expected policy behavior from true data loss before pruning or repairing local stores.

Diagnosis Decision Table

Symptom	First checks	Likely handoff
Finalized conflict, state-root mismatch, bad block hash, genesis mismatch	incidents doctor, conflicts list, chain verify, chain replay-verify	Conflict Handling
Node unhealthy or database down	node health, PostgreSQL/service logs, doctor --deep	Operator CLI
HashiCorp Raft sidecar down or no leader	raft-go doctor, raft status, raft leader	Raft Sidecar Operations
Governance/runtime membership mismatch	governance operator-membership plan --json	Raft Consensus Configuration
State-root readiness failed	state-root preflight, state-root status	State-Root Activation, Nervos Sidecar Recovery
Missing local payloads	node payloads, node reconcile-payload-availability --dry-run	Operator CLI, Data Buckets
Restore or data loss required	snapshots verify, production backup-drill, offline verification	Backup, Restore, And Recovery
Contract runtime failures	contracts alerts, contracts readiness --refresh, contracts runtime preflight	Smart Contract Runtime Operations
Security posture issue	production preflight, doctor --deep	Production Readiness, Admin GUI

Example: Active Incident With Conflict

Use this when a node records a finalized-history conflict or a peer is quarantined because it served conflicting finalized data.

1. Capture evidence before changing peer configuration:

   npm run noos -- production monitor-report --json
   npm run noos -- incidents doctor --json
   npm run noos -- conflicts list --json
   npm run noos -- chain head --json
   npm run noos -- node sync --json
   npm run noos -- peers scores --json
   npm run noos -- support bundle --output noos-support-bundle.json

2. Verify local deterministic state:

   npm run noos -- chain verify
   npm run noos -- chain replay-verify

3. If a peer is the suspected source, keep it quarantined while you compare histories:

   npm run noos -- peers quarantine <peerId> --reason "finalized conflict investigation"

4. Follow Conflict Handling to decide authoritative history and repair the bad peer or local node.

5. Resolve only after verification is green:

   npm run noos -- incidents resolve <incidentId> --reason "verified conflict recovery" --yes

Example: Raft Sidecar Down

Use this when a validator has no leader, consensus is degraded, or the HashiCorp sidecar is stopped/restarting.

1. Check current alerts and sidecar state:

   npm run noos -- production monitor-report --json
   npm run noos -- raft-go doctor --json
   npm run noos -- raft status --json
   npm run noos -- raft leader --json

2. Review the dry-run recovery plan:

   npm run noos -- raft-go restart --dry-run

3. Use the operating-system service manager or process supervisor if a restart is required. The current node API does not expose forced start/stop/restart.

4. Verify consensus and membership before admitting validator traffic:

   npm run noos -- governance operator-membership plan --json
   npm run noos -- production preflight --profile validator

5. If membership is mismatched, do not improvise. Use the approved membership workflow in Raft Consensus Configuration.

Example: Missing Payloads

Use this when observability reports missing local ciphertext.

1. Inspect payload state:

   npm run noos -- node payloads
   npm run noos -- node sync --json

2. Check whether missing payloads are expected. They can be normal for metadata-only buckets or nodes without the required permissions.

3. Run a reconciliation dry-run:

   npm run noos -- node reconcile-payload-availability --dry-run

4. If ciphertext is allowed but missing, run the node's backfill operator flow and inspect peer scoring. If ciphertext is present but no longer allowed, review before pruning:

   npm run noos -- node reconcile-payload-availability --prune-disallowed --yes

5. Do not mark the incident resolved until the node reports only expected missing payloads.

Example: State-Root Sidecar Failure

Use this when state-root readiness is red, the Nervos sidecar is missing or mismatched, or the chain is at/after a nervos-smt-v2 activation boundary.

1. Hold validator traffic until readiness is understood.

2. Check state-root status and preflight:

   npm run noos -- state-root status --json
   npm run noos -- state-root preflight --json
   npm run noos -- production monitor-report --json

3. If the sidecar is required now, there is no safe silent fallback to iden3-v1. Fix the sidecar or keep the node out of validator traffic.

4. Use Nervos Sidecar Recovery for repair and State-Root Activation for rollout context.

5. Verify before rejoin:

   npm run noos -- state-root preflight --json
   npm run noos -- production preflight --profile validator

Example: Restore After Database Loss

Use this when the database is corrupt, unavailable, or must be replaced.

1. Capture what you can before replacement:

   npm run noos -- incidents doctor --json
   npm run noos -- support bundle --output noos-support-bundle.json

2. Stop the node. If this is a validator, stop the HashiCorp Raft sidecar before changing database state.

3. Verify the backup artifact:

   npm run noos -- snapshots verify --file .\node-backup.noosnap.tar.gz

4. Follow the role-specific restore path in Backup, Restore, And Recovery.

5. Before rejoin:

   npm run verify:chain
   npm run verify:replay
   npm run noos -- production preflight --profile <observer|validator|single-node>
   npm run noos -- production monitor-report --json

6. Resolve only after the restored node is verified and no critical alert remains.

Example: Smart-Contract Runtime Failures

Use this when contract calls are failing, readiness is stuck, or the runtime sidecar is unavailable.

1. Inspect alerts and runtime preflight:

   npm run noos -- contracts alerts --json
   npm run noos -- contracts alerts <CONTRACT_ID> --json
   npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json
   npm run noos -- contracts runtime preflight --json

2. If failure rate is critical, stop routing traffic to the affected contract until the cause is understood.

3. Check whether the issue is runtime-sidecar health, bucket readiness, payload availability, permission prerequisites, fuel exhaustion, or contract logic.

4. Use Smart Contract Runtime Operations and Smart Contract Operations Runbooks.

5. After repair, confirm alerts are clear:

   npm run noos -- contracts alerts <CONTRACT_ID> --json
   npm run noos -- production monitor-report --json

Example: Resolve Correctly

Resolve only after the repair has been verified.

1. Confirm the active incident:

   npm run noos -- incidents active --json

2. Run the verification that matches the incident:

   npm run noos -- production monitor-report --json
   npm run noos -- production preflight --profile validator
   npm run noos -- chain verify
   npm run noos -- chain replay-verify

3. Resolve with a specific reason:

   npm run noos -- incidents resolve <incidentId> --reason "state-root sidecar upgraded and preflight verified" --yes

Use a reason that future operators can understand. Avoid vague reasons such as fixed or ok.

Evidence Handling

Keep evidence in the incident record:

• production monitor-report --json;
• incidents doctor --json;
• support bundle;
• relevant chain, raft, state-root, payload, snapshot, or contract command output;
• timestamps and node ids;
• repair actions taken;
• verification commands and results;
• final resolve reason.

Do not paste secrets into tickets or chat:

• operator tokens;
• private keys;
• raw DEKs;
• plaintext payloads;
• encrypted payload bytes;
• auth challenges;
• full unredacted .env files.

Before destructive restore or database replacement, preserve the current database and sidecar data directory when feasible.

Post-Incident Review

After the incident is closed, record:

• what happened;
• first alert time;
• affected nodes and roles;
• customer or application impact, if any;
• root cause;
• why existing checks did or did not catch it earlier;
• evidence collected;
• repair actions;
• verification commands;
• follow-up tasks;
• docs or runbook changes needed.

Good incident notes make the next incident shorter.

Not Covered

This page does not duplicate:

• finalized-history conflict adjudication: use Conflict Handling;
• restore procedures: use Backup, Restore, And Recovery;
• alert design: use Monitoring And Alerting;
• raw observability route definitions: use Observability API;
• smart-contract runtime repair details: use Smart Contract Runtime Operations.