Conflict Handling

This page is the specialized operator runbook for finalized-history conflicts. Use it when a node records divergence in finalized block history, state root, Merkle root, protocol version, genesis, or committed Raft execution.

For the conceptual model, see Fork Conflict Handling. For the broader incident lifecycle, see Incident Response.

Core Rule

NOOSChain assumes deterministic finality. Once a block is committed, local finalized history is immutable.

A peer that reports a different finalized block, state root, Merkle root, protocol version, or genesis is not a routine fork-choice event. It is evidence that one node, configuration, release, sidecar, snapshot, or peer set is unsafe.

Do not pick the longest chain. Do not force sync just to clear the alert.

The operator response is:

1. freeze unsafe writes or validator traffic;
2. preserve evidence;
3. identify the authoritative history;
4. repair the unsafe node or peer;
5. verify deterministic state;
6. resolve the incident only after repair.

Scope

Use this runbook for:

• same_height_different_hash;
• genesis_mismatch;
• previous_hash_mismatch;
• state_root_mismatch;
• block_hash_mismatch;
• merkle_root_mismatch;
• protocol_version_mismatch;
• Raft post-commit execution divergence.

Do not use this runbook for:

• normal transaction rejection;
• contract business-rule failure;
• missing local encrypted payload bytes without finalized divergence;
• temporary peer downtime;
• operator token or auth errors;
• routine smart-contract runtime errors.

Those may still page an operator, but they are not finalized-history conflicts unless the node records a conflict or incident.

Safety Rules

• Do not mark an incident resolved until the underlying cause is fixed.
• Do not submit validator traffic while incident mode is active.
• Do not unquarantine a conflicting peer until its chain, protocol version, state-root engine, release, and snapshot source are verified.
• Do not force sync until the authoritative peer set is known.
• Do not restore from a snapshot unless it is verified and belongs to the authoritative history.
• Do not treat incidents resolve as repair. It only closes the incident row.

Incident Mode

When a finalized conflict enters incident mode:

• POST /sync/run returns 409 CHAIN_INCIDENT_ACTIVE unless force=true is supplied;
• automatic sync manager passes stop;
• /consensus/propose-block refuses new proposals;
• local read APIs remain available;
• outbound transaction propagation is suppressed for locally submitted transactions;
• the conflicting peer is quarantined in trusted_peers;
• the peer score records a finalized_conflict event.

This is deliberate. Incident mode preserves evidence and prevents the node from silently extending or importing history before an operator decides which side is valid.

Conflict Types

Type	Meaning	Typical causes	Immediate action	Deeper checks
same_height_different_hash	Local and remote finalized heads share a height but not a block hash.	Split cluster, wrong peer set, restored node from another chain, bad manual sync source.	Quarantine peer, enter incident mode.	Compare validator heads, Raft quorum, genesis, release, snapshot source.
genesis_mismatch	Peer was initialized from a different genesis.	Wrong environment, wrong database, wrong trusted peer.	Quarantine peer.	Check chain info, trusted peer config, deployment environment.
previous_hash_mismatch	Fetched block does not extend expected parent.	Peer served different finalized branch or inconsistent range.	Quarantine peer.	Compare last common height and first divergent block.
state_root_mismatch	Block envelope or replayed deterministic state root differs.	State-root engine mismatch, sidecar version mismatch, non-deterministic adapter bug, wrong protocol activation.	Quarantine peer, enter incident mode.	Run replay, state-root preflight, engine/version comparisons.
block_hash_mismatch	Fetched block does not hash to advertised value.	Corrupt block, incompatible hashing, serialization bug.	Quarantine peer.	Compare block bytes, release version, protocol version.
merkle_root_mismatch	Transaction Merkle root does not match block metadata.	Corrupt transaction list, peer bug, serialization mismatch.	Quarantine peer.	Compare transaction list and block metadata.
protocol_version_mismatch	Remote block uses incompatible protocol version.	Mixed release, stale node, wrong protocol activation plan.	Quarantine peer.	Compare release and protocol schedule.

Immediate Triage

Capture evidence before changing peer configuration:

npm run noos -- production monitor-report --json
npm run noos -- incidents doctor --json
npm run noos -- conflicts list --json
npm run noos -- chain head --json
npm run noos -- chain info --json
npm run noos -- node sync --json
npm run noos -- peers scores --json
npm run noos -- support bundle --output noos-support-bundle.json

Then verify local deterministic state:

npm run noos -- chain verify
npm run noos -- chain replay-verify
npm run noos -- chain verification-run
npm run noos -- chain verification-status --json

If the conflict is state-root related:

npm run noos -- state-root status --json
npm run noos -- state-root preflight --json

If the node is a HashiCorp Raft voter:

npm run noos -- raft-go doctor --json
npm run noos -- raft status --json
npm run noos -- raft leader --json

Keep command outputs with timestamps.

Evidence Checklist

Record:

• local node id and role;
• conflicting peer id and base URL;
• conflict type;
• local height, block hash, state root, Merkle root, protocol version, and genesis hash;
• remote height, block hash, state root, Merkle root, protocol version, and genesis hash when available;
• state-root engine and engine version;
• state-root activation height;
• sidecar protocol/version/checkpoint metadata;
• Raft leader, term, quorum, voters, and membership state;
• NOOSChain release version on each participant;
• trusted peer configuration;
• snapshot source if any node was recently restored;
• support bundle path;
• operator decision about authoritative history.

Decide The Authoritative History

Use independent evidence. A single healthy-looking peer is not enough if the cluster may have split.

Check:

• which validators committed the height;
• whether a Raft leader and quorum were present;
• whether other validators report the same head hash and state root;
• whether observer nodes agree with the validator majority;
• whether the state-root engine activation schedule matches on all nodes;
• whether all nodes run the same release and protocol-version matrix;
• whether snapshots used during restore came from the expected environment;
• whether governance or disaster-recovery policy defines a tie-breaker.

For deterministic-finality deployments, the authoritative source is the operator-approved validator set or a verified snapshot from that set. It is not automatically the longest remote chain.

Example: Bad Peer, Local Node Healthy

Use this when local verification is green and the conflicting peer is the only outlier.

1. Keep the peer quarantined:

   npm run noos -- peers quarantine <peerId> --reason "finalized conflict investigation"

2. Verify local chain and replay:

   npm run noos -- chain verify
   npm run noos -- chain replay-verify

3. Confirm at least one independent validator or observer agrees with local head hash and state root:

   npm run noos -- chain head --json
   npm run noos -- raft status --json

4. Fix the bad peer out of band. Common fixes are replacing the trusted-peer entry, restoring the peer from an authoritative snapshot, correcting environment variables, or deploying the matching release and sidecars.

5. Recheck peer score and trusted-peer state:

   npm run noos -- peers scores --json
   npm run noos -- peers score <peerId> --json

6. Resolve only after local verification remains green and the bad peer cannot serve conflicting history.

Example: Local Node Restored From Wrong Snapshot

Use this when genesis, block hash, or protocol history suggests the local node was restored from the wrong environment or a stale snapshot.

1. Stop this node from accepting production or validator traffic.

2. Preserve evidence:

   npm run noos -- incidents doctor --json
   npm run noos -- support bundle --output noos-support-bundle.json

3. Compare local chain identity with the authoritative set:

   npm run noos -- chain info --json
   npm run noos -- chain head --json
   npm run noos -- chain verify

4. Identify an authoritative snapshot from the approved validator set.

5. Restore using Backup, Restore, And Recovery.

6. Verify before rejoin:

   npm run noos -- chain verify
   npm run noos -- chain replay-verify
   npm run noos -- production preflight --profile <observer|validator|single-node>

7. Re-enable traffic only after local head hash and state root match the authoritative set.

Example: State-Root Mismatch After Nervos Activation

Use this when the conflict type is state_root_mismatch or replay diverges around a state-root engine activation.

1. Hold validator traffic. Do not force sync.

2. Check local state-root status:

   npm run noos -- state-root status --json
   npm run noos -- state-root preflight --json
   npm run noos -- chain replay-verify

3. Compare across validators:

   • configured state-root engine;
   • governed activation height;
   • engine version in checkpoints;
   • Nervos SMT sidecar binary version;
   • sidecar protocol version;
   • checkpoint format version;
   • latest verified checkpoint metadata;
   • NOOSChain release version.

4. If preflight fails, fix the engine or sidecar before any sync. At or after a governed nervos-smt-v2 activation, there is no safe silent fallback to iden3-v1.

5. Use State-Root Activation for rollout context and Nervos Sidecar Recovery for repair.

6. Verify before resolve:

   npm run noos -- state-root preflight --json
   npm run noos -- chain replay-verify
   npm run noos -- production preflight --profile validator

Example: Cluster Split Or No Clear Majority

Use this when multiple validators report different finalized histories and there is no obvious bad peer.

1. Freeze validator writes across the deployment.
2. Collect incidents doctor, conflicts list, chain head, raft-go doctor, and support bundles from every validator.
3. Compare:
   • last common height;
   • first divergent height;
   • block hash;
   • state root;
   • proposer;
   • protocol version;
   • Raft term and leader;
   • runtime membership at the divergent height.
4. Decide authoritative history by governance or disaster-recovery policy.
5. Restore losing-side nodes from an authoritative snapshot or rebuild them from authoritative peers.
6. Keep all losing-side evidence until incident review is complete.

NOOSChain does not currently provide automatic reorg, slashing, Byzantine evidence adjudication, or committed-log reconciliation. The operator must choose and enforce the authoritative history.

Example: Protocol Version Mismatch After Mixed Release

Use this when peers disagree about protocol version or a block uses an unexpected protocol version.

1. Hold traffic on the node reporting the mismatch.

2. Capture local protocol and release state:

   npm run noos -- chain info --json
   npm run noos -- chain head --json
   npm run noos -- production report --json

3. Compare the release and protocol activation schedule across validators.

4. Repair by deploying the approved release or rolling back according to the release plan. Do not manually edit protocol rows.

5. Verify:

   npm run noos -- chain verify
   npm run noos -- chain replay-verify
   npm run noos -- production preflight --profile validator

Use Production Readiness for admission checks and Protocol Versioning for internals.

Forced Sync

Manual sync is intentionally blocked while incident mode is active. Forced sync is a recovery action after the authoritative peer set is known.

Use forced sync only when:

• authoritative history has been identified;
• the peer being synced from is part of that authoritative set;
• local configuration, release, protocol version, and state-root engine match;
• a support bundle and current evidence have already been preserved;
• an operator has approved the action.

Current explicit API call:

$env:NOOS_OPERATOR_TOKEN="<operator-token>"
curl.exe -X POST `
  -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" `
  -H "Content-Type: application/json" `
  -d "{\"force\":true}" `
  "http://localhost:3000/sync/run?force=true"

The response is an array of per-peer sync results. Treat failed peer results as continuing evidence. If forced sync reports a new conflict, stop and return to triage.

The CLI exposes npm run noos -- node sync --json for sync status, but not a mutating sync run command in this milestone.

Resolve The Incident

Resolve only when all of these are true:

• authoritative history is identified;
• bad peer, bad local configuration, stale snapshot, wrong release, or bad sidecar has been repaired;
• quarantined peers cannot serve conflicting history;
• chain verify passes;
• chain replay-verify passes;
• state-root preflight passes when an external engine is active or scheduled;
• Raft diagnostics are healthy for validator nodes;
• production preflight is green for the role being returned to service;
• evidence and operator decision are recorded.

Then close the incident:

npm run noos -- incidents active --json
npm run noos -- incidents resolve <incidentId> --reason "verified conflict recovery" --yes

Resolution does not delete conflict evidence and does not rewrite history. It marks the incident as handled so normal sync and proposal paths can resume according to node controls.

Related Pages

• Incident Response
• Monitoring And Alerting
• Backup, Restore, And Recovery
• Snapshots
• HashiCorp Raft Deployment
• Raft Sidecar Operations
• State-Root Activation
• Nervos Sidecar Recovery
• General: Fork Conflict Handling