State-Root Engine Migration Internals

This page is for developers changing NOOSChain state-root engines, engine selection, activation, replay, sync, snapshots, checkpoints, or proof support.

For the conceptual model, see State Root. For operator activation procedures, see State-Root Activation.

The core invariant is:

The block executor emits deterministic state leaves and mutations. The selected state-root engine folds them into state_root. Historical replay must use the state_root_engine_version recorded for each block.

Unknown engines fail closed. A node that does not know or cannot execute the engine recorded on a block, snapshot, or checkpoint must reject the work instead of guessing or falling back to a local default.

Current Engines

Engine	Status	Runtime	Root scheme	Sidecar	Proof support
iden3-v1	Legacy supported	TypeScript/Node.js with @iden3/js-merkletree	iden3 field-element SMT with NOOSChain hash-to-field adapter	No	iden3 proof semantics for iden3 roots
nervos-smt-v2	Default for new chains	Rust sparse-merkle-tree sidecar	Nervos SMT with Blake2b	Yes	nervos-smt-v2-proof-v1 through the sidecar

The frozen engine contracts are documented in:

• iden3 State-Root Contract
• Nervos State-Root Contract

Those pages are the source of truth for engine identifiers, key/value encoding, mutation ordering, root schemes, checkpoint behavior, proof formats, and stable error behavior.

Implementation Map

Engine identifiers and migration models:

• src/chain/state-root-engine-version.ts: known engine ids, CURRENT_STATE_ROOT_ENGINE_VERSION, migration models, known-engine validation, and root-scheme compatibility checks.
• src/chain/state-root-engine-contract.ts: frozen Nervos sidecar contract constants such as sidecar protocol, checkpoint format, empty root, proof format, hasher, and mutation ordering.

Selection and activation:

• src/chain/state-root-engine-config.ts: env fallback, governed activation lookup, next activation lookup, and block-height engine selection.
• src/chain/domain-handlers.ts: dispatch for transaction SCHEDULE_STATE_ROOT_ENGINE_ACTIVATION.
• src/chain/state-root-activation-readiness.ts: operator-facing readiness checks for scheduled and active Nervos activation.

Execution:

• src/chain/block-builder.ts: selects engine for candidate blocks.
• src/consensus/adapters/raft/raft-log-entry.ts: selects and persists engine metadata for Raft block entries.
• src/chain/smt-state-store.ts: engine adapter for root calculation, state leaves, checkpoints, and proof delegation.

Replay, sync, and snapshots:

• src/replay/replay-engine.ts: uses stored block engine metadata and reports state_root_engine_version_mismatch.
• src/node-sync/remote-block-validation.ts: rejects unknown or mismatched remote block engine versions.
• src/node-sync/remote-block-ingestion.ts: persists known engine metadata during sync ingestion.
• src/snapshots/*: exports, imports, streams, and verifies engine metadata.

Proofs:

• src/chain/state-proof-service.ts: resolves public namespace/entity ids to state keys and delegates proof generation/verification to the active engine.

Relevant migrations:

• src/db/migrations/033_state_root_engine_version.sql
• src/db/migrations/034_state_root_engine_activation_governance.sql
• src/db/migrations/036_state_root_sidecar_checkpoints.sql
• src/db/migrations/037_state_root_sidecar_checkpoint_versions.sql

Data Model

Data	Consensus state?	Purpose
blocks.state_root_engine_version	Yes	Canonical engine used for that block's root.
snapshots.state_root_engine_version	Yes for snapshot verification/import	Engine metadata for exported/restored state.
chain_checkpoints.state_root_engine_version	Yes for checkpoint verification	Engine metadata for checkpoint continuation.
state_root_engine_activations	Yes	Governed activation schedule and history.
state_leaves	Yes	Materialized latest consensus leaf set used for root rebuilds and snapshot hydration.
state_root_sidecar_checkpoints	Metadata is local operational evidence	Verified local Nervos checkpoint metadata.
Checkpoint files under NOOS_STATE_ROOT_CHECKPOINT_DIR	No	Local cache for faster Nervos execution/restart.

Checkpoint files are never consensus state. They are trusted only when the recorded engine id, sidecar protocol, checkpoint format, file hash, and loaded root match verified metadata. Losing checkpoint files is recoverable from state_leaves or a verified snapshot, but slower.

Engine Selection

Block execution selects an engine by height.

selectStateRootEngineForBlockHeight(database, height) resolves in this order:

1. If a committed activation row applies at height, use the activation's engine and report source governance.
2. Otherwise, use the env/test fallback from stateRootEngineVersionForBlockHeight and report source env.

The env fallback exists for genesis, single-engine deployments, and isolated tests. It is not a safe production migration path for an existing iden3 chain.

For replay, do not re-run current environment selection and reinterpret old blocks. Replay must use the engine metadata stored on each canonical block.

Migration Models

NOOSChain distinguishes two migration models:

Model	Meaning	Developer consequence
same_root_scheme	A replacement implementation produces byte-identical roots for every consensus namespace.	Prove equivalence across all state leaves before treating it as a drop-in replacement.
new_root_scheme	Roots intentionally differ for the same logical state.	Existing chains need an explicit activation height and persisted engine metadata.

iden3-v1 is marked same_root_scheme because future iden3-compatible implementations would have to produce the same roots.

nervos-smt-v2 is marked new_root_scheme. It is not iden3-compatible. It must not be used to continue an existing iden3 chain without an activation boundary.

Use a new engine id when changing any consensus-critical root behavior:

• key encoding;
• value encoding;
• mutation ordering;
• delete semantics;
• empty root;
• hasher or tree implementation;
• sidecar protocol version;
• checkpoint format;
• checkpoint rebuild semantics;
• proof format or proof encoding;
• deterministic root vectors.

Do not silently change the meaning of iden3-v1 or nervos-smt-v2.

Activation Boundary Semantics

An explicit engine migration has a height boundary:

1. Blocks below activation height keep the previous engine, such as iden3-v1.
2. The activation block and later blocks use the target engine.
3. Historical roots are not rewritten.
4. The selected engine is persisted on each block.
5. Snapshots and checkpoints preserve engine metadata.
6. Replay, sync, and import reject unknown engine versions.

This makes a root-scheme migration auditable. A chain can show exactly where the root scheme changed, and every node can replay the same boundary.

Local Activation Tests

Local tests can model a boundary with environment variables:

$env:NOOS_STATE_ROOT_ENGINE="nervos-smt-v2"
$env:NOOS_STATE_ROOT_ENGINE_ACTIVATION_HEIGHT="2"

With this configuration, blocks below height 2 execute with iden3-v1. Blocks at height 2 and above execute with nervos-smt-v2.

This is a test/development model. It is useful for unit, replay, and activation boundary tests, but it is not the production migration path for an existing chain.

Governed Activation Internals

Production-like migrations use transaction SCHEDULE_STATE_ROOT_ENGINE_ACTIVATION.

Example payload:

{
  "engineVersion": "nervos-smt-v2",
  "activationHeight": "12345",
  "reason": "planned state-root migration",
  "metadata": {
    "changeControlId": "CHG-1234"
  }
}

The transaction:

• requires chain:admin;
• validates that engineVersion is known by the binary;
• requires activation height to be in the future;
• writes state_root_engine_activations;
• includes the activation row in the state root under the state_root_engine_activation:<id> namespace;
• does not deploy sidecar binaries, configure checkpoint directories, mutate runtime Raft membership, or change trusted peers.

Operators still have to deploy and preflight the required sidecar on every validator before the activation height. The developer contract is that block production and execution fail closed if the scheduled engine cannot be executed.

For rollout procedures, see State-Root Activation and Nervos Activation Runbook.

Sidecar And Checkpoint Boundary

TypeScript owns deterministic transaction execution and mutation ordering. The selected state-root engine owns root calculation.

For nervos-smt-v2, the Rust sidecar receives state-root protocol messages:

• version and health checks;
• initialization and checkpoint load;
• ordered mutation batches;
• root reads;
• checkpoint export;
• proof generation and verification;
• shutdown.

The sidecar must not receive plaintext payloads, private keys, DEKs, bearer tokens, Raft control data, or raw SQL/database mutations.

For each Nervos block, execution should:

1. load the latest verified local checkpoint when available, or hydrate from materialized state_leaves;
2. apply the ordered mutation stream;
3. read the resulting root;
4. export a checkpoint;
5. hash the checkpoint file;
6. record verified checkpoint metadata.

The continuous sidecar supervisor and checkpoint files are local operational surfaces. They must never change historical replay semantics.

Proof Boundary

State proof APIs resolve a public namespace/entity id to a canonical state key, then delegate proof work to the selected engine.

Current behavior:

• iden3-v1 proofs use the legacy iden3 proof semantics for iden3 roots.
• nervos-smt-v2 proofs use the frozen nervos-smt-v2-proof-v1 document returned and verified by the sidecar.

Block-root correctness and public proof workflows are related but separate. A state-root engine can be valid for block roots only if its root vectors and replay behavior are deterministic. Proof APIs additionally require a frozen proof document format, verification behavior, and negative-proof semantics.

If a future engine changes proof format, use a new proof format and consider whether the engine id also needs to change.

Replay, Sync, And Snapshot Behavior

Replay:

• uses each block's stored state_root_engine_version;
• fails on unknown engines;
• fails if the required sidecar/engine cannot execute;
• reports state_root_engine_version_mismatch when canonical metadata diverges.

Sync:

• rejects remote blocks with unknown engine versions;
• compares fetched block metadata with local execution results;
• never trusts peer materialized state leaves directly.

Snapshots:

• preserve state-root engine metadata;
• verify the exported root before import;
• import state_root_engine_activations rows;
• preserve state_leaves so a Nervos sidecar can hydrate and export a fresh local checkpoint after restore.

A post-activation snapshot must continue with nervos-smt-v2. Local env defaults must not reinterpret it as iden3-v1.

Testing Matrix

Risk	Test evidence
Engine identifiers or migration models change	test:state-root-engine-version, test:state-root-engine-safety
Nervos root contract changes	test:state-root-nervos-vectors
Persistent sidecar/checkpoint behavior changes	test:state-root-persistent-sidecar, test:state-root-nervos
Activation boundary behavior changes	test:state-root-engine-activation
Readiness/preflight changes	test:state-root-activation-readiness
Governed activation transaction changes	test:state-root-engine-governance
Replay/sync/snapshot behavior changes	verify:replay, verify:chain, relevant snapshot tests
Multi-validator activation safety changes	hardness E2E for Noosraft and HashiCorp Go

Core commands:

npm run test:state-root-engine-version
npm run test:state-root-engine-safety
npm run test:state-root-nervos-vectors
npm run test:state-root-persistent-sidecar
npm run test:state-root-nervos
npm run test:state-root-engine-activation
npm run test:state-root-activation-readiness
npm run test:state-root-engine-governance
npm run verify:replay
npm run verify:chain
npm run test-e2e:hardness-multiple-producers-noosraft
npm run test-e2e:hardness-multiple-producers-hashicorp-go

Soak fast-smoke commands when the Nervos sidecar binary is built:

$env:NOOS_MP_NOOSRAFT_SOAK_FAST_SMOKE="true"
npm run soak:multiple-producers-noosraft
$env:NOOS_MP_HASHICORP_GO_SOAK_FAST_SMOKE="true"
npm run soak:multiple-producers-hashicorp-go

Custom soak runs can enable the activation segment with NOOS_MP_NOOSRAFT_SOAK_STATE_ROOT_ACTIVATION_ENABLED=true or NOOS_MP_HASHICORP_GO_SOAK_STATE_ROOT_ACTIVATION_ENABLED=true.

Developer Checklist

Before adding or changing a state-root engine:

• choose same_root_scheme or new_root_scheme;
• add or update the engine id in state-root-engine-version.ts;
• freeze the engine contract in docs and constants;
• add deterministic empty-root and mutation-batch vectors;
• update block, replay, sync, snapshot, checkpoint, and proof validation;
• update readiness/preflight if the engine needs an external sidecar;
• make unknown or unavailable engines fail closed;
• prove old blocks still replay with their recorded engine;
• update operator docs if deployment or activation behavior changes;
• regenerate reference docs if exported APIs changed.

Anti-Patterns

Avoid these:

• silently changing iden3-v1 or nervos-smt-v2 root semantics;
• continuing an existing iden3 chain on Nervos by env var only;
• falling back to iden3-v1 after Nervos blocks exist;
• using local env defaults to replay historical blocks;
• trusting sidecar checkpoints without verifying root and file metadata;
• treating checkpoint files as consensus state;
• conflating protocol_version with state_root_engine_version;
• conflating state-root sidecar protocol with chain protocol version;
• letting local sidecar health alter historical replay semantics;
• accepting unknown engine metadata in blocks, snapshots, or checkpoints.

Related Pages

• State Root
• iden3 State Root
• iden3 State-Root Contract
• Nervos SMT Sidecar
• Nervos State-Root Contract
• Replay Verification
• Snapshot Internals
• Protocol Versioning
• State-Root Activation