NOOSChain Docs

Appearance

Production Release Readiness ​

Operator checklist for admitting a production deployment.

Source: docs\operations\production-readiness.md.

Admission Command ​

Use the operator CLI from the node folder that contains the intended .env:

powershell
npm run noos -- production report --json
npm run noos -- production profile --profile validator
npm run noos -- production preflight
bash
npm run noos -- production report --json
npm run noos -- production profile --profile validator
npm run noos -- production preflight

production report always returns the full checklist. production preflight fails closed when any required check is red. Both commands are read-only: they do not change consensus state, Raft membership, trusted peers, snapshots, payload availability, incidents, or sidecar processes.

production profile returns the profile and release compatibility subset of the same report. Use --profile validator, --profile observer, or --profile single-node to make the expected role explicit. If no profile is provided, the CLI uses NOOS_PRODUCTION_PROFILE or infers the profile from node observability.

Production Use Sequence ​

  1. Build and stage artifacts:

    powershell
    npm run build
npm run build:raft-go
npm run build:state-root-sidecar-nervos
npm run admin:build

  2. Run database migrations:

    powershell
    npm run migrate

  3. Start the node and sidecars with production .env values:

    • NODE_ENV=production
    • NOOS_TLS_ENABLED=true
    • NOOS_MTLS_REQUIRED=true where possible
    • NOOS_OPERATOR_TOKEN set to a non-development secret
    • NOOS_CONSENSUS_MODE=raft for validator clusters
    • RAFT_ENGINE=hashicorp-go for production Raft candidates
    • GO_RAFT_TCP_TLS_ENABLED=true
    • GO_RAFT_TCP_MTLS_REQUIRED=true where possible
    • NOOS_STATE_ROOT_ENGINE=nervos-smt-v2 for new production chains
    • NOOS_STATE_ROOT_SIDECAR_COMMAND pointing at the deployed Nervos sidecar binary
    • NOOS_PRODUCTION_PROFILE=validator or observer to make the intended node role explicit for CLI preflight

  4. Verify operator access:

    powershell
    npm run noos -- doctor --deep

  5. Verify state-root readiness:

    powershell
    npm run noos -- state-root preflight

  6. Verify production admission:

    powershell
    npm run noos -- production profile --profile validator
npm run noos -- production preflight

  7. Export and verify a local backup:

    powershell
    npm run noos -- snapshots export --output node-backup.noosnap.tar.gz
npm run noos -- snapshots verify --file node-backup.noosnap.tar.gz

  8. Prove the backup can restore:

    powershell
    npm run noos -- production backup-drill --output ./backup-drill

  9. Review the role-specific restore/rejoin procedure before launch:

State-Root Activation ​

Production-like state-root activation should use the consensus-governed SCHEDULE_STATE_ROOT_ENGINE_ACTIVATION transaction. Before the activation height approaches, run:

powershell
npm run noos -- state-root preflight
npm run noos -- production preflight

The activation transaction only changes consensus state. It does not deploy sidecar binaries, rewrite config files, or restart validators. Every validator must already be able to execute the scheduled engine before the activation height arrives.

nervos-smt-v2 is the default state-root engine for new chains. Production safety comes from sidecar readiness, timeout handling, checkpoint validation, and fail-closed block execution. Existing iden3-v1 chains still need a consensus-governed activation boundary before they move to Nervos.

When Nervos is scheduled, configure sidecar lifecycle controls explicitly:

powershell
$env:NOOS_STATE_ROOT_SIDECAR_COMMAND="C:\path\to\noos-state-root-sidecar-nervos.exe"
$env:NOOS_STATE_ROOT_SIDECAR_PROTOCOL_VERSION="1"
$env:NOOS_STATE_ROOT_SIDECAR_TIMEOUT_MS="60000"
$env:NOOS_STATE_ROOT_SIDECAR_AUTO_RESTART="true"
$env:NOOS_STATE_ROOT_SIDECAR_MAX_RESTARTS="3"
$env:NOOS_STATE_ROOT_SIDECAR_RESTART_BACKOFF_MS="2000"
$env:NOOS_STATE_ROOT_SIDECAR_SUPERVISOR_ENABLED="auto"
$env:NOOS_STATE_ROOT_SIDECAR_HEALTH_INTERVAL_MS="10000"
$env:NOOS_STATE_ROOT_SIDECAR_PREWARM="false"
$env:NOOS_STATE_ROOT_CHECKPOINT_DIR="D:\nooschain\state-root-checkpoints"

Operator observability reports continuous supervisor state under consensus.stateRootEngine.supervisor: policy, whether the sidecar is required now, process state, health interval, last health check, last healthy timestamp, last error, version, health, restart count, and child process metadata. It also reports the latest sidecar session under consensus.stateRootEngine.sidecar.

production preflight treats this as a hard gate. Missing readiness, red readiness, wrong sidecar engine/protocol/checkpoint format, failed health, a non-running required supervisor, or repeated restart symptoms block production admission. Treat supervisor.state=degraded, missing health, recent timeout timestamps, or repeated restarts as a launch blocker near or after activation. Once a governed Nervos activation applies, NOOSChain must fail closed instead of silently executing blocks with iden3-v1.

When the supervisor is active, Nervos block execution uses the continuous sidecar through an exclusive execution lease. Each completed block exports a verified local checkpoint and records its file hash in state_root_sidecar_checkpoints. Startup/execution trusts only verified rows whose file still hashes to the recorded value.

Back up NOOS_STATE_ROOT_CHECKPOINT_DIR together with the PostgreSQL database. The checkpoint directory is local operational cache, not consensus state, but it is the fast restart path for post-activation Nervos nodes.

Backup And Restore Drill ​

production backup-drill is the operator proof that backup is more than a file on disk. It uses local DATABASE_URL and never calls HTTP snapshot export.

The drill:

  1. exports a consensus-state-only snapshot archive from the local DB;
  2. packages it as backup-drill.noosnap.tar.gz;
  3. unpacks and verifies the package;
  4. imports it into a generated isolated schema;
  5. runs chain verification on the restored schema;
  6. imports the same package into a second generated schema as a checkpoint replay/bootstrap proof;
  7. writes backup-drill-report.json in the output directory.

For production smart contracts, the drill must be run after contracts have been published, instantiated, called, migrated, and deactivated at least once in the rehearsal environment. The snapshot archive now covers contract code bytes, manifests, approved registry entries, runtime activation/rollout consensus rows, contract instances, contract state, events, structured call results, release workflow rows, and migration rows. Readiness and metrics are intentionally local operational rows; they should be rebuilt or re-collected after restore.

Run a dry-run first:

powershell
npm run noos -- production backup-drill --output ./backup-drill --dry-run

Then run the real drill:

powershell
npm run noos -- production backup-drill --output ./backup-drill

The generated restore schemas are dropped after the command finishes. The output directory keeps the packaged snapshot and the JSON report. The command does not modify the source chain schema.

For an actual node restore, do not treat backup-drill as the restore command. It is a proof. Follow the role-specific runbook in Backup, Restore, And Recovery so the node is stopped, restored into the right database, verified offline, and rejoined in the right role.

Consensus ​

For production validator clusters, treat HashiCorp Go Raft as the current Raft candidate. Noosraft remains useful for development and deterministic test harnesses, but it is not the production Raft engine.

Runtime Raft membership is still operational state. Governance validator transactions do not automatically mutate runtime membership. The operator flow is:

powershell
npm run noos -- governance operator-membership plan --json
npm run noos -- governance operator-membership execute --plan-file approved-plan.json --yes
npm run noos -- production preflight

Security ​

Production deployments should use:

  • HTTPS for TypeScript node APIs.
  • mTLS for operator, node-to-node, and Raft HTTP paths where possible.
  • TLS/mTLS for HashiCorp Raft TCP transport.
  • Non-development NOOS_OPERATOR_TOKEN.
  • A certificate rotation plan.
  • No private keys, DEKs, operator tokens, plaintext payloads, or encrypted payload bytes in logs or support bundles.

Before enabling production smart contracts, complete the Smart Contract Security Review Checklist. That checklist covers the Wasmtime runtime, host imports, registry policy, contract-to-contract calls, bucket access, migrations, replay, backup/restore, monitoring, and operator procedures.

External reviewers should start with the Smart Contract Audit Package and Smart Contract Threat Model before checking the readiness evidence.

Generate the automated launch-gate and sidecar provenance evidence:

powershell
npm run noos -- contracts launch-gate --profile mainnet --json
npm run noos -- contracts runtime provenance --json
npm run contracts:sidecar:provenance

Incident And Recovery ​

If preflight fails because of an incident, do not mark it resolved first. Use:

powershell
npm run noos -- incidents doctor --json
npm run noos -- support bundle --output noos-support-bundle.json

Only mark an incident as resolved after the underlying cause has been fixed and verification confirms no active divergence or availability issue remains.

If the incident requires replacing or restoring a node, follow Backup, Restore, And Recovery. Validator recovery is especially sensitive because the TypeScript chain database and the HashiCorp Raft sidecar data directory must not rejoin the cluster in a mismatched state.

Observability And Alerting ​

The preflight is not a replacement for alerting. Use the read-only monitoring report for polling/scheduled operator checks:

powershell
npm run noos -- production monitor-report --json

Production monitoring should alert on:

  • active incident;
  • consensus degraded;
  • missing leader or no quorum;
  • HashiCorp sidecar stopped or restarting repeatedly;
  • state-root readiness failed;
  • sync/backfill stuck;
  • peer scoring collapse;
  • payload availability unexpectedly missing;
  • snapshot backup missing or stale.

See Production Monitoring And Alerting for the full severity matrix, first diagnostic command, and recommended operator action for each alert.

For nervos-smt-v2, state-root readiness is a hard activation gate. Production preflight fails when the scheduled/current state-root engine is not ready. At or after the activation height, there is no silent fallback to iden3-v1; fix the sidecar or keep the node out of validator traffic.

The Admin GUI mirrors the same data. Dashboard shows a state-root readiness summary. Membership/Consensus show the active activation, next scheduled activation, readiness checks, Nervos sidecar supervisor state, sidecar process state, version/protocol/checkpoint metadata, and latest verified checkpoint.

For production smart contracts, use the Smart Contract Web Console as the operator surface for launch payload review, registry status, readiness, migration planning, deactivation dependency checks, monitoring, and launch-gate reports. Before mainnet review, run the Smart Contract Release Candidate Gate and the Smart Contract Incident Drills. Use the Smart Contract Package Signing Policy before approving packages. Use Smart Contract Layer Readiness to verify the current smart-contract layer evidence.

Use Nervos Activation Runbook for planned rollout and Nervos Sidecar Recovery for post-failure repair.

Performance Acceptance ​

Before launch, set explicit local acceptance thresholds for:

  • block commit p95;
  • observer catch-up time;
  • payload backfill throughput;
  • snapshot export/import duration;
  • replay verification duration;
  • sidecar restart recovery time.

Use benchmark:v1-readiness and benchmark:v1-readiness:check to measure those thresholds against the draft budgets in V1 Performance Targets. Keep soak, stability, and E2E runs separate; do not treat a successful preflight or budget check as a performance soak.

Release Packaging ​

Production artifacts should be versioned together:

  • TypeScript node build;
  • admin GUI build;
  • HashiCorp Raft sidecar binary;
  • state-root sidecar binary when enabled;
  • migration set;
  • documentation/runbook version.

The node should be able to report the versions and readiness status operators need before it joins production traffic.

The current compatibility gate verifies the node package version against the CLI package version and verifies local database migrations against the release's src/db/migrations directory. It also records the supported snapshot archive format version in the report. Future release gates should add explicit HashiCorp sidecar binary version stamping and state-root sidecar build metadata once those binaries expose stable release identifiers.

Audience-first NOOSChain documentation.

NOOSChain