Pre-v1 Readiness Debt

This page tracks the remaining release-readiness checks for a v1 launch. It is not a list of missing features. Completed engineering work is called out so the remaining entries stay focused on sign-off, measurement, and operator rehearsal.

State-Root Engine

Status: implemented. Nervos is the main SMT implementation for new chains.

Current path:

• New chains default to nervos-smt-v2.
• The Nervos sidecar under external/state-root-sidecar-nervos computes consensus state roots.
• TypeScript remains the deterministic NOOSChain state machine: it produces ordered state mutations, verifies block/state-root metadata, drives replay, and owns snapshots.
• Existing iden3-v1 histories remain supported for replay and migration tests. They must cross an explicit activation boundary before moving to Nervos because the root schemes differ.

Implemented / covered:

• Use State-Root V1 Readiness as the release checklist for the state-root engine.
• Nervos sidecar readiness is mandatory in noos state-root preflight and noos production preflight.
• Nervos sidecar restart/backoff, checkpoint recovery, corrupt-checkpoint rejection, wrong-version rejection, and timeout/degraded handling are covered by state-root sidecar tests.
• Noosraft and HashiCorp-Go hardness/soak paths cover Nervos activation, sidecar restart under load, snapshot export/import after activation, and observer catch-up after activation.
• State-proof generation/verification is covered by the Nervos proof test, HTTP API, CLI commands, and CLI/API docs.
• The V1 readiness benchmark matrix includes block execution, state-root engine comparison, snapshot archive import/export, sync catch-up, payload backfill, and persistent sidecar checkpoint/restart checks.

Remaining release sign-off:

• Review and sign off V1 Performance Targets for the intended production workload.
• Run benchmark:v1-readiness and benchmark:v1-readiness:check on production-like hardware and archive the reports with the release candidate.
• Treat any budget failure as a release blocker until the budget is revised with rationale or the implementation is optimized.

How to choose the production budgets:

1. Define the v1 workload envelope: expected validators/observers, average and peak submitted transactions per second, expected transaction mix, average public-index fanout, payload sizes, and largest expected catch-up window.
2. Choose an operator-visible block cadence target. For example, decide whether a committed block should normally land within 1 second, 2 seconds, or 5 seconds under peak-but-normal load.
3. Set p95 and p99 budgets from that cadence. A practical v1 starting point is p95 below the desired cadence and p99 below twice the cadence, with failed deterministic transactions included in the mix.
4. Define recovery budgets separately from steady-state commit latency: replay time for a known chain height, snapshot export/import time for expected data sizes, observer catch-up time after downtime, and payload backfill time for authorized ciphertext.
5. Run the benchmark matrix on the intended production hardware and database class, not only on developer machines.
6. Add headroom. Treat any benchmark that already uses more than about 60-70% of the budget as a pre-v1 optimization or capacity issue.
7. Record the chosen numbers in V1 Performance Targets and make benchmark reports and release notes point to the same targets.

Sync And Backfill Recovery

Status: implemented and benchmarked locally. Still requires release-scale measurement.

Implemented / covered:

• Sync manager filters active trusted peers through peer scoring and avoids scoring-blocked peers.
• Eligible peers are ordered by score/status, while payload backfill can recover from a different peer than block sync.
• Sync catch-up benchmarks cover fresh observer, snapshot bootstrap, observer restart recovery, policy/permission changes, missing payloads, and multi-peer failover.
• Payload backfill benchmarks cover the local payload store path and cached payload-store byte/count stats.
• Multi-producer Phase 3 soaks cover peer-scoring penalties, manual quarantine/ban, gossip restoration, and membership changes under load.
• Multi-producer Phase 4 soaks cover large payload/block profiles plus packaged snapshot export/verify/import/replay round trips.
• Cached storage/payload observability counters have realtime diagnostic endpoints and parity tests.
• Admin auto-refresh is disabled by default and expensive storage/payload views have explicit safeguards.

Remaining release sign-off:

• Run sync catch-up and payload backfill portions of benchmark:v1-readiness on production-like hardware with realistic chain height and payload volume.
• Include the Phase 3 and Phase 4 soak variants in the overnight evidence bundle. Baseline soak alone is not enough: Phase 3 is the coverage for scoring penalties and membership changes under load, while Phase 4 is the coverage for large payload/block pressure and packaged snapshot round trips.

Consensus Operations

Status: implemented. HashiCorp-Go Raft is the production Raft path; Noosraft remains useful for deterministic tests and development.

Implemented / covered:

• Runtime membership changes are explicit and operator-approved.
• Governance validator set, runtime Raft membership, and trusted peers are reconciled through operator tooling rather than silent mutation.
• HashiCorp-Go runtime membership changes use the sidecar membership APIs.
• TLS/mTLS production preflight is mandatory for TypeScript HTTP, HashiCorp sidecar HTTP, and HashiCorp Raft TCP transport.
• Chaos/stability coverage exists for sidecar death, unresponsive sidecars, quorum loss, leader failover, restart persistence, and membership changes.

Remaining release sign-off:

• Run the HashiCorp-Go and Noosraft multi-validator hardness suites for the release candidate and archive reports.
• Run the overnight soak variants long enough to include leader failover, sidecar restart, observer catch-up, gossip/TLS where applicable, and operator-approved membership reconciliation. Use the Phase 3 soak variants for membership/scoring/gossip coverage and Phase 4 variants for large payload/block plus snapshot archive coverage.

Operator Experience

Status: implemented. CLI and Admin GUI exist; launch workflows still need a clean-workstation rehearsal.

Implemented / covered:

• Operator CLI loads the current folder .env, has doctor, production preflight, monitor report, backup drill, state-root preflight, membership reconciliation, snapshot import/export, and transaction-building commands.
• Admin GUI is read-mostly, uses operator token auth, hides raw JSON behind inspectors, disables auto-refresh by default, and guards dangerous actions.
• Snapshot export/import is CLI-local and archive-first; HTTP/Admin GUI export is disabled for large-file safety.
• Production preflight catches missing TLS, missing sidecars, wrong sidecar versions, unsafe production defaults, and database/migration incompatibility.

Remaining release sign-off:

• Run the CLI recipes for initialization, bucket setup, identity registration transactions, membership reconciliation, snapshot export/import, and incident response from a clean operator workstation.
• Confirm a new operator can follow the runbooks without project-internal context.
• Confirm the release artifact layout matches the documented commands on Windows and Linux.

Documentation And Release Sign-Off

Status: implemented as living documentation. Release sign-off still needs a final doc review.

Implemented / covered:

• docs:build builds the generated TypeDoc reference and VitePress site.
• .env.example is grouped by context and documents known configuration keys.
• Command taxonomy tracks current script names and command purpose.
• Runbooks exist for Nervos sidecar operations and recovery, HashiCorp-Go Raft deployment, TLS/mTLS, snapshot export/import, incident response, validator governance, runtime membership reconciliation, and production readiness.

Remaining release sign-off:

• Keep docs:build green and review the generated site navigation.
• Review the generated docs from a clean checkout before release.
• Archive the final release evidence bundle:
  • benchmark:v1-readiness report;
  • benchmark:v1-readiness:check report;
  • Noosraft and HashiCorp-Go hardness reports;
  • overnight soak reports;
  • production preflight and state-root preflight reports;
  • backup drill report;
  • exact TypeScript node, HashiCorp sidecar, Nervos sidecar, migration, and docs versions.