NOOSChain Stability Testing

The test/stability tree is intentionally separate from test/integration. E2E tests prove feature behavior. Stability tests measure operating characteristics under repeatable workloads.

Current scope:

• Noosraft chaos coverage under test/stability/noosraft-chaos
• HashiCorp Raft chaos smoke coverage under test/stability/hashicorp-raft-chaos
• isolated PostgreSQL schemas by default
• machine-readable JSON reports

Additional long-running soak coverage lives outside this tree in test/soak/single-producer, test/soak/multiple-producers-noosraft, and test/soak/multiple-producers-hashicorp-go. These are separate because they are operational overnight harnesses rather than bounded stability benchmarks.

Configured but currently unavailable in this workspace:

• benchmark:performance points to a missing src/stability-test/benchmarks/benchmark-runner.ts
• stability:failure-stress points to a missing src/stability-test/failure-stress/failure-stress-runner.ts
• stability:size points to a missing src/stability-test/size-stress/size-stress-runner.ts

Not implemented in the current stability harnesses:

• CometBFT or libp2p behavior

Safety

The active stability harnesses refuse to run with NODE_ENV=production.

stability:noosraft-chaos creates isolated schemas named noos_raft_chaos_* and drops only those generated schemas unless RAFT_CHAOS_KEEP_DB=true. stability:go-raft-chaos currently delegates to the test:raft:go cluster path and writes a HashiCorp Raft chaos report; it uses isolated temporary schemas and sidecar data directories through that verifier path.

The unavailable benchmark/failure/size harness designs use the NOOS_STABILITY_*, NOOS_FAILURE_STRESS_*, and NOOS_SIZE_STRESS_* settings documented in their own files, but those runner files are not present in this workspace.

Configuration

Environment variables for the intended benchmark harness design:

• NOOS_STABILITY_KEEP_DB
• NOOS_STABILITY_SCHEMA_PREFIX
• NOOS_STABILITY_TX_COUNT
• NOOS_STABILITY_TXS_PER_BLOCK
• NOOS_STABILITY_PAYLOAD_SIZE_BYTES
• NOOS_STABILITY_BUCKET_COUNT
• NOOS_STABILITY_RECORD_INDEX_COUNT
• NOOS_STABILITY_EXPORT_REPORT_PATH
• NOOS_STABILITY_VERIFY_REPLAY
• NOOS_STABILITY_VERIFY_CHAIN
• NOOS_STABILITY_INCLUDE_SNAPSHOT
• NOOS_STABILITY_SEED

Sync/runtime tuning shared by the stability harnesses:

• NOOS_SYNC_MAX_BLOCK_RANGE, default 1000, hard maximum 10000
• NOOS_SMT_CACHE_MAX_ENTRIES, default 250000; set 0 only for deliberate local experiments with an unbounded block-local SMT cache

This bounds /chain/blocks and /chain/headers responses even when a test sends a very broad from/to range.

The persistent SMT adapter uses a block-scoped in-memory cache to reduce PostgreSQL round trips during execution. The cache is created fresh for each block execution and flushed inside the same block transaction. If NOOS_SMT_CACHE_MAX_ENTRIES is reached, the adapter logs a warning and stops adding new cache entries for the rest of that block while preserving the pending write buffer needed for correctness. The next block receives a fresh cache.

The intended performance benchmark design uses a signer pool sized to TXS_PER_BLOCK, because NOOSChain nonce rules allow only the next nonce per signer to be pending. That preserves the real mempool/consensus rules while still allowing full blocks once the benchmark runner exists again.

Commands

npm run stability:go-raft-chaos
npm run soak:single-producer
npm run soak:multiple-producers-noosraft
npm run soak:multiple-producers-hashicorp-go

The benchmark, failure-stress, and size-stress documents describe intended designs whose runners are not present in the current workspace. Failure-stress details and scenario coverage are documented in Failure Stress Testing. The single-producer soak harness is documented in Single-Producer Soak. The Noosraft multi-producer soak harness is documented in Multi-Producer Noosraft Soak. The HashiCorp Go multi-producer soak harness is documented in Multi-Producer HashiCorp Go Soak. The overall npm script naming convention is documented in Command Taxonomy.