NOOSChain Docs

Appearance

Raft Chaos Testing ​

npm run stability:noosraft-chaos runs a local stability harness for Noosraft. It is separate from normal E2E, hardness, size-stress, and soak tests.

The harness starts three NOOSChain voter nodes with isolated PostgreSQL schemas named noos_raft_chaos_*, initializes the same deterministic genesis on each node, starts Raft mode, submits signed transactions only to the current leader, commits blocks through the Raft adapter, and verifies that followers converge through committed Raft log entries.

This is not a production Raft certification suite. It is a development safety harness for crash/restart, leader changes, network disruption, snapshot catch-up, and deterministic NOOSChain replay.

Run ​

PowerShell:

powershell
npm run stability:noosraft-chaos

Bash:

bash
npm run stability:noosraft-chaos

HTTP transport smoke:

powershell
$env:RAFT_CHAOS_TRANSPORT="http"; $env:RAFT_CHAOS_DURATION_MS="120000"; npm run stability:noosraft-chaos

The default transport is in_process, which exposes extra test controls for disconnecting nodes, delaying messages, and dropping messages. HTTP mode starts real Fastify servers and verifies the HTTP Raft transport path, but network partitions/delay/drop injection are currently reported as skipped because the HTTP transport does not yet have a controllable proxy.

Configuration ​

Environment variables:

  • RAFT_CHAOS_DURATION_MS, default 300000
  • RAFT_CHAOS_NODE_COUNT, default 3; the MVP harness requires exactly 3 voters
  • RAFT_CHAOS_TXS_PER_STEP, default 10
  • RAFT_CHAOS_STEP_INTERVAL_MS, default 500
  • RAFT_CHAOS_TRANSPORT, in_process or http
  • RAFT_CHAOS_KILL_LEADER_EVERY_STEPS, default 50
  • RAFT_CHAOS_RESTART_DELAY_STEPS, default 10
  • RAFT_CHAOS_PARTITION_EVERY_STEPS, default 0
  • RAFT_CHAOS_DELAY_MESSAGES, default false
  • RAFT_CHAOS_DROP_MESSAGES, default false
  • RAFT_CHAOS_SNAPSHOT_THRESHOLD_ENTRIES, default 25
  • RAFT_CHAOS_VERIFY_EVERY_STEPS, default 50
  • RAFT_CHAOS_REPLAY_EVERY_STEPS, default 100
  • RAFT_CHAOS_KEEP_DB, default false
  • RAFT_CHAOS_LOG_DIR, default ./raft-chaos-results
  • RAFT_CHAOS_SEED, default raft-chaos-default

The runner refuses NODE_ENV=production. Cleanup drops only schemas matching the noos_raft_chaos_* prefix unless RAFT_CHAOS_KEEP_DB=true.

Scenarios ​

The harness records each scenario in the final report:

  • baseline_raft_commit: elect a leader, commit a block, verify all nodes converge.
  • follower_restart: stop a follower while the leader commits, restart it, and verify catch-up.
  • leader_restart: stop the leader, elect a new one, commit, restart the old leader, and verify catch-up.
  • repeated_leader_failover: repeat leader stop/restart cycles and verify no finalized divergence.
  • minority_partition: in-process only; disconnect one follower, continue with majority, heal, and verify convergence.
  • leader_isolated_from_majority: in-process only; isolate the old leader enough for a new leader to emerge and verify convergence after healing.
  • delayed_messages: in-process only; add transport delay and verify eventual commit.
  • dropped_messages: in-process only; temporarily drop messages and verify recovery.
  • snapshot_catchup_under_chaos: stop a follower beyond the snapshot threshold, restart it, and verify InstallSnapshot catch-up.
  • committed_state_root_mismatch_incident: inject a test-only conflicting committed entry and verify incident mode is entered instead of silently diverging.
  • persistence_restart_all_nodes: stop all nodes, restart with the same schemas, and verify persisted Raft term/vote/log state resumes safely.
  • no_quorum_no_commit: stop two voters and verify the remaining node does not falsely finalize.

Reports ​

Each run writes a timestamped directory under RAFT_CHAOS_LOG_DIR containing:

  • final-report.json: machine-readable summary, scenario results, timings, resources, and issues.
  • events.jsonl: major lifecycle and scenario events.
  • metrics.jsonl: compact metrics snapshots.
  • errors.jsonl: unexpected errors.

Important report fields:

  • summary.finalConvergenceOk: all active nodes ended at the same finalized head.
  • summary.chainVerificationOk: chain verification passed where scenarios required it.
  • summary.replayVerificationOk: deterministic replay passed where scenarios required it.
  • summary.snapshotsInstalled: follower InstallSnapshot count observed during snapshot catch-up.
  • timings.leaderElectionMs, blockCommitMs, catchupMs, snapshotInstallMs: basic p50/p95/p99 timing distributions.

Skipped scenarios are explicit in the report and are not silently omitted.

Not Covered ​

This harness does not cover:

  • Byzantine behavior
  • production-grade Raft certification
  • runtime Raft membership changes
  • external HashiCorp Raft sidecar behavior, which is covered separately by npm run stability:go-raft-chaos
  • OpenBFT or CometBFT
  • libp2p transport
  • production network proxy chaos

Audience-first NOOSChain documentation.

NOOSChain