Multi-Producer Noosraft Soak Test

npm run soak:multiple-producers-noosraft starts a local operational soak with three Noosraft validator/producers and three sync-only observers.

The purpose is to exercise long-running Noosraft ordering plus the same data, payload, observability, and verification surfaces covered by the single-producer soak.

Topology

• validator-a, validator-b, validator-c: Noosraft voters.
• observer-a, observer-b, observer-c: sync-only observers.
• RAFT_ENGINE=noosraft.
• HTTP Raft transport.
• Transaction gossip disabled in the baseline.

"Three producers" means all three validators can receive submissions, but only the current Noosraft leader may propose and commit blocks. Follower submissions are sampled and expected to return NOT_LEADER.

Gossip Variant

This baseline deliberately disables transaction gossip to isolate consensus ordering, operator membership reconciliation, and observer sync. That is useful, but it is not a full real-life network simulation.

The gossip-enabled variant is:

npm run soak:multiple-producers-noosraft:gossip

It keeps observers sync-only, but submits probe transactions to follower validators and verifies follower NOT_LEADER, gossip arrival in the current leader mempool, benign duplicate-gossip rejection, and leader commit.

The Phase 3 variant combines gossip with runtime membership changes under load:

npm run soak:multiple-producers-noosraft:phase3

It also exercises peer-scoring interaction with gossip: manual quarantine/ban blocks inbound gossip from that peer, unquarantine/unban/reset restores it, and peer scoring does not change validator governance or runtime voters.

The Phase 4 variant switches on the large payload / large block profile and adds packaged archive snapshot round-trip verification:

npm run soak:multiple-producers-noosraft:phase4

It exports a consensus-only .noosnap.tar.gz snapshot package, unpacks and verifies the archive, imports it into an isolated empty schema, and replay-verifies from the imported checkpoint. This is the soak path for long-running snapshot import/export safety and for larger block materialization pressure.

Nervos State-Root Activation Coverage

When the Nervos state-root sidecar binary exists, fast-smoke runs enable governed nervos-smt-v2 activation automatically. Custom runs can force the same path with:

NOOS_MP_NOOSRAFT_SOAK_STATE_ROOT_ACTIVATION_ENABLED=true npm run soak:multiple-producers-noosraft

The soak schedules activation through a signed chain-admin transaction while workload is active, commits across the activation boundary, verifies pre-activation blocks use iden3-v1, verifies activation and post-activation blocks use nervos-smt-v2, kills a supervised Nervos state-root sidecar after activation and waits for restart, exports/imports a post-activation snapshot archive, and confirms observers catch up after activation.

Commands

Smoke:

NOOS_MP_NOOSRAFT_SOAK_FAST_SMOKE=true npm run soak:multiple-producers-noosraft
NOOS_MP_NOOSRAFT_SOAK_FAST_SMOKE=true npm run soak:multiple-producers-noosraft:gossip
NOOS_MP_NOOSRAFT_SOAK_FAST_SMOKE=true npm run soak:multiple-producers-noosraft:phase3
NOOS_MP_NOOSRAFT_SOAK_FAST_SMOKE=true npm run soak:multiple-producers-noosraft:phase4
npm run dev:certs
NOOS_MP_NOOSRAFT_SOAK_FAST_SMOKE=true npm run soak:multiple-producers-noosraft:tls

PowerShell:

$env:NOOS_MP_NOOSRAFT_SOAK_FAST_SMOKE="true"; npm run soak:multiple-producers-noosraft

One hour:

NOOS_MP_NOOSRAFT_SOAK_DURATION_MS=3600000 npm run soak:multiple-producers-noosraft

Overnight:

NOOS_MP_NOOSRAFT_SOAK_DURATION_MS=28800000 npm run soak:multiple-producers-noosraft

What It Exercises

• Noosraft leader election.
• Leader-only block proposal.
• Follower NOT_LEADER transaction/proposal behavior.
• Validator block replication and deterministic execution.
• Observer HTTP sync from validators.
• per-record-key and per-bucket-key encrypted records.
• replicate-all, authorized-only, and metadata-only payload policies.
• bucket keys and key rotations.
• access-rule updates.
• bucket policy updates.
• identity and chain-access governance transactions.
• validator governance plus operator reconciliation plans.
• operator trusted-peer execution through /operator/membership/execute.
• Noosraft runtime membership emulation through controlled test config apply.
• payload backfill.
• observability APIs.
• cached observability counters compared with realtime diagnostics.
• snapshot verification and HTTP snapshot export disabled behavior.
• chain and replay verification.
• optional follower-to-leader transaction gossip probes when the :gossip command or NOOS_MP_NOOSRAFT_SOAK_GOSSIP_ENABLED=true is used.
• optional peer-scoring/gossip interaction checks through NOOS_MP_NOOSRAFT_SOAK_PEER_SCORING_EVERY_STEPS.
• runtime membership changes under active workload through the :phase3 command or NOOS_MP_NOOSRAFT_SOAK_MEMBERSHIP_UNDER_LOAD=true.
• large payload / large block stress through the :phase4 command, NOOS_MP_NOOSRAFT_SOAK_PHASE4_ENABLED=true, or NOOS_MP_NOOSRAFT_SOAK_LARGE_PROFILE=true.
• packaged archive snapshot export/verify/import/replay round-trip checks through NOOS_MP_NOOSRAFT_SOAK_SNAPSHOT_STREAMING_EVERY_STEPS.
• governed nervos-smt-v2 activation under load through NOOS_MP_NOOSRAFT_SOAK_STATE_ROOT_ACTIVATION_ENABLED=true or fast-smoke when the sidecar binary is already built.
• supervised Nervos state-root sidecar restart under post-activation load.
• post-activation observer sync/backfill catch-up and snapshot archive export/import/replay verification.
• optional HTTPS/mTLS transport probes when the :tls command or NOOS_MP_NOOSRAFT_SOAK_TLS_ENABLED=true is used. TLS mode covers the TypeScript node APIs and Noosraft HTTP transport using local dev certificates from dev-certs/.

Cached Counter Checks

The soak treats cached observability counters as a correctness surface, not just as UI decoration.

During the active loop it runs a lightweight probe of stable top-level cached counters against realtime diagnostics. That catches obvious drift without making the hot path depend on deep payload-policy comparisons while sync and backfill are still moving data between nodes.

At the end of the run it performs a full settled comparison through the verify_cached_observability_counters_after_quiescence action:

1. validators converge,
2. observers sync,
3. payload backfill completes,
4. observers sync again,
5. each live node compares cached /node/observability/storage and /node/observability/payloads against their /realtime diagnostics.

The final comparison includes row counts, payload totals, replication-policy breakdowns, and missing-payload-by-bucket counters. Any mismatch is reported as CACHED_COUNTERS_DIVERGED_AFTER_QUIESCENCE and fails the soak.

The settled check retries briefly because local operational counters such as peer_score_events can still change for a moment after final sync/backfill returns. Persistent drift after that quiet window is treated as a real counter failure.

Operator Reconciliation

Noosraft does not currently support live joint-consensus membership mutation. The soak still tests the operator workflow explicitly:

1. A governance transaction changes intended validator state.
2. The soak reads /operator/membership/reconciliation-plan.
3. Trusted-peer actions are marked operatorApproved: true.
4. Trusted-peer changes are submitted to /operator/membership/execute.
5. Runtime voter changes are applied by stopping test nodes, updating only the in-memory harness config, and restarting them.
6. The soak waits for leader election and validator convergence.

When the operator-emulated step demotes a validator to observer, the harness also prunes local availability-layer ciphertext that the node should no longer retain under current bucket policy and permissions. This is not a consensus state mutation; it removes rows from the local payload store and clears legacy payload columns for records that should be unavailable to the demoted observer.

This validates that runtime membership changes happen only after an explicit operator action. It is not a claim that Noosraft has live Raft joint consensus. The HashiCorp Go soak uses real HashiCorp membership APIs for the same operator-approved workflow.

Output

The final report kind is:

"multi_producer_noosraft_soak"

Important Noosraft summary fields include:

• leaderElections
• leaderChanges
• leaderSubmitAccepted
• followerSubmitRejected
• blocksProposedByLeader
• followerProposeRejected
• raftCommitsApplied
• validatorsConverged
• observersConverged
• operatorMembershipPlans
• operatorTrustedPeerActionsApplied
• operatorRuntimeConfigApplies
• gossipSubmissions
• gossipCommittedByLeader
• peerScoringGossipBlocked
• peerScoringGossipRestored
• peerScoringGovernanceUnaffected
• snapshotStreamingRoundTrips
• largeProfileBlocks
• stateRootActivationScheduled
• stateRootActivationBlocks
• stateRootPostActivationBlocks
• stateRootSidecarRestartProbes
• stateRootSidecarRestartsObserved
• stateRootSnapshotsAfterActivation
• stateRootObserverCatchupsAfterActivation
• runtimeMembershipChangedOnlyAfterOperatorAction
• noosraftStatusOk
• stateRootActivationUnderLoadOk
• stateRootSidecarRestartUnderLoadOk
• snapshotAfterActivationOk
• observerCatchupAfterActivationOk

events.jsonl and soak.log include action_result entries. Each entry states the action taken, expected result, actual result, and whether it passed.

Safety

The harness refuses NODE_ENV=production. It creates isolated schemas with the prefix noos_soak_multiple_producers_noosraft and only drops schemas matching that prefix unless NOOS_MP_NOOSRAFT_SOAK_KEEP_DB=true.

Not Covered

• HashiCorp Go Raft sidecars.
• live Noosraft joint consensus.
• Byzantine behavior.
• CometBFT or OpenBFT.
• libp2p transport.