Multi-Producer HashiCorp Go Raft Soak Test

npm run soak:multiple-producers-hashicorp-go starts a local operational soak with three HashiCorp Go Raft validator/producers and three sync-only observers.

The purpose is to exercise long-running external Raft sidecar ordering plus the same data, payload, observability, and verification surfaces covered by the Noosraft multi-producer soak.

Topology

• validator-a, validator-b, validator-c: HashiCorp Go Raft voters.
• observer-a, observer-b, observer-c: sync-only observers.
• RAFT_ENGINE=hashicorp-go.
• Local sidecars use HTTP control/callback APIs and TCP Raft transport.
• Transaction gossip disabled in the baseline.
• Observer manual sync keeps payload backfill enabled. After block metadata converges, observers try trusted peers for missing ciphertext so the soak validates eventual payload availability instead of depending on the first sync peer having local bytes.

"Three producers" means all three validators can receive submissions, but only the current Raft leader may propose and commit blocks. Follower submissions are sampled and expected to return NOT_LEADER.

Gossip Variant

This baseline deliberately disables transaction gossip to isolate consensus ordering, sidecar lifecycle, operator membership reconciliation, and observer sync. That is useful, but it is not a full real-life network simulation.

The gossip-enabled variant is:

npm run soak:multiple-producers-hashicorp-go:gossip

It keeps observers sync-only, but submits probe transactions to follower validators and verifies follower NOT_LEADER, gossip arrival in the current HashiCorp leader mempool, benign duplicate-gossip rejection, and leader commit.

The Phase 3 variant combines gossip with runtime membership changes under load:

npm run soak:multiple-producers-hashicorp-go:phase3

It also exercises peer-scoring interaction with gossip: manual quarantine/ban blocks inbound gossip from that peer, unquarantine/unban/reset restores it, and peer scoring does not change validator governance or runtime voters.

The Phase 4 variant switches on the large payload / large block profile and adds packaged archive snapshot round-trip verification:

npm run soak:multiple-producers-hashicorp-go:phase4

It exports a consensus-only .noosnap.tar.gz snapshot package, unpacks and verifies the archive, imports it into an isolated empty schema, and replay-verifies from the imported checkpoint. In fast-smoke mode the sidecar restart probe is disabled by default so this run isolates snapshot and large-block pressure.

Nervos State-Root Activation Coverage

When the Nervos state-root sidecar binary exists, fast-smoke runs enable governed nervos-smt-v2 activation automatically. Custom runs can force the same path with:

NOOS_MP_HASHICORP_GO_SOAK_STATE_ROOT_ACTIVATION_ENABLED=true npm run soak:multiple-producers-hashicorp-go

The soak schedules activation through a signed chain-admin transaction while HashiCorp Raft is ordering workload blocks, commits across the activation boundary, verifies pre-activation blocks use iden3-v1, verifies activation and post-activation blocks use nervos-smt-v2, kills a supervised Nervos state-root sidecar after activation and waits for restart, exports/imports a post-activation snapshot archive, and confirms observers catch up after activation.

Commands

Smoke:

NOOS_MP_HASHICORP_GO_SOAK_FAST_SMOKE=true npm run soak:multiple-producers-hashicorp-go
NOOS_MP_HASHICORP_GO_SOAK_FAST_SMOKE=true npm run soak:multiple-producers-hashicorp-go:gossip
NOOS_MP_HASHICORP_GO_SOAK_FAST_SMOKE=true npm run soak:multiple-producers-hashicorp-go:phase3
NOOS_MP_HASHICORP_GO_SOAK_FAST_SMOKE=true npm run soak:multiple-producers-hashicorp-go:phase4
npm run dev:certs
NOOS_MP_HASHICORP_GO_SOAK_FAST_SMOKE=true npm run soak:multiple-producers-hashicorp-go:tls

PowerShell:

$env:NOOS_MP_HASHICORP_GO_SOAK_FAST_SMOKE="true"; npm run soak:multiple-producers-hashicorp-go

One hour:

NOOS_MP_HASHICORP_GO_SOAK_DURATION_MS=3600000 npm run soak:multiple-producers-hashicorp-go

Overnight:

NOOS_MP_HASHICORP_GO_SOAK_DURATION_MS=28800000 npm run soak:multiple-producers-hashicorp-go

What It Exercises

• HashiCorp Go sidecar startup and status.
• Raft leader election.
• Leader-only block proposal.
• Follower NOT_LEADER transaction/proposal behavior.
• Validator block replication and deterministic execution.
• Observer HTTP sync from validators.
• encrypted records, bucket keys, access rules, policy updates, identity and governance transactions.
• operator membership reconciliation path.
• payload backfill.
• observability APIs and cached counter diagnostics.
• snapshot verification and HTTP snapshot export disabled behavior.
• chain and replay verification.
• optional follower-to-leader transaction gossip probes when the :gossip command or NOOS_MP_HASHICORP_GO_SOAK_GOSSIP_ENABLED=true is used.
• optional peer-scoring/gossip interaction checks through NOOS_MP_HASHICORP_GO_SOAK_PEER_SCORING_EVERY_STEPS.
• runtime membership changes under active workload through the :phase3 command or NOOS_MP_HASHICORP_GO_SOAK_MEMBERSHIP_UNDER_LOAD=true.
• large payload / large block stress through the :phase4 command, NOOS_MP_HASHICORP_GO_SOAK_PHASE4_ENABLED=true, or NOOS_MP_HASHICORP_GO_SOAK_LARGE_PROFILE=true.
• packaged archive snapshot export/verify/import/replay round-trip checks through NOOS_MP_HASHICORP_GO_SOAK_SNAPSHOT_STREAMING_EVERY_STEPS.
• governed nervos-smt-v2 activation under load through NOOS_MP_HASHICORP_GO_SOAK_STATE_ROOT_ACTIVATION_ENABLED=true or fast-smoke when the sidecar binary is already built.
• supervised Nervos state-root sidecar restart under post-activation load.
• post-activation observer sync/backfill catch-up and snapshot archive export/import/replay verification.
• optional HTTPS/mTLS transport probes when the :tls command or NOOS_MP_HASHICORP_GO_SOAK_TLS_ENABLED=true is used. TLS mode covers TypeScript node APIs, HashiCorp Go HTTP control/callback APIs, and the HashiCorp TCP Raft transport using local dev certificates from dev-certs/.
• HashiCorp sidecar supervision under active load. The baseline/TLS smoke path can terminate a non-leader sidecar and expects the TypeScript supervisor to restart it from the same data directory without finalized divergence. The sidecar death is expected to create an incident; the soak marks only that controlled incident resolved after restart and validator convergence, so the final no-active- incident assertion still catches unexpected incidents.

Cached Counter Checks

The soak uses the same counter model as the Noosraft soak. The active loop runs a lightweight cached-vs-realtime probe; the final verify_cached_observability_counters_after_quiescence action performs the full settled comparison and fails the soak on persistent drift.

Runtime Membership

The baseline smoke keeps HashiCorp runtime membership static so it can focus on ordering, sidecar startup, convergence, observer sync, and observability. The :phase3 command sets membershipUnderLoad=true, which runs the operator-approved membership sequence during active workload:

1. register observer-c as an intended validator through governance,
2. execute trusted-peer reconciliation,
3. add observer-c to the HashiCorp runtime voter set,
4. later suspend validator-b,
5. remove validator-b from runtime voters.

When a validator is demoted to observer, the harness prunes local availability-layer ciphertext that the node should no longer retain under current bucket policy and permissions. This removes rows from the local payload store and clears legacy payload columns only; it does not mutate consensus state or change state roots.

Set NOOS_MP_HASHICORP_GO_SOAK_OPERATOR_MEMBERSHIP_EVERY_STEPS to a positive value to exercise the same path during a custom longer run. The :phase3 command disables the sidecar restart probe by default to isolate membership/gossip/peer-scoring behavior; set NOOS_MP_HASHICORP_GO_SOAK_SIDECAR_RESTART_EVERY_STEPS explicitly to combine the scenarios.

Output

The final report kind is:

"multi_producer_hashicorp_go_soak"

Important HashiCorp/Nervos summary fields include:

• sidecarRestartProbes
• sidecarRestartsObserved
• stateRootActivationScheduled
• stateRootActivationBlocks
• stateRootPostActivationBlocks
• stateRootSidecarRestartProbes
• stateRootSidecarRestartsObserved
• stateRootSnapshotsAfterActivation
• stateRootObserverCatchupsAfterActivation
• hashicorpGoRaftStatusOk
• stateRootActivationUnderLoadOk
• stateRootSidecarRestartUnderLoadOk
• snapshotAfterActivationOk
• observerCatchupAfterActivationOk

Safety

The harness refuses NODE_ENV=production. It creates isolated schemas with the prefix noos_soak_mp_hcg and isolated sidecar data directories under the soak output area. Cleanup removes both unless NOOS_MP_HASHICORP_GO_SOAK_KEEP_DB=true.

Not Covered

• Byzantine behavior.
• CometBFT or OpenBFT.
• libp2p transport.
• transaction gossip in the baseline run.