Multi-Validator Hardness E2E

test-e2e:hardness-multiple-producers-noosraft and test-e2e:hardness-multiple-producers-hashicorp-go are separate from test-e2e:hardness-one-producer.

The one-producer hardness test focuses on a producer plus sync-only observers, payload materialization, permissions, policies, snapshots, and sync/backfill boundaries.

The multi-producer hardness test focuses on validators, Raft leader-only block production, validator governance, and the governance/runtime membership boundary.

Architecture

The harness starts three real Fastify validator nodes:

• validator-a
• validator-b
• validator-c

The Noosraft variant uses the in-repo Raft engine over authenticated HTTP for speed and repeatability. The HashiCorp Go variant starts three real Go sidecars and runs the same validator/governance/failover scenario against the external process engine.

Only the current Raft leader commits blocks. Followers may receive transaction submissions, but they return NOT_LEADER and do not produce blocks.

Coverage

The scenario verifies:

• genesis has three active validators
• runtime Raft voter set starts as a,b,c
• leader transaction submission and block commit works
• follower submission is rejected
• governance can register a new intended validator
• runtime Raft membership is not silently changed
• observability warns when governance active set differs from runtime voters
• reconciliation plans identify missing and unexpected runtime voters
• suspend, reactivate, retire, endpoint update, and voting-power update replay deterministically
• peer scoring does not mutate validator governance state
• a committed replicate-all encrypted record can be repaired through payload backfill so every validator stores local ciphertext
• duplicate committed-entry delivery is idempotent and does not create an incident; this protects the HashiCorp callback plus local replay race path
• when the Nervos state-root sidecar binary is built, chain-admin governance schedules a nervos-smt-v2 activation, every validator reports local readiness, and each validator reports a failed activation_height_reached readiness result when probed with a deliberately missing sidecar command
• the current leader refuses to produce the activation block while the required Nervos sidecar command is broken, proving activation-height block production fails closed instead of falling back to iden3
• after restoring the sidecar command, the cluster commits across the activation boundary, then each validator reports a failed post_activation readiness result when probed with a missing sidecar command
• the block before activation stores iden3-v1, the activation block stores nervos-smt-v2, and the Raft block envelope carries the selected engine metadata so followers do not replay the committed block with stale iden3 metadata
• leader failover works while governance state is present
• snapshots preserve governance state
• no validator finishes with an active incident
• all validators converge to the same head and state root

The HashiCorp Go variant additionally verifies:

• sidecar processes are running and supervised
• runtime status reports raftEngine=hashicorp-go
• committed log indexes are visible
• runtime voter membership exposes Raft TCP endpoints
• committed callback idempotency uses the same /internal/raft/committed receiver path used by the HashiCorp sidecar

State-Root Activation Failure Coverage

The state-root activation segment is shared by the Noosraft and HashiCorp Go hardness runners. The harness nodes run in one TypeScript process, so it does not model one operating-system process having a unique broken environment while the others remain unchanged. Instead it covers the production-critical boundary in two ways:

• it checks every validator database independently with a missing sidecar command override at the activation height and after activation;
• it temporarily breaks the process sidecar command and verifies the leader cannot produce the activation block, then restores the command and commits the block normally.

Those checks make the activation path fail closed for both consensus backends without pretending Noosraft or the test harness has per-process sidecar configuration isolation.

Expected Membership Warning

Registering or retiring validators changes consensus-state governance intent. The Raft voter set remains the configured runtime set until an operator approves and executes a runtime membership plan. This is intentional. Governance transactions never silently change Raft voters.

The fast Noosraft E2E path verifies the warning and plan behavior. The HashiCorp Go E2E path verifies the same behavior with real sidecars. The dedicated HashiCorp Raft sidecar runtime mutation verifier exercises actual HashiCorp Raft membership APIs:

npm run test:raft:membership

Command

PowerShell:

npm run test-e2e:hardness-multiple-producers-noosraft
npm run test-e2e:hardness-multiple-producers-hashicorp-go

Bash:

npm run test-e2e:hardness-multiple-producers-noosraft
npm run test-e2e:hardness-multiple-producers-hashicorp-go

Default output is readable pass/fail text. Use --json for the structured summary or --report-dir to write the summary as a CI artifact:

npm run test-e2e:hardness-multiple-producers-noosraft -- --json
npm run test-e2e:hardness-multiple-producers-hashicorp-go -- --report-dir ./test-results/noos

Not Covered

• Raft joint-consensus membership changes
• HashiCorp Raft runtime mutation in these E2E paths; use test:raft:membership for operator-approved runtime add/remove coverage
• Byzantine validator behavior
• slashing
• token staking