Two-node hardness extended testing

npm run test-e2e:hardness-one-producer runs the "two node hardness extended" integration scenario. It is a second, more aggressive two-node test and does not replace npm run test-e2e:simple.

This harness remains the one-producer/observer hardness test. Multi-validator Raft and validator-governance behavior lives in the separate test-e2e:hardness-multiple-producers-noosraft and test-e2e:hardness-multiple-producers-hashicorp-go harnesses documented in Multi-Validator Hardness E2E.

The normal E2E test proves the happy-path lifecycle across two HTTP nodes. The hardness test pushes malformed inputs, negative authorization paths, replayed nonces, bad node-auth headers, payload availability boundaries, backfill behavior, and replay determinism.

It also runs a focused Raft MVP check in isolated schemas so the hardness gate covers consensus-mode configuration, leader election, follower submission rejection, replicated committed blocks, and state-root convergence without turning the main two-node HTTP scenario into a full Raft soak.

How To Run

Windows PowerShell:

npm run test-e2e:hardness-one-producer

Bash:

npm run test-e2e:hardness-one-producer

Default output is readable pass/fail text. Use --json for the structured summary or --report-dir to write the summary as a CI artifact:

npm run test-e2e:hardness-one-producer -- --json
npm run test-e2e:hardness-one-producer -- --report-dir ./test-results/noos

To keep generated schemas for debugging:

Windows PowerShell:

$env:NOOSCHAIN_E2E_KEEP_DB="true"; npm run test-e2e:hardness-one-producer

Bash:

NOOSCHAIN_E2E_KEEP_DB=true npm run test-e2e:hardness-one-producer

The test environment refuses to run with NODE_ENV=production, binds only to localhost, and only drops generated schemas matching the E2E schema naming pattern.

What It Starts

• Two real Fastify NOOSChain servers
• Two isolated PostgreSQL schemas
• Full migrations in both schemas
• Single-node consensus adapters
• HTTP sync, transaction submission, transaction gossip, auth, record, and backfill APIs

Coverage

The hardness scenario verifies:

• safe empty head/info before sync
• genesis sync into Node B
• rejection of a second/different genesis after sync
• malformed transaction submission and gossip rejection
• unsupported protocol version rejection in gossip
• duplicate, stale, skipped, and concurrent nonce behavior
• deterministic nonce consumption by failed domain transactions
• permission denial, access-rule update/removal, read-only grants, write-only grants, admin implication, and actor mismatch failures
• bucket policy mutation, including non-admin denial, encryption-mode-change rejection, future-only replication changes, and no automatic ciphertext deletion/backfill
• database-level ciphertext materialization checks for access-rule downgrade/upgrade and policy transitions
• bucket index schema valid and invalid public indexes
• all payload replication policies:
  • replicate_encrypted_to_all_nodes
  • replicate_encrypted_to_authorized_nodes
  • metadata_only
• bounded /chain/blocks behavior, ciphertext-free consensus block DTOs, and sync catch-up with batched block transaction loading
• missing ciphertext sync with matching state roots
• permission-loss replication skip, permission-regain replication, pre-permission backfill skip, wrong peer failure, authorized batched backfill with client-side chunking, direct oversized payload batch rejection, and metadata-only skip
• encryption/decryption success and tamper failure cases
• per-bucket-key negative cases: record-before-key, key on per-record bucket, skipped versions, duplicate initial key, non-admin create/rotate, wrong key references, stale key references after rotation, and old record version pinning
• real per-bucket-key AES-GCM records decrypt through bucket-key envelopes for both version 1 and version 2
• raw bucket DEKs are checked against materialized key/record rows and canonical transaction envelopes to make sure plaintext key material does not leak into PostgreSQL or consensus data
• challenge reuse rejection, wrong signature rejection, bearer auth, deterministic expired bearer-token rejection through ManualClock, malformed bearer rejection, and injected production-mode public-key-header rejection
• node gossip auth failures for missing, unknown, stale, bad, and tampered headers
• peer scoring edges: healthy peer scoring, malformed peer penalties, quarantine threshold, banned-peer rejection, conflict scoring, gossip scoring, backfill scoring, manual ban/unban/reset, and unchanged validator membership/voting power
• sync no-op, batched catch-up, duplicate sync, and unsupported protocol verification in a rolled-back transaction
• observer read-only boundaries for transaction admission, transaction gossip, and block proposal, while sync/read/verification paths remain usable
• operator-only node observability routes, including incident visibility, peer/quarantine visibility, missing payload counters, and secret/ciphertext redaction
• Noosraft in-process mode: three voters elect a leader, followers reject direct transaction submission, a leader block commit is replicated, and all voters converge on the same state root
• finalized conflict handling: the deterministic-finality policy is checked for every declared conflict type (same_height_different_hash, previous_hash_mismatch, genesis_mismatch, protocol_version_mismatch, state_root_mismatch, block_hash_mismatch, and merkle_root_mismatch)
• sync-side finalized conflict handling: same-height different-hash peer data records a conflict, quarantines the peer, enters incident mode, refuses sync/propose while active, leaves the local head unchanged, and clears only through manual resolution
• malformed finalized block conflicts: previous-hash, Merkle-root, block-hash, and state-root mismatches are exercised through the HTTP sync path and must leave the local head unchanged
• malformed peer sync smoke checks for bad head/block responses with unchanged local head
• snapshot export/import, checkpoint replay, snapshot tamper rejection, ciphertext export-mode rules, and post-checkpoint sync boundary checks
• mempool cleanup when gossiped transactions later arrive in committed blocks
• protected read boundaries and public chain-head access
• final chain verification and replay verification on both nodes

The test deliberately confirms that materialized encrypted_payload availability may differ across nodes while block hashes and state roots still converge.

Raft crash/restart, leader-change, partition, delayed-message, dropped-message, snapshot catch-up, no-quorum, and persistence restart checks live in the separate npm run stability:noosraft-chaos harness. The hardness test keeps only limited Raft smoke coverage so normal E2E feedback does not become a long-running consensus chaos suite.

Intentionally Not Covered Yet

• production-grade Raft consensus certification
• CometBFT
• libp2p
• adversarial Byzantine validators
• validator slashing
• long-running performance/load testing
• full fork-choice behavior, because NOOSChain currently assumes a trusted topology and detects conflicts rather than resolving forks
• automatic pruning of pre-checkpoint history after snapshot bootstrap

Those areas are future milestones. This test is meant to harden the deterministic HTTP/single-node substrate before those distributed consensus and network layers arrive.