Failure-Stress Stability Testing

Current Status

package.json currently contains stability:failure-stress, but the referenced runner file src/stability-test/failure-stress/failure-stress-runner.ts is not present in this workspace. Treat this document as the intended failure-stress design until that runner is restored or reimplemented.

The intended harness lives under src/stability-test/failure-stress.

This is not the same as E2E correctness testing. E2E proves expected workflows through public APIs. Failure stress repeatedly pushes broken peers, malformed transactions, invalid sync responses, nonce storms, block shape pressure, permission churn, key/policy edge cases, large payloads, and snapshot tampering while checking that the local chain remains deterministic and recoverable.

Safety Model

The intended harness refuses NODE_ENV=production through the shared stability config. It creates isolated PostgreSQL schemas only and drops only schemas whose names start with the configured stability prefix. It never resets the normal dev schema and never drops a database.

The default schema prefix for this harness is noos_failure. You can override the shared stability prefix with NOOS_STABILITY_SCHEMA_PREFIX, but it must still be a safe noos... lowercase identifier.

Intended Running Commands

These commands are documented for the intended runner. In the current workspace, they fail until src/stability-test/failure-stress/failure-stress-runner.ts is restored.

Bash:

npm run stability:failure-stress

Windows PowerShell:

npm run stability:failure-stress

Keep generated schemas for inspection:

$env:NOOS_FAILURE_STRESS_KEEP_DB="true"; npm run stability:failure-stress

Export the full JSON report:

$env:NOOS_FAILURE_STRESS_EXPORT_REPORT_PATH="C:\tmp\noos-failure-stress.json"; npm run stability:failure-stress

Configuration

• NOOS_FAILURE_STRESS_ITERATIONS default 50
• NOOS_FAILURE_STRESS_TX_STORM_SIZE default 500
• NOOS_FAILURE_STRESS_INVALID_TX_STORM_SIZE default 300
• NOOS_FAILURE_STRESS_DUPLICATE_TX_STORM_SIZE default 300
• NOOS_FAILURE_STRESS_ACCESS_RULE_COUNT default 500
• NOOS_FAILURE_STRESS_RECORD_COUNT default 2000
• NOOS_FAILURE_STRESS_LARGE_METADATA_KEYS default 200
• NOOS_FAILURE_STRESS_LARGE_INDEX_KEYS default 100
• NOOS_FAILURE_STRESS_LARGE_PAYLOAD_BYTES default 1048576
• NOOS_FAILURE_STRESS_SMALL_BLOCK_TXS default 1
• NOOS_FAILURE_STRESS_LARGE_BLOCK_TXS default 500
• NOOS_FAILURE_STRESS_KEEP_DB default false
• NOOS_FAILURE_STRESS_EXPORT_REPORT_PATH default unset

The sync API range guard is controlled by NOOS_SYNC_MAX_BLOCK_RANGE, default 1000 with a hard maximum of 10000. Failure-stress malformed range cases should expect broad ranges to be clamped and unsafe heights to be rejected before PostgreSQL work begins.

The harness also uses the shared stability config for seed, replay/chain verification toggles, payload sizes, and schema prefix.

Intended Report

The runner should print compact machine-readable JSON and can write a full report. Each scenario reports:

• name
• ok
• expectedFailures
• unexpectedFailures
• durationMs
• details

Skipped scenarios are explicit report entries with details.skipped = true and a clear reason. They are not silently omitted.

Allowed Mutations During Failed Scenarios

Failed remote/network scenarios are expected to leave canonical chain state unchanged:

• blocks
• transactions
• block_transactions
• encrypted_records
• state_smt_roots
• state_smt_kv
• bucket_access_rules
• bucket_keys

Operational metadata may change:

• peer scores and score events
• peer sync state
• trusted peer quarantine fields
• conflict and incident records
• transaction propagation metadata
• auth challenge/session stores when auth stress is enabled

Some scenarios intentionally commit canonical failed transactions. Those are valid chain outcomes and must replay deterministically.

Intended Coverage

The intended coverage is:

• offline peer sync failure
• timeout, HTTP error, invalid JSON, malformed head/block responses
• same-height finalized conflict handling and incident cleanup
• unsupported protocol block rejection
• duplicate transaction storm
• invalid signature storm
• stale/skipped nonce and malformed envelope rejection
• tiny-block and large-block pressure
• access-rule churn, non-admin failure transactions, policy mutation, bucket-key invalid operation failures
• large encrypted payload and large public index pressure
• snapshot export/import and tamper rejection
• payload backfill failure/success batches with state-root invariance
• policy materialization checks across replication policy transitions
• auth challenge reuse, challenge expiry, bearer expiry, and production-mode dev header rejection through server injection
• manual peer ban/reset transport blocking checks
• final chain and replay verification

Explicitly skipped in that design milestone:

• true mid-batch process interruption fault injection

The remaining skipped mid-batch case is intentionally visible because current HTTP sync commits each verified remote block independently. A true "block 1 commits, block 2 fails, roll back the whole fetched batch" test needs either batch-level sync transactions or a block-boundary fault injector.

Future Additions

• overnight soak
• 10,000,000-entry size stress
• Raft-specific chaos
• CometBFT-specific chaos
• libp2p transport chaos
• crash/restart fault injection
• multi-node payload backfill storms
• long-running peer scoring and quarantine loops