Single Producer Soak Test

npm run soak:single-producer starts three local NOOSChain application instances:

• one Producer, which signs transactions and commits blocks
• Observer A, which syncs from the producer and has ciphertext access for the authorized buckets
• Observer B, which syncs from the producer but intentionally lacks some ciphertext permissions

Transaction gossip is disabled. Observers reject transaction admission and only advance through HTTP block sync. This keeps the topology simple and exercises the substrate NOOSChain depends on: deterministic execution, replay, snapshots, payload availability policy, access rules, bucket keys, and backfill.

Safety

The harness refuses NODE_ENV=production.

It creates isolated schemas with the prefix noos_soak_single_producer and only drops schemas matching that prefix. Set NOOS_SOAK_KEEP_DB=true to preserve schemas for inspection.

Logs and reports redact plaintext, encrypted payload bytes, DEKs, bucket DEKs, private keys, tokens, and secrets.

Commands

Smoke:

NOOS_SOAK_FAST_SMOKE=true npm run soak:single-producer

PowerShell:

$env:NOOS_SOAK_FAST_SMOKE="true"; npm run soak:single-producer

One hour:

NOOS_SOAK_DURATION_MS=3600000 npm run soak:single-producer

Overnight:

NOOS_SOAK_DURATION_MS=28800000 npm run soak:single-producer

What It Exercises

• per-record-key encrypted records
• per-bucket-key encrypted records
• bucket key creation and sampled bucket-key use
• replicate-all, authorized-only, and metadata-only payload policies
• deterministic failed transactions
• mempool replay protection for stale/duplicate nonces
• observer HTTP sync and idempotent catch-up
• payload backfill
• chain verification
• replay verification
• snapshot export and verification
• HTTP snapshot export disabled behavior
• operator observability endpoints protected by operator bearer auth
• cached storage/payload observability counters compared with realtime diagnostics
• bucket observability and bucket counter shape
• access-rule observability and filters
• organization/user identity observability
• low-frequency organization, user, node, bucket, access-rule, and chain-access transactions
• observer read-only enforcement for transaction submission
• incident checks

Final Report

Every run writes final-report.json inside the run directory under NOOS_SOAK_LOG_DIR, defaulting to ./soak-results/run-<timestamp>/. The same JSON is printed to stdout when the run finishes.

Top-level fields:

• kind: Always single_producer_soak. This lets automation distinguish this report from E2E, stability, or size-stress reports.
• startedAt: ISO timestamp when the runner started.
• finishedAt: ISO timestamp when the runner finished final cleanup/reporting.
• durationMs: Actual wall-clock runtime. This may be longer than NOOS_SOAK_DURATION_MS because the harness finishes the current step and final verification before exiting.
• config: The effective runtime configuration after env defaults and NOOS_SOAK_FAST_SMOKE overrides are applied.
• nodes: Final chain position for each node.
• summary: Main pass/fail counters and invariant checks.
• timings: Bounded timing histograms for operational paths.
• throughput: Average and recent-window throughput rates.
• resources: Memory, event-loop, and DB row-count samples.
• interrupted: Whether the report was produced after SIGINT/SIGTERM.
• issues: Structured unexpected problems encountered during the run.
• ok: Final boolean. This is the field CI or scripts should use for success.

config

The config object records the exact settings used for the run:

• durationMs: Target soak duration in milliseconds.
• stepIntervalMs: Delay between scenario steps.
• txsPerStep: Number of workload transactions generated per step.
• txsPerBlock: Maximum transactions per producer block.
• syncEverySteps: How often observers sync from the producer.
• verifyEverySteps: How often chain verification runs during the loop.
• replayEverySteps: How often replay verification runs. 0 disables periodic replay and only treats replay as skipped/implicitly ok for the run.
• snapshotEverySteps: How often producer snapshot export/verification runs.
• backfillEverySteps: How often observers attempt payload backfill.
• observerOfflineEverySteps: Offline simulation cadence. 0 disables it.
• observerOfflineDurationSteps: How long an observer remains offline when offline simulation is enabled.
• payloadSizeBytes: Plaintext payload size used before encryption.
• keepDb: Whether generated schemas are preserved after the run.
• exportReportPath: Optional extra path where the final report is written.
• logDir: Base directory for run logs.
• seed: Deterministic workload seed.
• fastSmoke: Whether smoke-mode defaults were applied.
• resourceSampleEverySteps: How often the runner samples DB row counts and resource details.
• metricsFlushIntervalMs: Maximum interval between compact metrics.jsonl snapshots.
• warnBlockCommitP95Ms: Warning threshold for block commit p95.
• warnMemoryMb: Warning threshold for sampled process memory.
• smtCacheMaxEntries: The effective NOOS_SMT_CACHE_MAX_ENTRIES guard for the block-local persistent SMT adapter cache. The default is 250000; 0 disables the guard for deliberate local experiments.
• observabilityEverySteps: How often operator observability endpoints are polled on all three nodes.
• counterVerifyEverySteps: How often cached storage/payload counters are compared with realtime diagnostic endpoints.
• operatorAuthEverySteps: How often valid/missing/wrong operator bearer token behavior is checked.
• observerReadOnlyEverySteps: How often observers are probed to confirm they reject transaction admission.
• governanceEverySteps: How often low-frequency identity, bucket, access-rule, and chain-access transactions are submitted by the producer admin.
• snapshotHttpCheckEverySteps: How often the harness confirms HTTP snapshot export remains disabled.

nodes

nodes contains:

• producer.finalHeight and producer.stateRoot
• observerA.finalHeight and observerA.stateRoot
• observerB.finalHeight and observerB.stateRoot

A healthy completed run should have all three nodes at the same final height and state root. Observer payload availability may differ by bucket policy, but the consensus state root must match.

summary

Counters:

• steps: Scenario loop steps completed.
• transactionsSubmitted: Transactions admitted into the producer mempool.
• transactionsExecuted: Transactions that executed successfully in committed blocks.
• deterministicFailures: Transactions included in blocks that failed deterministically, such as invalid indexes or permission failures. These are expected consensus outcomes and should replay identically.
• mempoolRejections: Transactions rejected before block inclusion, such as stale or duplicate nonce attempts.
• blocksCommitted: Blocks committed by the producer after genesis/setup.
• syncRuns: Observer sync attempts.
• syncFailures: Sync attempts that failed verification or transport checks.
• backfillRuns: Payload backfill attempts across observers.
• backfillFetched: Missing ciphertext payloads fetched successfully.
• backfillSkipped: Missing payloads skipped because policy or permissions did not allow retrieval.
• backfillFailed: Backfill attempts that failed unexpectedly.
• snapshots: Snapshot export/verification attempts.
• chainVerifications: Chain verification runs performed during the loop.
• replayVerifications: Replay verification runs performed during the loop.
• observerOfflineEvents: Offline simulation events triggered.
• unexpectedErrors: Unhandled or unexpected runner/scenario errors.
• observabilityPolls: Protected observability endpoint calls made during the run.
• observabilityFailures: Observability calls that failed or returned sensitive-looking fields.
• counterConsistencyChecks: Cached-vs-realtime counter comparison attempts.
• identityTransactions: Low-frequency organization/user/node/chain-access transactions admitted into the producer mempool.
• bucketLifecycleTransactions: Low-frequency bucket lifecycle transactions admitted into the producer mempool.
• accessRuleMutations: Low-frequency bucket access-rule add/update/remove transactions admitted into the producer mempool.
• operatorAuthChecks: Operator auth probe attempts.
• observerRejectedTransactions: Observer transaction submissions that were correctly rejected.
• snapshotHttpExportDisabledChecks: HTTP snapshot export disabled checks.
• peerScoreEvents: Reserved for future peer-scoring soak probes.

Invariant booleans:

• chainVerificationOk: Final chain verification passed.
• replayVerificationOk: Final replay verification passed, or replay was not scheduled for this run.
• headsMatch: Producer and observers ended at the same block hash/height.
• stateRootsMatch: Producer and observers ended with the same state root.
• payloadPoliciesRespected: Materialized ciphertext availability matched the configured bucket replication policies and observer permissions.
• permissionsRespected: Permission assertions passed.
• noActiveIncidents: No unresolved chain incident remained at the end.
• noUnexpectedErrors: unexpectedErrors was zero.
• observabilityOk: All required observability polling and redaction checks passed.
• cachedCountersMatchRealtime: Cached storage/payload counters matched realtime diagnostic endpoints when sampled.
• bucketCountersOk: Bucket observability returned usable summaries and bucket rows.
• identityCountersOk: Organization/user observability returned usable summaries and rows.
• accessRulesOk: Access-rule observability returned usable summaries and rule rows.
• operatorAuthOk: Missing/wrong operator tokens were rejected and the valid token was accepted.
• observerReadOnlyOk: Observers rejected transaction admission attempts.
• mempoolObservabilityOk: Mempool observability returned pending counts, type summaries, and recent hashes without payload exposure.
• snapshotHttpExportDisabledOk: /snapshots/export over HTTP returned 410, preserving the local CLI/export-only boundary for large snapshots.

ok is true only when all required invariant booleans are true and issues is empty. Zero-valued counters, such as backfillFetched: 0, are not failures by themselves.

timings

Each timing entry has this shape:

{
  "count": 123,
  "min": 1.2,
  "max": 450.8,
  "avg": 42.1,
  "p50": 30.4,
  "p95": 120.7,
  "p99": 240.3
}

Percentiles are calculated from a capped rolling sample window. This keeps the harness memory-safe for overnight runs while still showing useful tail latency. p50 is the normal case; p95 and p99 reveal spikes and are usually more important for operational tuning.

Tracked timings:

• txBuildMs: workload payload creation plus canonical signing.
• mempoolAdmissionMs: local producer mempool validation and insertion.
• blockBuildMs: block assembly from pending transactions.
• blockExecutionMs: deterministic block execution and state updates.
• blockCommitTotalMs: total block lifecycle time around build and execution.
• syncRunMs: full observer sync cycle.
• syncBatchMs: individual observer sync from producer.
• replayVerifyMs: replay verification runtime.
• chainVerifyMs: chain verification runtime.
• snapshotExportMs: consensus snapshot export runtime.
• snapshotVerifyMs: snapshot verification runtime.
• snapshotImportMs: reserved for future import spot checks.
• backfillRunMs: full observer payload backfill cycle.
• backfillFetchMs: individual observer backfill call.
• stepDurationMs: end-to-end scenario step runtime.
• encryptPayloadMs: encryption time for generated encrypted records.
• decryptPayloadMs: reserved for sampled decrypt checks.
• executionLoadBlockMs: block protocol/root metadata lookup.
• executionLoadTransactionsMs: block transaction row load.
• executionLoadPreviousRootMs: previous SMT root lookup.
• executionSmtStoreInitMs: persistent SMT store construction.
• executionLoadChangePositionMs: existing state-change position lookup.
• executionSavepointMs: PostgreSQL savepoint/release/rollback overhead.
• executionAuthMs: canonical tx hash/signature validation and nonce consumption.
• executionActorResolutionMs: signer-to-actor lookup.
• executionHandlerMs: protocol domain handler execution.
• executionStateApplyMs: block state-change audit writes, state_leaves updates, and SMT leaf updates.
• executionSmtFlushMs: batched persistent SMT node flush into state_smt_kv. This flush runs inside the block database transaction, so a crash or rollback cannot leave a partially committed block root.
• executionSmtCacheEntries: SMT adapter cache entries after flush.
• executionSmtPendingEntries: pending buffered SMT entries after flush. This should normally be 0.
• executionSmtCacheDisabledForBlock: 1 for blocks where the cache guard was reached and new cache entries were bypassed for the rest of that block.
• executionSmtMaxCacheEntries: maximum in-memory SMT cache entries observed during a block execution.
• executionSmtMaxPendingEntries: maximum buffered SMT writes observed during a block execution.
• executionSmtCacheHits: SMT node reads served from the block-local cache.
• executionSmtCacheMisses: SMT node reads that had to touch PostgreSQL.
• executionSmtCacheBypassWrites: attempted cache writes skipped because the cache guard was reached for that block.
• executionSmtPendingHits: SMT reads served from the buffered write set.
• executionSmtWrites: SMT node/root writes issued by the iden3 tree.
• executionSmtFlushWrites: rows written during the batched state_smt_kv flush.
• executionSmtPendingEntriesBeforeFlush: pending buffered writes immediately before flush.
• executionSmtCacheEntriesBeforeFlush: block-local cache entries immediately before flush.
• executionSmtCacheLimitEntries: configured NOOS_SMT_CACHE_MAX_ENTRIES value observed by block execution.
• executionTxStatusUpdateMs: transaction status/failure update.
• executionPersistStateRootMs: state root persistence to block/root tables.
• executionLoadFinalBlockMs: final block row reload before hash finalization.
• executionFinalizeBlockHashMs: final block hash update after state root.

The SMT cache is process memory, but it is intentionally scoped to one block execution. A new adapter is created for each block and becomes eligible for garbage collection after that block finishes. The guard NOOS_SMT_CACHE_MAX_ENTRIES defaults to 250000; 0 disables the guard. If a block reaches the guard, the adapter logs a warning and stops adding new cache entries for the rest of that block. Pending SMT writes remain buffered so the tree can still read its own writes before the transactional flush. The next block gets a fresh cache and normal caching resumes. Real execution errors still occur inside the outer PostgreSQL transaction, so partial block state rolls back together.

throughput

throughput contains:

• txPerSecondAvg: submitted transactions divided by total runtime.
• txPerSecondRecentP95: p95 of recent per-sample transaction throughput.
• blocksPerSecondAvg: committed blocks divided by total runtime.
• blocksPerSecondRecentP95: p95 of recent per-sample block throughput.
• recordsPerSecondAvg: successfully executed transactions divided by runtime.
• syncRunsPerMinute: observer sync attempts per minute.

Average throughput includes verification, sync, snapshot, and cleanup overhead. Recent p95 values are better for spotting active-run bursts.

resources

resources contains process and database samples:

• memoryStartMb: Runner RSS near startup.
• memoryMaxMb: Maximum sampled RSS during the scenario loop.
• memoryEndMb: Runner RSS at report generation.
• rssMaxMb: Maximum resident set size.
• heapMaxMb: Maximum V8 heap used.
• externalMaxMb: Maximum external memory reported by Node.
• arrayBuffersMaxMb: Maximum ArrayBuffer memory reported by Node.
• eventLoopLagP95Ms: p95 observed event-loop delay.
• dbRowCounts: Latest sampled producer row counts for blocks, transactions, encrypted_records, state_smt_kv, and snapshots.

Use these values to spot obvious memory growth during long runs. They are process-level samples, not detailed heap profiles.

Warnings

The harness adds warning issues without automatically failing the run:

• WARN_BLOCK_COMMIT_P95 if blockCommitTotalMs.p95 exceeds NOOS_SOAK_WARN_BLOCK_COMMIT_P95_MS.
• WARN_MEMORY_MB if sampled memory exceeds NOOS_SOAK_WARN_MEMORY_MB.
• WARN_SMT_CACHE_NEAR_LIMIT if executionSmtMaxCacheEntries.max exceeds 80% of NOOS_SMT_CACHE_MAX_ENTRIES.
• WARN_SMT_CACHE_DISABLED_FOR_BLOCK if at least one block reached the SMT cache guard and continued with new cache entries disabled for the rest of that block.

Warnings are intentionally visible in issues so overnight runs do not hide performance drift, but ok is still governed by correctness invariants unless an unexpected error occurs.

issues

issues is an array of structured problems:

{
  "step": 123,
  "code": "SNAPSHOT_VERIFY_FAILED",
  "message": "..."
}

step may be omitted for startup, setup, cleanup, or fatal runner failures. Warning entries with WARN_* codes are informational and do not make ok false by themselves. Non-warning issues make ok false.

Related Log Files

The final report should be read together with:

• events.jsonl: step, block, sync, backfill, snapshot, chain verify, replay verify, lifecycle events, and action_result entries.
• metrics.jsonl: compact periodic snapshots containing height, state root, submitted/executed transactions, block commit p95, sync p95, throughput, memory, and producer DB row counts.
• errors.jsonl: unexpected errors with redacted stack/details.
• soak.log: human-readable JSONL event log.

action_result entries are the operator-readable trail for what the harness is doing. Each one records:

• action: the exact scenario action that just ran, such as observer sync, snapshot verification, operator-auth probing, or final convergence checks.
• expected: the desired invariant or behavior for that action.
• actual: the observed result, counters, heights, roots, or booleans.
• ok: whether the actual result matched the expectation.

These entries are intentionally higher-level than raw HTTP/debug logs. They answer: what did the soak try, what was supposed to happen, and what actually happened.

The JSONL files are suitable for later graphing. A future milestone can expose the same measurements through Prometheus or OpenTelemetry.

What It Does Not Cover

• Raft leader election
• CometBFT voting or quorum certificates
• libp2p transport
• Byzantine validators
• automatic fork choice or reorg