HTTP Node Sync

NOOSChain node sync is a simple HTTP pull model for trusted private deployments. It is not peer discovery and it is not consensus. Peers are manually configured in trusted_peers.

The receiver fetches:

• remote chain head
• ordered blocks
• ordered transactions inside each block

Block Range Performance And Limits

HTTP catch-up uses GET /chain/blocks?from=&to=&limit=. The serving node loads each block range with batched database queries: headers are fetched once, then all block transactions for the returned heights are fetched in one ordered join. The block_transactions(block_height, position) index supports this range-and-order access pattern.

NOOS_SYNC_MAX_BLOCK_RANGE controls the maximum number of blocks served by /chain/blocks and /chain/headers in one response. The default is 1000; the code refuses values above 10000. The public endpoints reject malformed, negative, and unsafe huge heights before querying PostgreSQL. Broad ranges are clamped by the configured limit, so a request with a very large to height cannot make the node scan an unbounded range.

Fastify responses larger than 1KB are compressed with Brotli or gzip when the client advertises Accept-Encoding: br or gzip. This is especially useful for block ranges with many canonical transaction envelopes.

The receiver does not copy remote domain tables, SMT tables, state leaves, or materialized state. Every synced block is inserted as source='synced', then executed locally through the same protocol-versioned block executor used for local blocks. The node recomputes Merkle roots, block hashes, and SMT state roots and compares them with the remote block commitment.

Each synced transaction includes its canonical signed consensus envelope when available. Fresh peers must persist and transmit this envelope so receivers can recompute transaction hashes and verify signatures without relying on database insertion timestamps. For ADD_ENCRYPTED_RECORD, the consensus envelope intentionally omits encryptedPayload; ciphertext is availability-layer data and is fetched separately through the permission-gated payload endpoint. See Canonical Transaction Envelope.

Canonical consensus payloads are stored exactly as received so transaction hashes and block hashes remain verifiable. Ciphertext is never required for Merkle, block-hash, state-root, or replay verification. During execution, a local payload-availability policy may choose not to materialize ciphertext at all. New local ciphertext is stored in encrypted_record_payloads; the older encrypted_records.encrypted_payload and payload_available_locally fields remain compatibility fallbacks. This means two nodes can have identical consensus state roots while one has ciphertext in its local payload store and another has only consensus metadata.

replicate_encrypted_to_authorized_nodes is enforced as data placement, not as metadata visibility. Nodes can still sync block and record metadata for a bucket whose payloads they are not allowed to store. To receive or serve encrypted payload bytes for that policy, a node must present an organization-signed node membership credential and prove possession of the node private key. A claimed organizationId in local config or a public-key header is not enough. See Node Membership Credentials.

Trust Model

Trusted peers are manually added by API or database configuration. trust_mode is currently metadata (trusted or observer) and no automatic peer discovery exists yet.

Observer nodes are first-class sync targets. They reject transaction admission and gossip, but they can sync finalized blocks, replay them deterministically, serve permitted reads, and perform availability-layer payload backfill. This is the intended role for follower nodes in the size-stress design; the stability:size runner itself is not present in the current workspace.

In NOOS_CONSENSUS_MODE=raft, voting nodes commit through the Raft adapter rather than HTTP sync. Observers still do not vote; they remain HTTP sync clients that pull committed blocks from a producer/leader. This keeps consensus ordering separate from observer catch-up and preserves the existing payload replication policy behavior for non-voting nodes.

Synced blocks are never trusted blindly. If local execution cannot reproduce the remote state root and block hash, the current block transaction rolls back and sync stops with an error.

Periodic verification is also local. With CHAIN_VERIFICATION_ENABLED=true, the node verifies its own stored chain incrementally in chunks, with optional full sweeps. A peer may provide blocks, but it never provides trusted proof that this node's database is valid.

Peer scoring is a separate local transport defense. Sync success and failures update peer_scores; repeated malformed responses can quarantine a peer, and manual bans disable local transport. This does not change validator membership, voting power, finalized blocks, or protocol state.

The periodic sync manager combines trusted-peer configuration with local peer scores before each run:

• only active trusted peers are considered;
• peers whose score status is quarantined or banned are skipped for normal sync/backfill transport;
• remaining peers are ordered by local scoring status and score, so healthy higher-scoring peers are tried before degraded lower-scoring peers;
• the manager still walks every eligible peer in the ordered set, because block sync and payload availability are separate responsibilities. A peer may be valid for committed block catch-up while lacking ciphertext for some buckets, so a later eligible peer can still recover missing payload bytes.

This ordering is local and operational. It is not peer discovery, consensus membership, or a validator-set decision.

Malformed peer responses are rejected at the HTTP boundary before canonical chain state can be trusted. Remote heads and block ranges are shape-checked and then verified for continuity, transaction ordering, transaction hashes, Merkle roots, protocol support, required state roots, and block hash consistency. A failed sync may update local peer_sync_state.last_error, but blocks, transactions, SMT storage, and domain state must remain unchanged. See Malformed Peer Hardening for the dedicated malformed-peer verifier.

Fork Handling

Fork handling now goes through the swappable fork/conflict policy layer. If the local node already has a finalized block at the same height with a different hash, sync records a same_height_different_hash conflict, quarantines the peer, enters incident mode, and does not mutate local chain state. There is still no automatic reorg or longest-chain rule because NOOSChain currently assumes deterministic finality in a closed trusted validator topology.

Incident mode protects the audit trail. While active, /sync/run refuses unless explicitly forced, automatic sync loops do not run, and /consensus/propose-block refuses to commit new blocks. Read APIs remain available for investigation. See Fork Conflict Handling.

/sync/run and /sync/status are protected by the node-operator bearer-token guard used by observability routes. Send Authorization: Bearer <NOOS_OPERATOR_TOKEN>. Outside production, when NOOS_OPERATOR_TOKEN is not set, the built-in development token is dev-operator-token. Chain verification endpoints under /chain/verification/* use normal protected API actor authentication.

/node/observability/sync exposes operator-facing recovery telemetry for the same manager. It reports whether sync is currently running, the current phase and peer, the last peer/result, blocks fetched and applied, payload backfill attempted/fetched/skipped/failed counters, peer failover count, last error, and the last time catch-up applied new blocks successfully. These fields are local runtime observability only; they do not change consensus state and they reset when the process restarts.

When payload backfill runs, the same observability payload includes stable reason counts and batch counts. HTTP payload batches remain capped at 100 record ids by /records/payloads, while local DB update batches can be larger because they write only already verified ciphertext to the narrow encrypted_record_payloads availability table. The default local DB update batch is 500 verified payload rows; tune it with benchmark:payload-backfill before changing production settings.

Future Raft or CometBFT adapters will own ordering/finality. This HTTP sync layer is a catch-up and replication primitive, not a consensus protocol.

Snapshot Bootstrap

A node may bootstrap from a verified snapshot checkpoint instead of replaying from genesis. In that case /chain/head can report the checkpoint as the local head before any later blocks are synced. Sync then requests blocks after the checkpoint height, and the first synced block must have previous_hash equal to the checkpoint block hash.

Snapshot sync still does not trust remote materialized state. The snapshot is verified locally, SMT storage is reconstructed locally, and all post-checkpoint blocks execute through the normal protocol-versioned pipeline.

Determinism

Sync uses historical block.protocol_version, so old blocks execute under their declared protocol semantics. Encrypted payload bytes remain availability-layer data; deterministic replay and sync roots are anchored by payload_hash and metadata, not by decrypted plaintext.

Bucket replication policies affect local ciphertext transfer/storage and, for per_bucket_key buckets, organization bucket-key sharing for node replication:

• replicate_encrypted_to_all_nodes: synced nodes store ciphertext locally
• replicate_encrypted_to_authorized_nodes: synced nodes store ciphertext only when the verified local node organization has bucket:read_encrypted or bucket:admin
• metadata_only: synced nodes store metadata/state with no ciphertext and no organization bucket-key sharing for replication

Access-rule updates and removals affect future sync decisions as soon as the corresponding block executes locally. If bucket:read_encrypted is removed for a node organization, future authorized-only records sync without ciphertext and explicit payload backfill skips them. If permission is later restored, newly synced records may include ciphertext again and older missing ciphertext can be fetched by explicit backfill.

Payload backfill is now available as a separate availability-layer operation. It may fill encrypted_record_payloads for records already present locally, but it does not alter blocks, transactions, state roots, or replay results. Unauthorized nodes still receive ordered block data and consensus record metadata, but not ciphertext bytes. Backfill uses a batched POST /records/payloads endpoint when available and falls back to single-record GET /records/:recordId/payload for older or mock peers. The batch endpoint accepts up to 100 record ids per request; direct oversized calls return 400 INVALID_PAYLOAD_BATCH, and the built-in client chunks larger backfill sets automatically before sending them. The dev/operator /payload-backfill/run and /records/missing-payloads routes are protected by the node-operator bearer-token guard and remain disabled in production mode.

Removing permission does not automatically prune ciphertext that is already physically stored. Current read APIs still enforce current permissions, so stored ciphertext is not returned to actors that no longer have bucket:read_encrypted. Operators can explicitly prune local availability storage with noos node reconcile-payload-availability --prune-disallowed --yes when the goal is to remove ciphertext that current policy/permissions no longer allow, or with noos node payload-store-gc --yes for broader age/bucket/policy retention cleanup. These commands affect local ciphertext availability only and do not change consensus state roots.

For per_bucket_key buckets, bucket-key creation and rotation sync as ordinary consensus transactions. A synced node reconstructs bucket_keys locally through replay; it never copies a peer's materialized key table. Records remain pinned to the bucket key id/version committed in their transaction. Bucket-key envelope recipients are validated against bucket replication policy and access rules during creation/rotation. Replication fetches may return the bucket-key envelope metadata needed by an authorized node for the record version it is allowed to store, but they must not provide ciphertext or bucket-key material to a node whose organization is outside the current replication boundary.

Bucket replication policy can change through UPDATE_BUCKET_POLICY. Sync evaluates the current local bucket policy when materializing newly synced encrypted records. Policy changes are future-facing: switching to metadata_only does not delete ciphertext already stored locally, and switching away from metadata_only does not automatically backfill older missing ciphertext.

Synced nodes re-execute transactions with the historical protocol version. They verify signatures, resolve signer identities, consume logical nonces, and reproduce key-rotation and permission outcomes locally instead of trusting remote materialized state.

Transaction gossip is separate from chain sync. Gossip shares pending transactions opportunistically; sync verifies committed block history and remains the source of truth for catch-up.

Future work can replace HTTP with libp2p transports, add light-client proofs, and integrate sync with Raft/CometBFT finality.

TLS/mTLS

Node sync clients support HTTPS peer URLs with NOOS_TLS_* settings. mTLS can require a client certificate for protected node-to-node traffic. Signed node headers remain required where already enforced; TLS/mTLS is an additional transport layer, not a replacement for node identity signatures.