Payload Backfill

Payload backfill is availability-layer synchronization. It lets a node that already has deterministic consensus state for an encrypted record fetch ciphertext later after gaining permission.

Backfill never creates blocks, never changes transactions, never updates SMT storage, and never changes state roots. It only updates local availability-layer ciphertext storage.

Current storage is split deliberately:

• encrypted_records is the consensus record metadata table. Its encrypted_payload and payload_available_locally columns are legacy/local-availability fallback fields and are not part of consensus state hashing.
• encrypted_record_payloads is the preferred local ciphertext store. Backfill writes verified ciphertext here so large payload restores do not rewrite high-traffic consensus metadata rows.

Read APIs and backfill helpers check encrypted_record_payloads first, then fall back to the legacy encrypted_records columns for older data and imported snapshots.

Retrieval

Source nodes expose GET /records/:recordId/payload for single-record retrieval and POST /records/payloads for batched retrieval. The batch route accepts up to 100 record ids per request and returns the same response shape per record. Oversized or malformed batch requests fail with 400 INVALID_PAYLOAD_BATCH rather than a server error. Direct API callers must honor this cap; the built-in backfill client handles chunking automatically. The route resolves the caller through the API actor layer, requires bucket:read_encrypted unless the caller is an authorized node whose local replication policy permits ciphertext, and returns ciphertext only if the source node has it locally. For per_bucket_key records, an authorized node replication response may also include the bucket-key envelope metadata for the record's committed key version, because bucket-key sharing follows the same replication policy as ciphertext placement. The route never returns private keys, never returns raw DEKs, and never decrypts.

The backfill client prefers the batch endpoint to avoid one HTTP request per record during sync catch-up. When more than 100 records need ciphertext, the client automatically splits work into multiple 100-record batches before making HTTP requests, so large backfill runs do not trigger the server's batch guard. If a peer does not support batching, the client falls back to the single-record endpoint.

HTTP fetch batching is separate from local database update batching. The peer request batch is capped at 100 by the API contract and can be tuned with PAYLOAD_BACKFILL_HTTP_BATCH_SIZE; the local update batch defaults to 500 verified payload rows and can be tuned with PAYLOAD_BACKFILL_DB_UPDATE_BATCH_SIZE. Backfill reports both batch counts in its result so operators can see whether time is going into peer requests or local availability-layer writes.

The 500-row default is benchmark-derived, not a consensus rule. On the current PostgreSQL path, larger update batches reduced statement count but increased availability-update time because each statement carries trigger and tuple-update work. Operators should rerun benchmark:payload-backfill on their storage profile before changing this value.

Backfill results also include reasonCounts. Stable reasons include:

• fetched: ciphertext was fetched, hash-verified, and stored locally.
• metadata_only: the current bucket policy says this node should not store ciphertext.
• not_authorized: the bucket is authorized-only and this node's organization does not currently have encrypted-read/admin permission.
• node_membership_unverified: the bucket is authorized-only and this node did not present a valid organization-signed node membership proof.
• node_membership_mismatch: the verified node membership did not match the local node id or organization.
• no_local_node_public_key: the node identity needed for encrypted payload authorization was missing locally.
• remote_unavailable: the peer could not provide ciphertext for a fetchable record.
• hash_mismatch: the peer returned ciphertext metadata that did not match the local consensus payload_hash.
• peer_error: the peer request failed or returned malformed payload data.
• local_update_failed: the verified payload could not be written to local availability storage.

The local dev/operator routes POST /payload-backfill/run and GET /records/missing-payloads are disabled in production mode. They are also protected by the node-operator bearer-token guard used by observability routes: send Authorization: Bearer <NOOS_OPERATOR_TOKEN>. Outside production, when NOOS_OPERATOR_TOKEN is not set, the built-in development token is dev-operator-token.

Validation

Backfill compares the remote payloadHash with the local consensus payload_hash before storing ciphertext. This does not prove plaintext by itself, but it ensures the remote response is tied to the same consensus record metadata. Future work should add signed payload availability receipts and multi-peer verification.

Malformed payload responses fail closed. If a peer returns invalid JSON, available=true without ciphertext, the wrong recordId, an unexpected payloadHash, HTTP 401/403/500, or times out, backfill records a local failure or skip and does not update consensus state. The only successful mutation is the availability-layer write of the exact expected record/hash into encrypted_record_payloads or, for legacy rows, the equivalent local ciphertext fallback fields.

Late Authorization

For replicate_encrypted_to_authorized_nodes, a node may sync record metadata before it has ciphertext permission. If its organization later receives bucket:read_encrypted, newly synced records may arrive with ciphertext immediately, while old records can be fetched by explicit backfill.

The same boundary applies to per_bucket_key material used by replication. A node organization that cannot store the ciphertext under the current bucket replication strategy must not receive bucket-key envelopes through replication fetches. When access is removed, future backfill skips that bucket and future bucket-key rotations reject envelopes for the removed organization. Existing historical key versions can remain decryptable by old envelope recipients until the bucket is rotated and future records use the new version.

If bucket:read_encrypted is removed, backfill skips that bucket for the node even if ciphertext is available from a peer. Restoring permission later enables future backfill again. This is intentionally availability-layer behavior: permission removal does not alter canonical transactions, blocks, SMT roots, or past record state.

metadata_only buckets do not backfill ciphertext. Their policy says local ciphertext storage should remain absent even if a peer has bytes.

Backfill uses the current bucket replication policy, not the policy that existed when a record was created. If a bucket is changed to metadata_only, future backfill skips its missing ciphertext. If a bucket is changed from metadata_only to an authorized or all-node replication policy, previously missing ciphertext can be fetched only by an explicit backfill run and only when permissions also allow it.

Backfill itself does not prune ciphertext already stored locally. API reads still enforce current permissions and return encryptedPayload: null when the actor lacks bucket:read_encrypted.

For operator maintenance, POST /operator/payload-availability/reconcile and the CLI wrapper noos node reconcile-payload-availability can compare local ciphertext availability against the node's current bucket replication policy and permissions. Dry-run mode reports:

• ciphertext present locally but no longer allowed for this node
• ciphertext missing locally but allowed for this node

When explicitly run with --prune-disallowed --yes, the CLI clears only local availability storage for disallowed rows:

• encrypted_record_payloads
• encrypted_records.encrypted_payload
• encrypted_records.payload_available_locally

This is still availability-layer maintenance. It does not create transactions, does not alter blocks, does not update SMT storage, and does not change state roots. It is useful after an operator demotes a validator to an observer or after bucket permissions/policies change and the operator wants local storage to reflect the current retention rules.

For broader storage-pressure cleanup, POST /operator/payload-store/gc and the CLI wrapper noos node payload-store-gc can select local ciphertext by bucket, replication policy, age, and limit. Dry-run is the default; non-dry-run requires explicit operator confirmation in the CLI. This command is a retention tool, not a permission resolver: it prunes local ciphertext selected by filters even if current policy would still allow the node to retain it.

GET /operator/payload-store/stats reports local payload-row count, approximate ciphertext bytes, oldest/newest stored timestamps, and grouping by bucket replication policy without returning ciphertext. If cached payload availability counters are suspected stale after manual database repair, POST /operator/payload-store/repair-counters rebuilds the payload counter tables from realtime database contents.

Peer Scoring

Backfill records payload_backfill_success and payload_backfill_failure events for the source peer. Quarantined or banned peers are skipped by default; operator tooling may explicitly force a one-off attempt for quarantine cases, but a banned peer should be unbanned first.

When the sync manager runs automatic block sync with payload backfill enabled, it orders eligible trusted peers by local peer score and still walks the full eligible peer list. This is intentional: a valid block source may not have ciphertext for every record, especially with authorized-only or metadata-only bucket policies. A later eligible peer can therefore fill missing ciphertext without changing consensus state or replay results.

Future Work

TODO:

• payload source verification
• multi-peer payload retrieval
• encrypted payload deduplication
• object storage backend for encrypted_record_payloads
• resumable backfill
• automated retention policy scheduling for payload-store GC
• signed payload availability receipts