Node Observability API

NOOSChain exposes local operator observability under /node/observability. These routes are intended for a future management GUI and local node operators. They are not public chain APIs and they do not participate in consensus.

In development, the HTTP API can also expose Swagger/OpenAPI documentation for these routes and the rest of the API surface. See Swagger / OpenAPI for NOOS_SWAGGER_ENABLED, /docs, /openapi.json, and production refusal rules.

Authentication

Observability uses a separate operator guard. Normal user bearer tokens do not grant observability access.

Production deployments must set:

NOOS_OPERATOR_TOKEN=<long-random-operator-token>

Requests must include:

Authorization: Bearer <NOOS_OPERATOR_TOKEN>

If NOOS_OPERATOR_TOKEN is missing in production, observability requests fail with OBSERVABILITY_AUTH_NOT_CONFIGURED. In non-production only, a development fallback token dev-operator-token is accepted for local verification and GUI prototyping.

Routes

• GET /node/observability/overview: node identity, runtime role, uptime, chain head, protocol version, genesis hash, high-level health, and consensus sidecar summary.
• GET /node/observability/consensus: detailed consensus, Raft, sidecar, and governance/runtime membership reconciliation summary.
• GET /node/observability/health: DB connectivity, chain initialization, consensus status, sync manager status, verification status, peer score summary, memory usage, and process uptime.
• GET /node/observability/metrics: JSON counters for blocks, transactions, mempool, failed transactions, encrypted records, buckets, peers, selected row counts, process memory, and consensus summary.
• GET /node/observability/chain: chain head, genesis metadata, protocol upgrades, cached verification status, checkpoints, and snapshots.
• GET /node/observability/mempool: pending count, ages, counts by type, and recent pending transaction hashes. includePayload=true is for operator debugging only and still passes through redaction.
• GET /node/observability/peers: trusted peers, sync state, score status, quarantine/ban state, and last sync errors.
• GET /node/observability/sync: sync manager state, local head, per-peer sync information, current/last phase, last peer/result, block fetched/applied counters, payload backfill attempted/fetched/skipped/failed counters, payload backfill reason counts, HTTP/DB backfill batch counts, peer failover count, last error, and last successful catch-up time.
• GET /node/observability/incidents: active incident, recent incidents, open conflicts, and quarantined peers.
• GET /node/observability/storage: cached selected table row counts, snapshot count, latest snapshot metadata, missing payload count, and available payload count. The counters are maintained by database triggers when blocks, transactions, encrypted records, local encrypted payload rows, SMT rows, snapshots, peers, or peer score events change.
• GET /node/observability/storage/realtime: diagnostic live-scan version of the storage endpoint. It may be expensive on large databases and is intended for operator truth checks, not GUI polling.
• GET /node/observability/payloads: cached encrypted record availability totals and counts by bucket replication policy. It never returns ciphertext bytes.
• GET /node/observability/payloads/realtime: diagnostic live-scan version of the payload availability endpoint. It may be expensive on large databases.
• GET /node/observability/contracts: aggregate smart-contract execution metrics by contract id, including calls, failures, fuel used, host calls, emitted events, storage writes, encrypted-record metadata writes, and recent failure codes.
• GET /node/observability/contracts/{contractId}: aggregate metrics and recent call metrics for one contract.
• GET /node/observability/contracts/{contractId}/calls: bounded per-call contract metrics. Optional filters are method, status, failureCode, fromBlock, toBlock, limit, offset, and order. See Smart Contract Operations Runbooks for incident triage using these metrics.
• GET /operator/payload-store/stats: operator-only local payload-store row, byte, age, and policy grouping diagnostics. It does not return ciphertext bytes.
• GET /operator/payload-store/stats/realtime: operator-only exact payload-store row, byte, age, and policy grouping scan. It can be expensive on large nodes and is intended for diagnostics/counter truth checks rather than polling dashboards.
• POST /operator/payload-store/gc: operator-only dry-run or confirmed local ciphertext retention pass filtered by age, bucket, replication policy, and limit. It prunes local availability-layer bytes only.
• POST /operator/payload-store/repair-counters: operator-only repair action that rebuilds cached payload availability counters from realtime database contents.
• GET /node/observability/buckets: bucket metadata plus cached per-bucket counters. recordTransactionCount means distinct encrypted-record transaction hashes in encrypted_records for that bucket. Access-rule, bucket-key, active key-version, index-schema, record, available-payload, and missing-payload counts are reported separately. This route does not return encrypted payload bytes, key envelopes, DEKs, private keys, or secrets.
• GET /node/observability/bucket-access-rules: bounded read-only list of explicit bucket_access_rules rows with joined bucket and principal display metadata. Optional filters are bucketId, userId, organizationId, limit, and offset; userId and organizationId are mutually exclusive. The endpoint returns normalized permission strings and never returns raw public keys, payload bytes, key material, or secrets.
• GET /node/observability/organizations: organization identity metadata plus cached counters for users, nodes, validators, buckets, records, payload availability, bucket keys, access rules, and signed transactions. Raw public keys are not returned; the endpoint exposes only key presence and SHA-256 public-key fingerprints.
• GET /node/observability/users: user identity metadata plus cached counters for buckets, records, payload availability, bucket keys, access rules, signed transactions, and pending mempool transactions. Use ?organizationId=<org-id> to restrict the response to one organization. Raw public keys are not returned.

/node/observability/chain?runVerify=true runs chain verification. GUI polling should not use it at high frequency. Replay verification is intentionally not run by this endpoint because it can be expensive on long chains.

Redaction

All observability responses pass through a defensive redactor. Values are redacted for keys matching private keys, tokens, secrets, passwords, DEKs, auth challenges, and encrypted payload fields.

The API must not expose plaintext payloads, private keys, DEKs, bucket DEKs, bearer token secrets, raw auth challenges, operator tokens, or encrypted payload bytes.

For RAFT_ENGINE=hashicorp-go, observability includes the HashiCorp Raft sidecar process state: running/dead status, PID, uptime, restart count, last exit code/signal, last error, next restart time, Raft role, leader, term, log indexes, commit index, and applied index. These are operator fields only and never include private keys or callback signing material.

Consensus observability also includes validator governance fields:

• active, suspended, and retired validator counts
• the current node's governance validator status
• active validator set hash
• active governance validator node IDs
• runtime Raft voter node IDs
• governance/runtime validator set hashes
• missing runtime validators, unexpected runtime voters, and endpoint mismatches
• runtime Raft membership voters/non-voters, leader id, last membership change, and whether runtime membership currently matches governance
• a warning when runtime voters differ from governance active validators

The warning is expected after governance membership changes until an operator executes an approved runtime membership plan or updates deployment configuration. Governance transactions do not silently hot-reconfigure Raft voters.

State-root observability is reported under consensus.stateRootEngine:

• current, source, appliesAtNextHeight, activeActivation, and nextActivation describe the selected state-root engine and any governed activation boundary.
• readiness contains the activation-readiness checks used by operators and fail-closed production gates.
• supervisor describes the continuous Nervos sidecar lifecycle service when enabled: policy, whether it should run, whether it is required now, health interval, last health check, last healthy timestamp, last error, version, health, and child-process status.
• sidecar describes the latest observed persistent sidecar session. This may be a transient execution/checkpoint session or the child owned by the continuous supervisor.
• checkpoint describes the configured local checkpoint directory and latest verified Nervos checkpoint metadata: height, block hash, state root, checkpoint hash, engine version, sidecar protocol version, checkpoint format version, and creation time. It does not expose checkpoint file contents.

The supervisor is the lifecycle and execution process boundary for Nervos state-root work when it is active. It does not expose raw state, private keys, checkpoint contents, or payload data.

Continuous Nervos execution also records local checkpoint metadata in state_root_sidecar_checkpoints. Those rows are intentionally not returned as checkpoint file contents through observability; operators should inspect them through local database/admin tooling when debugging restart behavior. The checkpoint files themselves stay on disk under NOOS_STATE_ROOT_CHECKPOINT_DIR.

GUI Guidance

GUI polling should prefer /overview, /health, /metrics, and /sync. Manual "verify now" actions may call /chain?runVerify=true. For storage and payload availability, GUI polling should use the cached /storage and /payloads endpoints. The /storage/realtime and /payloads/realtime variants are for explicit troubleshooting because they run live database scans. Bucket observability uses cached per-bucket counters maintained by database triggers so the GUI can list buckets without running per-bucket count(*) scans. Bucket metadata itself still comes from the consensus buckets table. Organization and user observability uses the same cached-counter model. Public key fingerprints are identifiers for operators only and should not be treated as authorization secrets.

The high-volume encrypted-record counter path is maintained with statement-level trigger delta updates: inserts, deletes, payload-availability changes, bucket moves, and record transaction-hash changes adjust only the affected policy, bucket, organization, and user counters. Exact refresh functions remain in the database for repair, migration backfill, and diagnostics. The realtime endpoints are the operator truth-check when cached counters are suspected to be stale. Local ciphertext availability is counted from encrypted_record_payloads first with legacy encrypted_records availability fields as a fallback, so cached and realtime observability both match the current availability-store model.

Consensus status includes Raft details when NOOS_CONSENSUS_MODE=raft: current role, current term, voted-for node, known leader id, commit index, last applied index, last log index, last log term, transport mode, configured peer endpoints, peer connectivity status, snapshot enabled status, snapshot baseline index/term, installed snapshot count, and whether the adapter is using Noosraft or external HashiCorp Raft sidecar engine. For the HashiCorp Raft sidecar, consensus status also includes external process health fields such as process status, PID, uptime, restart count, last exit details, last error, and next restart time. These fields are operational observability only; they do not expose private keys, transaction plaintext, DEKs, or payload bytes.

Future Work

• Prometheus-compatible metrics endpoint.
• OpenTelemetry tracing.
• Structured log streaming.
• Operator audit logs.
• Role-based operator permissions.
• Multi-node fleet observability aggregation.

TLS Observability

Overview, health, and metrics include TLS/mTLS status: whether TLS is enabled, whether mTLS is required, whether a CA/certificate is configured, and the configured server name. Certificate contents and private keys are never returned.

Operator CLI

The operator CLI wraps these observability APIs through commands such as npm run noos -- node health and npm run noos -- node metrics. It uses Authorization: Bearer <NOOS_OPERATOR_TOKEN> and supports the same HTTPS/mTLS certificate material through CLI flags or NOOS_CLI_TLS_* environment variables. See Operator CLI.