Monitoring And Alerting

Production monitoring answers a narrow operational question: what should page an operator right now?

It is not the same as production admission, incident repair, or raw observability. Use:

Need	Page
Admit a node before launch, rejoin, or upgrade	Production Readiness
Inspect raw node observability routes	Observability API
Diagnose and repair an incident	Incident Response
Handle finalized conflicts or divergence	Conflict Handling
Operate contract runtime and contract alerts	Smart Contract Runtime Operations

Monitoring Model

Monitoring turns observability into operator action.

Signal source	What it tells you
production monitor-report	Current node-level critical, warning, and info alerts.
/node/observability/*	Raw local node health, consensus, sync, payload, storage, incident, and runtime state.
Contract monitoring endpoints	Contract runtime, readiness, call failures, and Prometheus-style metrics.
External host monitoring	CPU, memory, disk, PostgreSQL, network, process supervisor, certificates, and load balancers.

Alerts are symptoms. Incidents are tracked operational facts. A critical alert should usually open or update an incident, but alerts do not automatically fix the node or mark incidents resolved.

CLI Alert Report

Use:

npm run noos -- production monitor-report --json

The command is read-only. It collects operator observability endpoints and returns:

• critical: production traffic or validator participation should be stopped, drained, or held until fixed;
• warning: operator action is needed, but the node may still serve limited traffic depending on role and local policy;
• info: useful context that should not page by default.

monitor-report is designed for polling glue. It does not create incidents, restart sidecars, unban peers, repair payload availability, mutate runtime Raft membership, or submit transactions.

Keep the JSON output as an artifact when opening an incident. Do not expose the report publicly; it is an operator artifact and can reveal local topology and operational state.

Paging Policy

Use this as a starting point and tune it for your environment:

Severity	Default response
Critical	Page immediately. Hold validator admission or drain production traffic until the condition is understood.
Warning	Create a ticket or operator task. Page if repeated, worsening, or near launch/activation.
Info	Retain for context. Do not page unless local policy says otherwise.

Suggested polling:

• validator nodes: every 30-60 seconds;
• observers and single-node environments: usually every 60-300 seconds;
• launch, rejoin, activation, or incident windows: poll more frequently and archive reports.

Avoid alerting directly on one noisy raw field when a normalized monitor-report rule exists. Use raw observability to investigate after the alert fires.

Alert Categories

Node And Database

Watch for unhealthy node status, PostgreSQL disconnects, migration readability failures, process restarts, memory pressure, and API TLS problems.

First commands:

npm run noos -- node health --json
npm run noos -- node metrics
npm run noos -- production monitor-report --json

Handoff:

• Operator CLI
• Production Readiness
• Observability API

Consensus And Peers

Watch for degraded consensus, missing Raft leader, no quorum, HashiCorp sidecar down, peer scoring collapse, missing active trusted peers, and governance versus runtime membership mismatch.

First commands:

npm run noos -- raft status
npm run noos -- raft leader
npm run noos -- raft-go doctor --json
npm run noos -- peers scores --json
npm run noos -- governance operator-membership plan --json

Handoff:

• HashiCorp Raft Deployment
• Raft Consensus Configuration
• Raft Sidecar Operations
• Conflict Handling

State-Root Sidecar

Watch for failed state-root readiness, missing required Nervos sidecar, sidecar version mismatch, unhealthy supervisor, repeated restarts, checkpoint validation failure, and activation windows where readiness is red.

First commands:

npm run noos -- state-root preflight --json
npm run noos -- production monitor-report --json

At or after a governed nervos-smt-v2 activation boundary, there is no safe silent fallback to iden3-v1. Keep the node out of validator traffic until readiness is green.

Handoff:

• State-Root Activation
• Nervos Activation Runbook
• Nervos Sidecar Recovery

Sync, Backfill, And Payloads

Watch for sync/backfill last errors, stuck catch-up, unexpected missing local payloads, payload backfill failures, and payload-store counter drift.

First commands:

npm run noos -- node sync --json
npm run noos -- node payloads
npm run noos -- node reconcile-payload-availability --dry-run

Missing payloads can be expected for metadata-only buckets or unauthorized nodes. Alert only when the node is expected to hold those payloads.

Handoff:

• Operator CLI
• Data Buckets
• Permission Model

Snapshots And Backups

Watch for missing or stale snapshot metadata, failed snapshot verification, and missing backup-drill evidence.

First commands:

npm run noos -- snapshots list
npm run noos -- snapshots verify --file node-backup.noosnap.tar.gz
npm run noos -- production backup-drill --output ./backup-drill

Handoff:

• Backup, Restore, And Recovery
• Snapshots
• Production Readiness

Smart-Contract Runtime

Watch for contract runtime sidecar unavailable, runtime sidecar mismatch, contract readiness stuck, high contract failure rate, repeated runtime errors, and fuel exhaustion spikes.

High-signal thresholds used by current alerts include:

• contract failure rate warning at 5% failed calls in the latest 100-call sample;
• contract failure rate critical at 20% failed calls in the latest 100-call sample;
• runtime error and fuel exhaustion alerting when repeated failures appear in a short window.

First commands:

npm run noos -- contracts alerts --json
npm run noos -- contracts alerts <CONTRACT_ID> --json
npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json
npm run noos -- contracts runtime preflight --json

Handoff:

• Smart Contract Runtime Operations
• Smart Contract Operations Runbooks
• Smart Contract Web Console

Security Posture

Watch for disabled HTTP TLS, missing expected mTLS, development operator tokens, unexpected CORS exposure, and support bundles or logs containing unredacted secrets.

First commands:

npm run noos -- production preflight
npm run noos -- doctor --deep

Handoff:

• Production Readiness
• Admin GUI
• Operator CLI

Verification And Long-Running Work

Watch for chain verification already running, replay verification occupying the node, long snapshot operations, and expensive realtime observability scans.

First commands:

npm run noos -- chain verify
npm run noos -- chain replay-verify
npm run noos -- node storage

Treat these as context unless they block production admission, recovery, or a time-sensitive incident.

Smart-Contract Monitoring Exports

For external alert managers and dashboards, smart-contract monitoring exposes:

GET /monitoring/contracts
GET /monitoring/contracts/alerts
GET /metrics/contracts

Set the operator token before polling:

$env:NOOS_OPERATOR_TOKEN="<operator-token>"
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/monitoring/contracts
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/monitoring/contracts/alerts
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/metrics/contracts

Use /monitoring/contracts for JSON dashboards. Use /monitoring/contracts/alerts for a normalized alert feed. Use /metrics/contracts for Prometheus-compatible scraping.

This page does not repeat every contract metric. For metric details, alert thresholds, and incident actions, see Smart Contract Runtime Operations and Smart Contract Operations Runbooks.

Relationship To Preflight

production preflight is an admission gate. Run it before launch, rejoin, upgrade activation, and role changes.

production monitor-report is a current-state alert report. It is safe to run continuously and should not mutate the node.

Use both:

npm run noos -- production monitor-report --json
npm run noos -- production preflight --profile validator

Do not substitute a green monitor report for a launch preflight, and do not substitute a one-time preflight for continuous monitoring.

Relationship To Incidents

Alerts are symptoms. Incidents are tracked operational facts.

An alert does not automatically create an incident in this milestone. If an alert points to finalized divergence, unsafe runtime state, failed sidecar, missing expected payloads, or data-loss risk, open or update an incident and collect evidence:

npm run noos -- incidents doctor --json
npm run noos -- support bundle --output noos-support-bundle.json

Only mark an incident resolved after the cause is repaired and verification is green. Use Incident Response and Conflict Handling.

External Monitoring

NOOSChain observability does not replace host and infrastructure monitoring. Your external monitoring should still cover:

• CPU, memory, disk, and filesystem pressure;
• PostgreSQL availability, WAL growth, locks, and replication where applicable;
• network reachability and packet loss;
• process supervisor state and restart loops;
• certificate expiry and TLS termination health;
• load balancer health checks;
• OS logs and container/orchestrator events;
• backup storage health and off-node copy status.

Feed those alerts into the same incident workflow as NOOSChain alerts so the operator team has one response path.

Not Covered

This page does not provide:

• managed Prometheus, Grafana, PagerDuty, Opsgenie, or cloud-monitor setup;
• cloud-specific dashboard provisioning;
• automatic sidecar, peer, payload, or incident repair;
• host-level alert rules beyond the signals listed above;
• automatic incident creation.