Build A Three-Validator NOOSChain Cluster

This tutorial builds a new NOOSChain network on three Linux VMs. It is a provider-neutral recipe: use a cloud private network, a private VLAN, or a WireGuard overlay. The finished cluster has three HashiCorp Raft voters, Nervos SMT state roots, Wasmtime contract execution, internal TLS/mTLS, and an optional public interaction UI with a public certificate.

This creates a new chain. Do not use the cleanup steps against an existing chain whose state must be retained. The concrete Noosware installation that proved this procedure is recorded separately in HashiCorp Raft Deployment.

1. Understand The Result

You will deploy the same immutable container image three times. The image does not contain three validators; each VM runs one independently configured copy with its own identity, PostgreSQL database, Raft data, and Nervos checkpoints.

Component	Placement	Network exposure
NOOSChain validator	A, B, and C	Internal TCP 9001
HashiCorp Raft HTTP control	A, B, and C	Internal TCP 9101
HashiCorp Raft replication	A, B, and C	Internal TCP 10001
PostgreSQL	One per validator	Docker network only
Nervos SMT sidecar	Child process in each node	No network port
Wasmtime sidecar	Child process in each node	No network port
Smallstep step-ca	A	Internal TCP 9443
Caddy interaction UI	A, optionally	Public TCP 80/443, UDP 443

Three voters retain quorum while one validator is unavailable. Two unavailable validators stop consensus. HashiCorp Raft is crash-fault tolerant; it is not a Byzantine-fault-tolerant consensus protocol.

The three identity settings represent different software layers but identify the same logical validator in this deployment:

NODE_ID=validator-a
RAFT_NODE_ID=validator-a
HASHICORP_GO_RAFT_NODE_ID=validator-a

Do not confuse the three address types:

Genesis field	Example	Meaning
endpoint	https://10.20.0.11:9001	NOOSChain API and node callback address
raftHttpUrl	https://10.20.0.11:9101	HashiCorp sidecar control API
raftTcpAddress	10.20.0.11:10001	HashiCorp Raft replication address

HASHICORP_GO_RAFT_BIND=0.0.0.0:10001 is a listen address inside the container. HASHICORP_GO_RAFT_ADVERTISE=10.20.0.11:10001 is the reachable address placed in live Raft membership. Never advertise 0.0.0.0, a Docker container address, or loopback to another VM.

2. Choose Hosts And Record Inventory

Start with three fresh 64-bit Linux VMs. A practical initial size is 2 vCPU, 4 GB RAM, and 40 GB SSD per VM. Capacity depends on transaction volume, payloads, retention, and backup policy; monitor and resize before resource pressure affects quorum.

Place validators in separate failure domains when possible. Latency between Raft voters affects write latency, so measure it before launch.

Copy deploy/three-validator/inventory.example.env to a private working directory outside the repository. Replace every documentation address and record:

cluster ID and chain name
image name and immutable digest
public DNS name, if used
SSH user and trusted operator source addresses
node IDs
public IPs
internal IPs
private interface names
provider regions/failure domains

The addresses 192.0.2.0/24 and 10.20.0.0/24 in this tutorial are examples. They must not be copied unchanged into a real deployment.

3. Protect Credentials Before Starting

Apply these rules throughout the installation:

1. Never put passwords, operator tokens, private keys, enrollment tokens, or provider API tokens in source control, chat, tickets, or screenshots.
2. Generate a temporary Ed25519 SSH deployment key on the operator workstation.
3. Install only the .pub line on each VM.
4. Generate each node private key on its own VM. Do not copy it back.
5. Keep admin and organization private keys on the workstation or in a secret manager.
6. Keep the Smallstep root CA key offline. Deploy only the online intermediate bundle to A.
7. Back up private material to encrypted storage before deleting temporary access.

On the workstation:

ssh-keygen -t ed25519 -a 100 -f ~/.ssh/nooschain-cluster-deploy \
  -C nooschain-temporary-deployment

Install nooschain-cluster-deploy.pub into /home/noos-deploy/.ssh/authorized_keys through the provider console or an interactive root session. On every VM:

install -d -m 0700 -o noos-deploy -g noos-deploy /home/noos-deploy/.ssh
chown noos-deploy:noos-deploy /home/noos-deploy/.ssh/authorized_keys
chmod 0600 /home/noos-deploy/.ssh/authorized_keys

Test public-key-only access before disabling passwords:

ssh -o PreferredAuthentications=publickey -o PasswordAuthentication=no \
  -i ~/.ssh/nooschain-cluster-deploy noos-deploy@<A_PUBLIC_IP>

Repeat for B and C. Keep the provider console available while changing SSH or firewall rules.

4. Prepare Each VM

The following commands target Ubuntu/Debian. For another supported Linux distribution, install the equivalent packages using the vendor's documented Docker Engine repository.

sudo apt update
sudo apt upgrade -y
sudo apt install -y ca-certificates curl jq openssl ufw chrony
sudo systemctl enable --now chrony
timedatectl status

Install Docker Engine and the Compose plugin using the current official Docker instructions. Do not use Docker's convenience script for production. At the time of writing, the Ubuntu repository installation is:

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
sudo tee /etc/apt/sources.list.d/docker.sources >/dev/null <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin

Recheck the linked Docker page when installing because supported releases and repository instructions change. Verify:

sudo systemctl enable --now docker
sudo docker run --rm hello-world
sudo docker compose version

If the deployment user is intentionally trusted with root-equivalent Docker access:

sudo groupadd -f docker
sudo usermod -aG docker noos-deploy

Log out and back in, then verify docker info. Membership in the docker group is privileged; remove it after handoff if the operating model does not require it.

Create the installation directory:

sudo install -d -m 0700 -o noos-deploy -g noos-deploy \
  /home/noos-deploy/nooschain-cluster

Record the result on every VM:

uname -a
cat /etc/os-release
nproc
free -h
df -hT /
timedatectl show -p NTPSynchronized
docker version
docker compose version

5. Create The Internal Network

Every validator needs a stable address reachable by the other two validators. Choose one of the following patterns.

5.1 Provider private subnet

Create one private network/VPC and a subnet such as 10.20.0.0/24. Attach all three VMs and assign fixed addresses:

validator-a  10.20.0.11
validator-b  10.20.0.12
validator-c  10.20.0.13

Provider consoles differ. On each VM, use ip -br address and ip route to find the internal interface and confirm its address. Do not invent an OS network configuration when the provider already injects one.

If the provider requires guest-side configuration, first use its documented address, prefix, MTU, and routing values. On Ubuntu with Netplan, create a provider-specific file such as the following on A:

network:
  version: 2
  ethernets:
    <A_PRIVATE_INTERFACE>:
      addresses:
        - <A_PRIVATE_IP>/<PRIVATE_PREFIX_LENGTH>
      routes:
        - to: <PRIVATE_SUBNET_CIDR>
          scope: link

Use sudo netplan try from a provider-console session before sudo netplan apply; do not replace the public-interface configuration. Repeat with each VM's unique address. A private network with a provider gateway may need the provider's route instead of the link-scope example.

Test all six directions:

ping -c 3 <B_PRIVATE_IP>
ping -c 3 <C_PRIVATE_IP>

Run equivalent checks from B and C.

5.2 WireGuard overlay

If the provider has no shared private network, create a WireGuard overlay and use its fixed addresses as the internal addresses. Exchange only WireGuard public keys, allow its UDP listen port only from the other two public IPs, and route validator traffic through the overlay. Follow the OS/vendor WireGuard documentation and verify all six directions before continuing.

5.3 Direct public addresses

This is the least-preferred fallback. Use mTLS and restrict every internal port to the other validators' exact /32 public addresses. Never expose 9001, 9101, 9443, or 10001 to the whole internet.

6. Apply The Network Firewall Matrix

Configure the provider firewall first. Keep SSH open until a more restrictive operator source range has been tested.

Direction	Protocol/port	Source	Destination
Inbound	TCP 22	Trusted operators	A, B, C
Inbound	TCP 9001, 9101, 10001	Other validator IPs	A, B, C
Inbound	TCP 9443	B and C	A only
Inbound	TCP 80, 443	Internet	Public ingress VM only
Inbound	UDP 443	Internet, optional	Public ingress VM only

Leave outbound HTTPS/DNS/NTP and required provider networking available. PostgreSQL 5432 must not be published. Nervos and Wasmtime need no firewall ports.

Copy configure-host-firewall.sh to each VM. Determine the private interface with ip -br address. Preview A's UFW reset without changing it:

./configure-host-firewall.sh \
  --private-interface <A_PRIVATE_INTERFACE> \
  --private-ip <A_PRIVATE_IP> \
  --peer-ip <B_PRIVATE_IP> \
  --peer-ip <C_PRIVATE_IP> \
  --public-ingress --ca-host --dry-run

Remove --dry-run and run with sudo after reviewing the output. Configure B and C without --public-ingress or --ca-host. Do one VM at a time and open a new SSH session immediately after every change.

Docker-published ports can interact with host firewall rules. Bind internal ports only to the internal address as the supplied Compose file does, enforce the provider firewall too, and test exposure from an unrelated external host.

7. Build And Test The Image In Development

Do this on the operator workstation before touching production data. The local tests use disposable development identities, no public interfaces, HashiCorp Raft, Nervos SMT, and the packaged Wasmtime runtime.

npm ci
npm run docker:build
npm run docker:smoke
npm run docker:test-two-validators
npm run docker:test-two-validators-one-observer

The smoke test commits two transactions. The two-validator tests commit ten transactions and require matching heights, block hashes, and Nervos state roots. The observer variant also proves one non-voter catches up and cannot propose blocks.

For an interactive development validator:

Copy-Item docker/env/development-validator.env.example docker/config/validator.env
$env:NOOS_VALIDATOR_ENV_FILE = "./config/validator.env"
docker compose -f docker/compose.local.yaml up -d
docker compose -f docker/compose.local.yaml exec validator noos node health --json
docker compose -f docker/compose.local.yaml exec validator noos raft-go health --json
docker compose -f docker/compose.local.yaml exec validator noos state-root preflight --json
docker compose -f docker/compose.local.yaml exec validator noos contracts runtime preflight --json

Development examples contain known credentials and disable TLS. Never expose them to the internet or reuse their keys, databases, genesis, or Raft volumes in production. Stop the interactive stack with docker compose ... down. Only add --volumes after confirming the state is disposable.

8. Produce One Immutable Production Image

Prefer a release built in trusted CI, pushed to a registry, signed, and referenced by digest:

registry.example.com/nooschain@sha256:<digest>

The repository's release commands build the TypeScript node, CLI, migrations, HashiCorp binary, Nervos sidecar, and Wasmtime sidecar into one image:

$env:NOOSCHAIN_RELEASE_IMAGE = "registry.example.com/nooschain"
$env:NOOSCHAIN_VERSION = "0.1.0"
npm run docker:release-build
npm run docker:release-push

On every VM, authenticate without placing the registry password on the command line, then pull and inspect the exact digest:

docker login registry.example.com
docker pull 'registry.example.com/nooschain@sha256:<digest>'
docker image inspect 'registry.example.com/nooschain@sha256:<digest>' \
  --format '{{json .RepoDigests}} {{.Id}}'

If a registry is unavailable, build once, docker save once, calculate a SHA-256 checksum, copy the same archive to all VMs, verify it, and then run docker load. Do not compile independently on the three VMs.

docker save -o nooschain-production.tar nooschain:<version>
Get-FileHash -Algorithm SHA256 nooschain-production.tar
scp -i $HOME\.ssh\nooschain-cluster-deploy nooschain-production.tar `
  noos-deploy@<A_PUBLIC_IP>:~/

On every VM:

sha256sum ~/nooschain-production.tar
docker load --input ~/nooschain-production.tar
rm ~/nooschain-production.tar

Record the source revision, release version, archive checksum if used, registry digest, and loaded image ID in the installation record.

9. Stage The Deployment Files

Copy these files from deploy/three-validator into ~/nooschain-cluster on every VM:

compose.yaml
configure-node.sh
enroll-node-certificate.sh
raft-membership.sh
raft-status.sh
renew-node-certificate.sh
install-renewal-cron.sh

Copy compose.ca.yaml only to A. Copy compose.public.yaml, Caddyfile, and set-public-upstream.sh only to the public ingress VM. Copy the built web UI into its web/ directory.

Do not copy .env, node.env, a private key, or an enrollment token from a previous cluster.

For example, from the repository root on a Unix-like workstation:

scp -i ~/.ssh/nooschain-cluster-deploy \
  deploy/three-validator/{compose.yaml,configure-node.sh,enroll-node-certificate.sh,raft-membership.sh,raft-status.sh,renew-node-certificate.sh,install-renewal-cron.sh} \
  noos-deploy@<A_PUBLIC_IP>:~/nooschain-cluster/

Repeat for B and C, then copy the A-only overlays separately. On PowerShell, pass the source paths individually if brace expansion is unavailable.

10. Create Production Node Identities

On each VM:

cd ~/nooschain-cluster
umask 077
mkdir -p secrets config pki
openssl genpkey -algorithm ED25519 -out secrets/node.key
openssl pkey -in secrets/node.key -pubout -out secrets/node.pub
chmod 0400 secrets/node.key
chmod 0644 secrets/node.pub

Copy only node.pub from each VM into a private workstation staging directory, named after that validator. Do not download node.key.

Copy deploy/three-validator/inventory.example.json to the staging directory, replace every example value, and choose an immutable createdAt. Put the three public files next to the generator output path as validator-a.pub, validator-b.pub, and validator-c.pub (or use the IDs in your inventory).

Generate admin/organization identities, genesis, and membership credentials:

tsx scripts/deployment/generate-three-validator-material.ts `
  --inventory C:\secure\nooschain\inventory.json `
  --secret-dir C:\secure\nooschain\identity `
  --output-dir C:\secure\nooschain\deployment

The two private PEM files remain under --secret-dir. Back them up encrypted. The output directory contains public deployment material. Review genesis.json and confirm every node has the expected endpoint, raftHttpUrl, and raftTcpAddress.

Copy the identical genesis.json to config/genesis.json on all validators. Copy only the matching <node-id>-membership.json to secrets/node-membership.json on each VM. Compare the genesis SHA-256 on all three hosts before continuing.

11. Create The Internal Certificate Authority

Browser TLS and validator identity solve different problems. Caddy obtains a public server certificate from Let's Encrypt. Validator APIs and Raft require renewable client and server identities, so this recipe uses a private Smallstep step-ca.

Initialize it on the operator workstation:

.\deploy\three-validator\initialize-step-ca.ps1 `
  -AuthorityDirectory "$HOME\.nooschain\pki\example-production-1" `
  -CaName "Example NOOSChain Production CA" `
  -CaUrl "https://<A_PRIVATE_IP>:9443" `
  -CaDnsName "ca.nooschain.internal"

Follow Smallstep's production considerations. Back up the complete authority directory in encrypted offline storage. Never copy the offline root key or provisioner password to a validator. Copy only online-bundle to A as ~/nooschain-cluster/step-ca-online.

Copy online-bundle/certs/root_ca.crt and online-bundle/certs/intermediate_ca.crt into each node's pki/ directory. These certificates are public trust material, not private signing keys.

On A, create a mode-0600 .env containing only the private bind address, then start the standalone CA overlay. It does not depend on validator configuration:

printf 'PRIVATE_IP=%s\n' '<A_PRIVATE_IP>' > .env
chmod 0600 .env
docker compose -f compose.ca.yaml up -d step-ca
curl --fail --silent --show-error \
  --cacert step-ca-online/certs/root_ca.crt \
  https://<A_PRIVATE_IP>:9443/health

12. Enroll A Certificate On Each VM

For each node, create a short-lived, one-use token on the workstation:

.\deploy\three-validator\create-node-enrollment-token.ps1 `
  -AuthorityDirectory "$HOME\.nooschain\pki\example-production-1" `
  -NodeId validator-a -PrivateIp <A_PRIVATE_IP> `
  -OutputFile C:\secure\nooschain\validator-a-token

Copy the public root certificate, public intermediate certificate, and that node's token to the matching VM. On the VM:

mv validator-a-token enrollment-token
./enroll-node-certificate.sh validator-a <A_PRIVATE_IP> \
  https://<A_PRIVATE_IP>:9443

The TLS private key is created on the VM. The helper deletes the token and checks the chain, expiry margin, and client/server purposes. Repeat separately for B and C. Delete all token copies from the workstation.

Verify manually on every VM:

openssl verify -CAfile pki/root_ca.crt \
  -untrusted pki/intermediate_ca.crt pki/node-leaf.crt
openssl x509 -in pki/node-leaf.crt -noout -dates -ext subjectAltName -purpose
stat -c 'mode=%a uid=%u gid=%g %n' pki/node.key

13. Render Production Configuration

On A:

NOOSCHAIN_IMAGE='registry.example.com/nooschain@sha256:<digest>' \
RAFT_CLUSTER_ID='nooschain-example-production-1' \
RAFT_VOTER_NODE_IDS='validator-a,validator-b,validator-c' \
NOOS_STEP_CA_URL='https://<A_PRIVATE_IP>:9443' \
NOOSCHAIN_PUBLIC_DNS='chain.example.com' \
./configure-node.sh validator-a <A_PRIVATE_IP> true

Run the same command on B and C with their node ID/address and final argument false. Exactly one brand-new node may use bootstrap true.

The rendered configuration enables:

NODE_ENV=production
NOOS_CONSENSUS_MODE=raft
RAFT_ENGINE=hashicorp-go
RAFT_IMPLEMENTATION=external_process
RAFT_AUTO_BUILD_BLOCKS=true
NOOS_STATE_ROOT_ENGINE=nervos-smt-v2
NOOS_SMART_CONTRACTS_ENABLED=true
NOOS_CONTRACT_RUNTIME_PREFLIGHT=required
NOOS_TLS_ENABLED=true
NOOS_MTLS_REQUIRED=true
GO_RAFT_TLS_ENABLED=true
GO_RAFT_MTLS_REQUIRED=true
GO_RAFT_TCP_TLS_ENABLED=true
GO_RAFT_TCP_MTLS_REQUIRED=true

nervos-smt-v2 must be selected when the production genesis block is created. Do not reuse an existing iden3-v1 database and try to change its engine with only an environment variable.

Validate without printing secrets:

docker compose config --quiet
stat -c 'mode=%a uid=%u gid=%g %n' \
  .env node.env secrets/node.key secrets/node-membership.json

Expected modes are 0600 for environment/secret-value files, 0400 owned by container UID/GID 10001 for the node key, and read-only access for the public signed membership credential.

14. Start The Fresh Production Cluster

Start only A's validator first:

docker compose up -d validator
docker compose ps
curl -fsS --cacert pki/root_ca.crt --cert pki/node.crt --key pki/node.key \
  https://<A_PRIVATE_IP>:9001/health
./raft-status.sh

A must report itself as the only voter and the leader. Initialize it with the reviewed genesis:

curl -fsS -H 'content-type: application/json' \
  --cacert pki/root_ca.crt --cert pki/node.crt --key pki/node.key \
  --data-binary @config/genesis.json \
  https://<A_PRIVATE_IP>:9001/chain/init

Start B, initialize it from its byte-identical genesis, and verify its chain identity. From A, add it as a non-voter:

# On B
docker compose up -d validator
curl -fsS -H 'content-type: application/json' \
  --cacert pki/root_ca.crt --cert pki/node.crt --key pki/node.key \
  --data-binary @config/genesis.json \
  https://<B_PRIVATE_IP>:9001/chain/init

# On A
./raft-membership.sh add-nonvoter validator-b <B_PRIVATE_IP>:10001
./raft-membership.sh status

Wait until B's commit and last-applied indexes equal A's, then promote it:

./raft-membership.sh add-voter validator-b <B_PRIVATE_IP>:10001
./raft-membership.sh status

On C, start and initialize the validator using <C_PRIVATE_IP>. From A, add C as a non-voter, wait for complete catch-up, and then promote it. Never promote both joining nodes concurrently. The final membership must contain exactly three voters and no unexpected server IDs or addresses.

15. Verify TLS, Consensus, Nervos, And Wasmtime

From every validator, an authenticated request must succeed:

curl -fsS --cacert pki/root_ca.crt --cert pki/node.crt --key pki/node.key \
  https://<PEER_PRIVATE_IP>:9001/health

The same request without --cert and --key must fail during the TLS handshake. Do the equivalent test for :9101. Then run inside each validator:

docker exec \
  -e NOOS_CLI_BASE_URL=https://127.0.0.1:9001 \
  -e NOOS_CLI_TLS_CA_PATH=/run/secrets/tls/root_ca.crt \
  -e NOOS_CLI_TLS_CERT_PATH=/run/secrets/tls/node.crt \
  -e NOOS_CLI_TLS_KEY_PATH=/run/secrets/tls/node.key \
  nooschain-cluster-validator-1 \
  noos production preflight --profile validator --json

Also verify raft-status.sh, state-root readiness, Nervos replay, contract runtime preflight, and the same height/block hash/state root on all three nodes.

The image also provides the CLI without installing Node.js on the VM. Use exec for commands that need the running node's local database or loopback sidecars, and the tools container for API-only commands:

docker compose exec validator noos raft-go health --json
docker compose exec validator noos state-root readiness --json
docker compose exec validator noos contracts runtime preflight --json
docker compose run --rm cli node health --json

The CLI service mounts certificates and sets its API endpoint to the validator container. Do not put the operator token directly in a command line; use the mode-0600 secret already read by the node-side helpers.

Submit test transactions using an admin key from the workstation, preferably through SSH tunnels or the filtered public API. The deployment verifier accepts a URL list and count:

tsx scripts/deployment/verify-three-validator-cluster.ts `
  --secret-dir C:\secure\nooschain\identity `
  --urls https://chain.example.com/api `
  --count 10

Require all ten transactions to execute and the heads to converge.

16. Publish The Interaction UI With Public TLS

This section is optional. Create the public DNS record only for the ingress VM. Do not publish the operator UI.

On A, confirm .env contains:

PUBLIC_DNS=chain.example.com
NOOSCHAIN_API_UPSTREAM=<CURRENT_LEADER_PRIVATE_IP>:9001

Start Caddy:

docker compose -f compose.yaml -f compose.public.yaml up -d caddy

Caddy obtains and renews the browser-facing public certificate. Its upstream connection uses the internal CA and a client certificate. The supplied Caddyfile blocks initialization, operator, governance, peer, sync, metrics, snapshot, and direct block-production routes.

Verify externally:

curl -fsS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' \
  https://chain.example.com/health

The current routing helper is manual. After leadership changes:

./set-public-upstream.sh <NEW_LEADER_PRIVATE_IP>:9001

It recreates only Caddy. Automate leader-aware routing before depending on unattended public transaction submission.

17. Install Certificate Renewal

Install staggered daily renewal checks so planned restarts do not overlap:

# A
./install-renewal-cron.sh 10
# B
./install-renewal-cron.sh 30
# C
./install-renewal-cron.sh 50

Run renew-node-certificate.sh once on every node. A certificate outside the renewal window must remain unchanged. Monitor certificate-renewal.log, CA health, expiry, and validator restart results.

Back up A's online CA database/configuration/intermediate material together, and the offline authority separately. A CA outage does not immediately stop existing TLS sessions, but it prevents enrollment and renewal.

18. Prove Failover

Run this drill before launch:

1. Record leader, term, membership, commit index, and all three heads.
2. Stop the leader's validator container.
3. Wait for B or C to become leader.
4. Confirm the two survivors retain three configured voters and quorum.
5. Point Caddy at the new leader if public transaction testing is required.
6. Submit one signed transaction and confirm it commits on both survivors.
7. Restart the old leader.
8. Wait for its commit/applied indexes and chain head to catch up.
9. Restore public routing and record the evidence.

Do not stop a second validator while the first is unavailable.

19. Create An Off-Node Backup And Restore It

On each validator, run the built-in drill:

docker exec \
  -e NOOS_CLI_BASE_URL=https://127.0.0.1:9001 \
  -e NOOS_CLI_TLS_CA_PATH=/run/secrets/tls/root_ca.crt \
  -e NOOS_CLI_TLS_CERT_PATH=/run/secrets/tls/node.crt \
  -e NOOS_CLI_TLS_KEY_PATH=/run/secrets/tls/node.key \
  nooschain-cluster-validator-1 \
  noos production backup-drill \
  --output /var/lib/nooschain/state-root-checkpoints/backup-drill \
  --json

The drill must verify the archive, import it into an isolated schema, run structural chain verification, and replay Nervos state deterministically.

Create a PostgreSQL custom-format dump, export a packaged chain snapshot, write SHA256SUMS, and copy them off the VM. Store the copy in encrypted offline or object storage. A workstation-only copy is off-node but not geographically or media redundant.

Create the database dump on A without printing the database password:

umask 077
mkdir -p ~/nooschain-backups/$(date -u +%Y%m%dT%H%M%SZ)
backup_dir=$(find ~/nooschain-backups -mindepth 1 -maxdepth 1 -type d \
  -printf '%T@ %p\n' | sort -n | tail -1 | cut -d' ' -f2-)
docker compose exec -T postgres pg_dump -U nooschain -d nooschain -Fc \
  > "$backup_dir/nooschain.pgdump"
sha256sum "$backup_dir"/* > "$backup_dir/SHA256SUMS"
sha256sum --check "$backup_dir/SHA256SUMS"

Restore the dump into a disposable PostgreSQL container, run chain verification and deterministic replay, compare height/state root, and destroy the drill environment. Never restore stale Raft data into a running cluster member. See Backup, Restore, And Recovery for recovery paths and Snapshots for the complete command reference.

20. Switch To Production Service

There is no single safe “production switch.” Admit the cluster only when all of these are true:

• development volumes and keys were not reused;
• the production image digest and source revision are recorded;
• production genesis is identical on A, B, and C;
• Nervos is active from the production genesis;
• exactly three expected Raft voters are present;
• node API, sidecar HTTP, and Raft TCP require mTLS;
• authenticated checks succeed and unauthenticated checks fail;
• ten test transactions and a leader-failover transaction converged;
• Nervos verification/replay and Wasmtime preflight pass;
• provider and host firewalls match the documented matrix;
• public ingress exposes only the intended UI/API subset;
• certificate renewal was tested;
• an off-node backup and isolated restore drill passed;
• production preflight passes on all three validators;
• monitoring and an incident owner exist.

21. Harden The VMs

Apply changes one VM at a time and recheck quorum after each:

1. Disable SSH password authentication and direct root login after key access and provider-console recovery are proven.
2. Restrict SSH to trusted operator IPs or a management VPN.
3. Remove the temporary deployment key after handoff.
4. Review whether the operator still needs root-equivalent Docker membership.
5. Enable automatic OS security updates under an approved maintenance policy.
6. Keep provider and host firewalls enforced.
7. Keep PostgreSQL off host ports.
8. Retain read-only containers, dropped capabilities, and no-new-privileges.
9. Configure Docker log rotation and system log retention.
10. Alert on disk/inode pressure, memory, container restarts, certificate expiry, backup age, and time drift.
11. Encrypt off-node backups and protect admin/organization/CA keys in a secret manager or offline store.
12. Reboot or upgrade only one voter at a time.

22. Monitor And Operate The Cluster

At minimum, monitor:

• presence of one leader and leader-election churn;
• three expected voters and no unexpected addresses;
• Raft term, commit index, last-applied index, and replication lag;
• chain height and stalled automatic block production;
• matching block hashes and state roots;
• validator/database/container readiness;
• Nervos sidecar restarts and checkpoint freshness;
• Wasmtime preflight and execution failures;
• PostgreSQL size, backup completion, and restore-drill age;
• public HTTPS health and Caddy upstream leader;
• node and CA certificate expiry;
• CPU, memory, disk, inodes, and clock synchronization.

Use Monitoring And Alerting to define thresholds and Incident Response to assign response ownership.

23. Upgrade Without Losing Quorum

For every release:

1. Build and test one new immutable image; record its digest and migrations.
2. Create and verify a backup.
3. Upgrade one follower.
4. Wait for catch-up and run production preflight.
5. Upgrade the other follower and repeat verification.
6. Move or confirm leadership away from the final old node if necessary.
7. Upgrade the final node.
8. Verify membership, heads, Nervos replay, Wasmtime, mTLS, public ingress, and backup readiness again.

Database migrations may make image rollback unsafe. Define and test the rollback boundary before starting the rollout; never discover it during an incident.

24. Troubleshooting Map

Symptom	First checks
SSH key rejected	File location, ownership, modes, username, public-key-only test
docker group missing	Install Docker first or create the group explicitly
Docker cannot bind internal IP	Interface/address inventory and provider attachment
Certificate SAN mismatch	Node ID, internal IP, CA URL, issued SAN list
certificate required	Supply node certificate/key; confirm mTLS is expected
Enrollment token rejected	CA health, token age, clock sync, one-time use
Genesis mismatch	SHA-256 and byte-identical genesis.json on every VM
Multiple independent leaders	More than one node bootstrapped; stop and investigate before writing
Node remains non-voter	Catch-up indexes, advertised address, mTLS, membership command
Transaction stays pending	Leader health and RAFT_AUTO_BUILD_BLOCKS settings
Heads differ	Stop launch, capture diagnostics, inspect Raft apply/replay errors
Nervos not ready	Sidecar binary/path, protocol version, checkpoint directory/permissions
Wasmtime preflight fails	Binary/path, runtime/ABI version, host imports, execution policy
Public writes hit a follower	Update Caddy upstream to current leader
Backup exists but readiness warns	Complete and record an isolated restore drill

Do not wipe a node, remove a Raft server, restore a database, or force a new cluster configuration until the failure mode and quorum consequences are understood.

25. Installation Record And Handoff

Keep a durable record containing no secret values:

• VM provider, region, size, OS, hostname, public/internal addresses;
• node IDs and public-key fingerprints;
• chain/cluster ID and genesis SHA-256;
• image version, source revision, digest, and SBOM/signature references;
• firewall rule identifiers and verification results;
• CA root fingerprint, certificate expiry, and renewal schedule;
• final leader/term/voter membership and converged chain head;
• preflight, transaction, failover, Nervos replay, and Wasmtime evidence;
• backup locations, checksums, retention, and restore-drill evidence;
• monitoring destinations, alert owners, and incident contacts;
• temporary credentials removed at handoff.

The installation is complete only when another authorized operator can use this record to verify, renew, back up, upgrade, and recover the cluster without access to the original installer's shell history.

Related Guides

• Docker Deployment
• HashiCorp Raft Deployment
• Raft Sidecar Operations
• Production Readiness
• State-Root Activation
• Nervos Sidecar Recovery
• Smart Contract Runtime Operations
• Backup, Restore, And Recovery
• Monitoring And Alerting
• Incident Response