HashiCorp Raft Deployment

This page is for operators deploying the external HashiCorp Raft sidecar for NOOSChain Raft validator clusters.

Use it to build the sidecar, configure node and sidecar endpoints, start a cluster, verify readiness, and understand safe membership and recovery boundaries.

Related pages:

Need	Page
General consensus model	Consensus
Raft configuration details	Raft Consensus Configuration
Day-two sidecar diagnostics	Raft Sidecar Operations
Finalized-history conflicts	Conflict Handling
Restore and replacement decisions	Backup, Restore, And Recovery
Adapter internals	Raft Adapter Internals

Deployment Model

Each validator has four important pieces:

Piece	Purpose
TypeScript NOOSChain node	Runs API, transactions, deterministic block execution, state-root verification, replay, and incident handling.
HashiCorp Raft sidecar HTTP control API	Receives block proposals and exposes Raft status/control endpoints to the TypeScript node.
HashiCorp Raft TCP transport	Carries Raft replication traffic between sidecars.
Raft data directory	Stores HashiCorp Raft log, stable store, and snapshots.

The sidecar commits opaque NOOSChain block DTOs. It does not mutate the NOOSChain database directly. The TypeScript node remains responsible for deterministic execution, state roots, replay verification, and conflict incident mode.

Use HashiCorp Raft for production-like Raft validator clusters. Use Noosraft for development, deterministic tests, and simple local harnesses. This is crash-fault-tolerant Raft, not Byzantine consensus.

Build The Sidecar

Build for the local platform:

npm run build:raft-go

Default outputs:

Platform	Output
Windows	dist/raft-go/hashicorp-raft.exe
Linux/macOS	dist/raft-go/hashicorp-raft

Override the output directory:

$env:NOOS_GO_RAFT_OUTPUT_DIR="D:\nooschain\bin"
npm run build:raft-go

Cross-compile examples:

$env:GOOS="linux"
$env:GOARCH="amd64"
npm run build:raft-go

$env:GOOS="windows"
$env:GOARCH="amd64"
npm run build:raft-go

Check the configured binary:

npm run noos -- raft-go version

Endpoint Model

Each node has three separate endpoint families. Keep them distinct.

Endpoint	Example	Used by
TypeScript HTTP API	https://node-a.example:3000	Operators, clients, Admin GUI, committed callback receiver.
Sidecar HTTP control API	https://node-a.example:9101	Local TypeScript node to sidecar bridge.
Sidecar TCP Raft transport	node-a.example:9201	Raft replication between HashiCorp sidecars.

Operator observability uses the TypeScript API, not the sidecar TCP transport.

Example local three-node ports:

Node	TypeScript API	Sidecar HTTP	Sidecar TCP
node-a	http://127.0.0.1:3001	http://127.0.0.1:9101	127.0.0.1:9201
node-b	http://127.0.0.1:3002	http://127.0.0.1:9102	127.0.0.1:9202
node-c	http://127.0.0.1:3003	http://127.0.0.1:9103	127.0.0.1:9203

See examples/hashicorp-raft/ for a local three-node cluster.

Core Configuration

Set the node to use the external HashiCorp Raft engine:

NOOS_CONSENSUS_MODE=raft
RAFT_ENGINE=hashicorp-go
RAFT_TRANSPORT=external_process
RAFT_NODE_ID=node-a
RAFT_CLUSTER_ID=cluster-main

Configure the sidecar process boundary:

HASHICORP_GO_RAFT_BINARY_PATH=dist/raft-go/hashicorp-raft.exe
HASHICORP_GO_RAFT_HTTP_BIND=http://127.0.0.1:9101
HASHICORP_GO_RAFT_BIND=127.0.0.1:9201
HASHICORP_GO_RAFT_DATA_DIR=.noos-raft-go/node-a
HASHICORP_GO_RAFT_CALLBACK_BASE=http://127.0.0.1:3001

Configure supervision:

GO_RAFT_AUTO_RESTART=true
GO_RAFT_MAX_RESTARTS=3
GO_RAFT_RESTART_BACKOFF_MS=2000

HASHICORP_GO_RAFT_DATA_DIR is persistent runtime state. Do not delete it in production unless you are intentionally replacing or removing that Raft member.

Peer Configuration

HASHICORP_GO_RAFT_PEERS is a JSON list of the sidecar peers.

Example for node-a:

[
  {
    "nodeId": "node-a",
    "httpUrl": "http://127.0.0.1:9101",
    "raftAddress": "127.0.0.1:9201",
    "callbackBaseUrl": "http://127.0.0.1:3001"
  },
  {
    "nodeId": "node-b",
    "httpUrl": "http://127.0.0.1:9102",
    "raftAddress": "127.0.0.1:9202",
    "callbackBaseUrl": "http://127.0.0.1:3002"
  },
  {
    "nodeId": "node-c",
    "httpUrl": "http://127.0.0.1:9103",
    "raftAddress": "127.0.0.1:9203",
    "callbackBaseUrl": "http://127.0.0.1:3003"
  }
]

In .env, keep it on one line or use your deployment system's multiline environment support.

Every node should agree on the cluster membership it starts with. Later runtime membership changes are explicit operator actions; governance transactions do not silently hot-reconfigure Raft voters.

Example: Single Local Validator

Use this only for local smoke tests. It exercises the sidecar boundary but does not prove production quorum behavior.

NODE_ENV=development
NOOS_CONSENSUS_MODE=raft
RAFT_ENGINE=hashicorp-go
RAFT_TRANSPORT=external_process
RAFT_NODE_ID=node-a
RAFT_CLUSTER_ID=local-one
HASHICORP_GO_RAFT_BINARY_PATH=dist/raft-go/hashicorp-raft.exe
HASHICORP_GO_RAFT_HTTP_BIND=http://127.0.0.1:9101
HASHICORP_GO_RAFT_BIND=127.0.0.1:9201
HASHICORP_GO_RAFT_DATA_DIR=.noos-raft-go/node-a
HASHICORP_GO_RAFT_CALLBACK_BASE=http://127.0.0.1:3001
HASHICORP_GO_RAFT_PEERS=[{"nodeId":"node-a","httpUrl":"http://127.0.0.1:9101","raftAddress":"127.0.0.1:9201","callbackBaseUrl":"http://127.0.0.1:3001"}]

Start the TypeScript node. It starts and supervises its sidecar.

Check:

npm run noos -- raft-go doctor --json
npm run noos -- raft status --json
npm run noos -- raft leader --json

Example: Three-Node Local Cluster

Use separate .env files, databases, API ports, sidecar HTTP ports, sidecar TCP ports, and data directories.

node-a:

RAFT_NODE_ID=node-a
HASHICORP_GO_RAFT_HTTP_BIND=http://127.0.0.1:9101
HASHICORP_GO_RAFT_BIND=127.0.0.1:9201
HASHICORP_GO_RAFT_DATA_DIR=.noos-raft-go/node-a
HASHICORP_GO_RAFT_CALLBACK_BASE=http://127.0.0.1:3001

node-b:

RAFT_NODE_ID=node-b
HASHICORP_GO_RAFT_HTTP_BIND=http://127.0.0.1:9102
HASHICORP_GO_RAFT_BIND=127.0.0.1:9202
HASHICORP_GO_RAFT_DATA_DIR=.noos-raft-go/node-b
HASHICORP_GO_RAFT_CALLBACK_BASE=http://127.0.0.1:3002

node-c:

RAFT_NODE_ID=node-c
HASHICORP_GO_RAFT_HTTP_BIND=http://127.0.0.1:9103
HASHICORP_GO_RAFT_BIND=127.0.0.1:9203
HASHICORP_GO_RAFT_DATA_DIR=.noos-raft-go/node-c
HASHICORP_GO_RAFT_CALLBACK_BASE=http://127.0.0.1:3003

All three nodes use the same HASHICORP_GO_RAFT_PEERS list containing all three members.

After startup:

npm run noos -- raft status --json
npm run noos -- raft membership --json
npm run noos -- production preflight --profile validator

Example: Production-Style TLS/mTLS

Production deployments should use TLS for the TypeScript API, sidecar control and callback path, and Raft TCP replication transport.

Sidecar HTTP control/callback TLS:

GO_RAFT_TLS_ENABLED=true
GO_RAFT_TLS_CERT_PATH=/etc/noos/tls/node-a.crt
GO_RAFT_TLS_KEY_PATH=/etc/noos/tls/node-a.key
GO_RAFT_TLS_CA_PATH=/etc/noos/tls/ca.crt
GO_RAFT_MTLS_REQUIRED=true

Raft TCP transport TLS:

GO_RAFT_TCP_TLS_ENABLED=true
GO_RAFT_TCP_TLS_CERT_PATH=/etc/noos/raft-tcp/node-a.crt
GO_RAFT_TCP_TLS_KEY_PATH=/etc/noos/raft-tcp/node-a.key
GO_RAFT_TCP_TLS_CA_PATH=/etc/noos/raft-tcp/ca.crt
GO_RAFT_TCP_MTLS_REQUIRED=true
GO_RAFT_TCP_TLS_SERVER_NAME=node-a.raft.internal

The TCP layer verifies certificates against the configured CA and can require mutual certificates. It does not yet bind a Raft node id to a specific certificate subject, so issue one certificate per sidecar and keep the peer network private.

Run the production gate:

npm run noos -- production preflight --profile validator

Startup And Shutdown

Startup:

1. Start the TypeScript NOOSChain node.
2. The TypeScript process starts and supervises its HashiCorp Raft sidecar.
3. The sidecar loads Raft state from HASHICORP_GO_RAFT_DATA_DIR.
4. The node catches up committed entries through sidecar log-entry recovery if callbacks were missed while TypeScript was unavailable.

Shutdown:

1. Stop clients and block proposals.
2. Stop TypeScript nodes.
3. TypeScript asks the sidecar to shut down, then terminates it if needed.
4. Back up HASHICORP_GO_RAFT_DATA_DIR only while the sidecar is stopped or with filesystem-safe backup tooling.

Supervision

If the sidecar exits unexpectedly, TypeScript marks consensus degraded or opens an incident, then restarts the sidecar when GO_RAFT_AUTO_RESTART=true. Restarts use backoff and stop after GO_RAFT_MAX_RESTARTS.

If the sidecar process is alive but /raft/status stops responding, the health loop treats it as unhealthy, terminates it, and restarts it under the same policy.

On startup or sidecar restart, TypeScript asks the sidecar for committed log entries through /raft/log-entries. The sidecar returns opaque NOOSChain block entries. TypeScript executes and verifies them deterministically.

Observability

Use:

npm run noos -- raft status --json
npm run noos -- raft leader --json
npm run noos -- raft membership --json
npm run noos -- raft-go doctor --json
npm run noos -- production monitor-report --json

Observability reports sidecar running status, PID, uptime, restart count, last exit code/signal, last error, next restart time, Raft role, leader, term, last log index/term, commit index, and applied index.

No private keys, DEKs, tokens, or plaintext payloads are exposed.

For more day-two commands, see Raft Sidecar Operations.

Validator Governance And Runtime Membership

Validator governance transactions define the intended active validator set in NOOSChain consensus state. HashiCorp Raft runtime membership does not automatically change when governance changes.

After registering, retiring, suspending, or reactivating a validator, operators should review the runtime membership reconciliation plan:

npm run noos -- governance operator-membership plan --json

Execute only after operator approval:

npm run noos -- governance operator-membership execute --plan-file approved-plan.json --dry-run
npm run noos -- governance operator-membership execute --plan-file approved-plan.json --yes

Runtime membership execution calls sidecar membership endpoints that wrap HashiCorp Raft AddVoter, AddNonvoter, and RemoveServer. Operations are sequenced one at a time and stop on unsafe quorum or health checks.

Observability reports a warning when governance active validators and runtime Raft voters differ.

Example: Membership Reconciliation

Use this after a governance validator-set change.

1. Inspect the mismatch:

   npm run noos -- governance runtime-membership
   npm run noos -- governance operator-membership plan --json

2. Review the plan. Confirm it does not remove quorum or add an unprepared node.

3. Save the approved plan as approved-plan.json.

4. Dry-run:

   npm run noos -- governance operator-membership execute --plan-file approved-plan.json --dry-run

5. Execute:

   npm run noos -- governance operator-membership execute --plan-file approved-plan.json --yes

6. Verify:

   npm run noos -- raft membership --json
   npm run noos -- production preflight --profile validator

Example: Sidecar Degraded Or Restarting

Use this when monitoring reports the sidecar down, degraded, or restarting.

npm run noos -- production monitor-report --json
npm run noos -- raft-go doctor --json
npm run noos -- raft status --json

If a restart is required, review the manual plan:

npm run noos -- raft-go restart --dry-run

Actual lifecycle control should use the process supervisor, operating-system service manager, or node restart procedure. Do not delete the data directory to clear a restart issue.

After repair:

npm run noos -- raft-go doctor --json
npm run noos -- production preflight --profile validator

Recovery Boundaries

Sidecar restart preserves Raft state when HASHICORP_GO_RAFT_DATA_DIR is intact. Losing that directory is not a normal restart.

If a node loses the Raft data directory:

• keep the node out of validator traffic;
• do not reuse the old validator identity blindly;
• inspect runtime membership from healthy validators;
• restore or replace according to the recovery runbook.

Use Backup, Restore, And Recovery for role-specific restore decisions and Conflict Handling if finalized history diverges.

Example: Safe Shutdown And Backup

Use this before host maintenance or filesystem backup.

1. Drain client and validator traffic.

2. Stop the TypeScript node.

3. Confirm the sidecar is stopped:

   npm run noos -- raft-go status --json

4. Back up the TypeScript database and HASHICORP_GO_RAFT_DATA_DIR.

5. Start the node again.

6. Verify:

   npm run noos -- raft-go doctor --json
   npm run noos -- production preflight --profile validator

Example: Local Dev Cluster Reset

Use this only for development clusters.

1. Stop all nodes.

2. Remove only the intended development schemas.

3. Delete only the matching development Raft directories, for example:

   .noos-raft-go/node-a
   .noos-raft-go/node-b
   .noos-raft-go/node-c

4. Start the cluster from fresh databases and fresh sidecar data directories.

Never apply this reset procedure to production data.

Limitations

• This is an MVP crash-fault-tolerant Raft path, not Byzantine consensus.
• Runtime membership changes are explicit operator actions.
• Governance transactions do not autonomously reconfigure voters.
• The sidecar does not implement production secret management.
• OpenBFT, CometBFT, slashing, and Byzantine evidence handling are not part of this deployment path.