NOOSChain Docs

Appearance

Raft Membership Reconciliation ​

NOOSChain separates validator governance from runtime Raft membership.

Governance validator state is consensus state. It is changed only by signed, protocol-versioned governance transactions such as registering, suspending, reactivating, retiring, or updating a validator.

Runtime Raft membership is deployment state. It is the set of voters currently configured in Noosraft or the HashiCorp Raft sidecar. Governance transactions do not automatically mutate runtime Raft membership. Operators must review a reconciliation plan and explicitly execute approved runtime changes.

Purpose ​

The reconciliation tooling compares:

  • active validators from consensus governance state
  • currently running Raft voters from runtime consensus status
  • local trusted_peers used for HTTP sync, gossip, payload backfill, and operator transport

It then produces a safe operator plan. The plan is guidance by default. For the HashiCorp Raft sidecar, an operator can explicitly execute approved plan steps through the protected TypeScript governance API, which calls HashiCorp Raft membership APIs.

APIs ​

Protected chain-admin APIs:

  • GET /governance/runtime-membership
  • GET /governance/runtime-membership/reconciliation-plan
  • POST /governance/runtime-membership/execute

Operator-token APIs:

  • GET /operator/membership/reconciliation-plan
  • POST /operator/membership/execute

The operator membership plan is the preferred operational view because it checks all three planes together:

text
governance active validators
runtime Raft membership
trusted_peers

The execute endpoint accepts explicit operator-approved actions such as:

  • add_trusted_peer
  • update_trusted_peer
  • disable_trusted_peer
  • add_runtime_voter
  • add_runtime_nonvoter
  • remove_runtime_voter

It does not submit governance transactions. If a node should become an active validator, the operator must first submit and finalize the appropriate signed governance transaction.

The execute endpoint accepts only explicit operator-approved actions. It does not infer approval from governance state:

json
{
  "actions": [
    {
      "action": "add_runtime_voter",
      "nodeId": "validator-d",
      "raftAddress": "127.0.0.1:12004",
      "operatorApproved": true
    }
  ]
}

For the HashiCorp Raft sidecar, the executor uses the sidecar membership control endpoints:

  • GET /raft/membership
  • POST /raft/membership/add-voter
  • POST /raft/membership/add-nonvoter
  • POST /raft/membership/remove-voter

Operators should normally call the protected TypeScript governance API rather than the sidecar directly. The sidecar control API should be bound to a local or private management interface and protected with the existing TLS/mTLS deployment settings where exposed outside localhost. If NOOS_OPERATOR_TOKEN is set, the sidecar membership endpoints require Authorization: Bearer <token>, and the TypeScript executor forwards the configured token.

Operator observability:

  • GET /node/observability/overview
  • GET /node/observability/consensus

The observability response includes:

  • governanceValidatorSetHash
  • runtimeValidatorSetHash
  • membershipInSync
  • reconciliationWarnings
  • missingRuntimeValidators
  • unexpectedRuntimeValidators
  • endpointMismatches

Plan Contents ​

The reconciliation plan reports:

  • missingRuntimeValidators: active governance validators that are not runtime Raft voters.
  • unexpectedRuntimeValidators: runtime Raft voters that are not active in governance state.
  • endpointMismatches: validators where governance endpoint metadata differs from runtime peer endpoint configuration.
  • runtimeConfigMismatches: duplicate runtime node IDs or incomplete governance endpoint metadata.
  • votingPowerMismatches: Raft does not use governance voting power in this MVP; future BFT backends may.
  • safeOperations: manual-only operator recommendations.

The operator membership plan additionally reports:

  • missingTrustedPeers: active governance validators missing from trusted_peers.
  • trustedPeerEndpointMismatches: trusted peer HTTP endpoints that differ from governance endpoint metadata.
  • trustedPeerModeMismatches: active validators configured as observer peers.
  • unexpectedTrustedPeers: trusted peers that are not active governance validators. These are not automatically disabled because they may be intentional observers, bootstrap peers, or repair peers.
  • governanceActionsRequired: guidance for cases that require signed governance transactions rather than operator-only transport changes.

Example recommendation:

json
{
  "action": "add_runtime_voter",
  "nodeId": "validator-d",
  "raftAddress": "127.0.0.1:12004",
  "requiresRuntimeChange": true,
  "requiresOperatorApproval": true,
  "requiresQuorum": true,
  "requiresJointConsensus": true,
  "manualOnly": true,
  "safeSequenceStep": 1
}

Runtime Execution ​

Runtime execution is currently available for the HashiCorp Raft engine. It uses HashiCorp Raft's built-in membership APIs:

  • AddVoter
  • AddNonvoter
  • RemoveServer

The executor applies one approved operation at a time. Before each operation it checks that a leader exists, the cluster is reachable through the sidecar, and the requested action is safe enough for this MVP. After each operation it waits for the sidecar to report a stable leader and then refreshes the runtime membership view.

For a replacement such as [a,b,c] -> [a,c,d], the safe plan is:

  1. add d as a voter
  2. verify the cluster is stable as [a,b,c,d]
  3. remove b
  4. verify the cluster is stable as [a,c,d]

The executor deliberately refuses direct quorum-shrinking operations such as removing a voter from a three-voter cluster without first adding a replacement.

Safe Workflow ​

  1. Commit governance transactions first.
  2. Query GET /operator/membership/reconciliation-plan or use npm run noos -- governance operator-membership plan --json.
  3. Review the manual plan across governance, Raft runtime membership, and trusted_peers.
  4. Execute only approved trusted-peer/runtime actions with POST /operator/membership/execute or npm run noos -- governance operator-membership execute --plan-file actions.json --yes.
  5. Verify cluster convergence and chain/replay health.
  6. Re-run reconciliation until the operator plan reports inSync=true.

Tutorials ​

These examples use the operator CLI. The same workflow can be performed through the HTTP APIs listed above.

Set the operator target first:

powershell
$env:NOOS_CLI_BASE_URL="https://127.0.0.1:7001"
$env:NOOS_OPERATOR_TOKEN="<operator-token>"
bash
export NOOS_CLI_BASE_URL="https://127.0.0.1:7001"
export NOOS_OPERATOR_TOKEN="<operator-token>"

1. Add A Validator Node ​

Adding a validator is a three-plane operation:

text
governance validator state
trusted_peers HTTP transport
runtime Raft voter membership

Step 1: submit and finalize a signed governance transaction.

Use the normal transaction flow to submit REGISTER_VALIDATOR_NODE with the new node metadata:

json
{
  "nodeId": "validator-d",
  "organizationId": "org-validator-d",
  "publicKey": "<validator-d-node-public-key>",
  "endpoint": "https://validator-d.example.com:7001",
  "raftHttpUrl": "https://validator-d.example.com:9101",
  "raftTcpAddress": "validator-d.example.com:12001",
  "role": "validator",
  "votingPower": 1,
  "metadata": {}
}

This must be signed by a chain:admin actor. The operator membership CLI does not create governance validators directly.

Step 2: inspect the operator reconciliation plan.

powershell
npm run noos -- governance operator-membership plan --json

Expected findings:

  • missingTrustedPeers contains validator-d.
  • raftMembershipPlan.missingRuntimeValidators contains validator-d.
  • safeOperations includes add_trusted_peer.
  • safeOperations includes add_runtime_voter.

Step 3: create an explicit action file.

Example add-validator-d-actions.json:

json
{
  "actions": [
    {
      "action": "add_trusted_peer",
      "nodeId": "validator-d",
      "baseUrl": "https://validator-d.example.com:7001",
      "operatorApproved": true
    },
    {
      "action": "add_runtime_voter",
      "nodeId": "validator-d",
      "raftAddress": "validator-d.example.com:12001",
      "operatorApproved": true
    }
  ]
}

Step 4: execute the approved operator actions.

powershell
npm run noos -- governance operator-membership execute --plan-file add-validator-d-actions.json --yes --json

Step 5: verify alignment.

powershell
npm run noos -- governance operator-membership plan --json
npm run noos -- chain verify
npm run noos -- chain replay-verify

The desired result is:

  • governance active validators include validator-d
  • trusted_peers includes validator-d as trusted
  • runtime Raft voters include validator-d
  • no membership warnings remain for validator-d

2. Add An Observer Node ​

Observers are transport peers, not governance validators and not Raft voters. They sync blocks and may serve/read allowed availability-layer payloads, but they do not vote in Raft.

For a sync-only observer, do not submit REGISTER_VALIDATOR_NODE and do not add the node as a Raft voter.

Add the observer as a trusted peer with trustMode: observer:

powershell
npm run noos -- peers list --json

Then call the operator peer API, or use the CLI if peer creation is exposed in your current build. HTTP example:

http
POST /peers
Authorization: Bearer <operator-token>
Content-Type: application/json

{
  "id": "observer-a",
  "nodeId": "observer-a",
  "baseUrl": "https://observer-a.example.com:7001",
  "trustMode": "observer"
}

After adding it:

powershell
npm run noos -- peers list --json
npm run noos -- governance operator-membership plan --json

An observer may appear in trusted_peers, but it should not appear in the governance active validator set or runtime Raft voters. If the observer is configured with trustMode: observer, the operator plan should not recommend adding it as a Raft voter.

If an observer is accidentally added as trustMode: trusted, the plan may flag it as an unexpected trusted peer. Fix that by updating/re-adding it as an observer peer or disabling it if it is not intended.

3. Remove A Validator Or Observer Node ​

Removal depends on the node role.

Remove A Validator ​

Validator removal should be staged so the cluster never loses quorum unexpectedly.

Step 1: submit and finalize a governance transaction.

Use one of:

  • SUSPEND_VALIDATOR for temporary removal from the active governance set.
  • RETIRE_VALIDATOR for permanent removal. Retired validators cannot be reactivated.

This must be signed by a chain:admin actor.

Step 2: inspect the operator plan.

powershell
npm run noos -- governance operator-membership plan --json

Expected findings:

  • runtime Raft may still list the validator as a voter.
  • the plan may include remove_runtime_voter.
  • trusted_peers may still contain the node.

Step 3: execute runtime removal only when quorum-safe.

Example remove-validator-b-actions.json:

json
{
  "actions": [
    {
      "action": "remove_runtime_voter",
      "nodeId": "validator-b",
      "operatorApproved": true
    }
  ]
}
powershell
npm run noos -- governance operator-membership execute --plan-file remove-validator-b-actions.json --yes --json

The executor refuses unsafe quorum-shrinking operations in the MVP. For example, removing a voter from a three-voter cluster without first adding a replacement is rejected.

Step 4: disable or reclassify the trusted peer.

If the retired/suspended node should not be used for sync, gossip, or backfill:

powershell
npm run noos -- peers quarantine validator-b --reason "validator suspended"
npm run noos -- peers ban validator-b --reason "validator retired" --yes

Or disable the trusted peer through the operator API:

http
POST /peers/validator-b/disable
Authorization: Bearer <operator-token>

If the node should continue as a sync-only observer, keep it in trusted_peers, but configure it as trustMode: observer rather than a runtime Raft voter.

Step 5: verify.

powershell
npm run noos -- governance operator-membership plan --json
npm run noos -- chain verify
npm run noos -- chain replay-verify

Remove An Observer ​

Observers are local transport peers. Removing an observer does not require a governance transaction and does not require Raft membership changes.

Disable the peer:

http
POST /peers/observer-a/disable
Authorization: Bearer <operator-token>

Or ban/quarantine it if the removal is due to operational risk:

powershell
npm run noos -- peers quarantine observer-a --reason "operator removal"
npm run noos -- peers ban observer-a --reason "operator removal" --yes

Then verify:

powershell
npm run noos -- peers list --json
npm run noos -- governance operator-membership plan --json

The observer should be absent or disabled in trusted_peers, absent from runtime Raft voters, and absent from governance active validators.

Current Limits ​

  • No automatic Raft membership mutation from governance transactions.
  • No automatic trusted-peer mutation from governance transactions.
  • Runtime mutation is explicit and operator-approved.
  • No fully autonomous orchestration yet.
  • No Byzantine slashing or staking.
  • Peer scoring remains local operational defense and never changes validator governance state.

Future work can add richer autonomous orchestration, backend-specific joint consensus workflows, or BFT validator-set update machinery while preserving the same governance/runtime boundary.

Audience-first NOOSChain documentation.

NOOSChain