Smart Contract Runtime Operations

This page is for operators who run the NOOSChain smart-contract runtime in production or pre-production. It covers activation, Wasmtime sidecar rollout, runtime preflight, local contract readiness, monitoring, incident response, and the evidence to preserve when something goes wrong.

Operators do not normally author contracts, but they are responsible for the node-side environment that makes contract execution safe and replayable.

Related pages:

Need	Page
General contract model	Smart Contracts
Contract author workflow	Smart Contract Developers Overview
Package hashes, signatures, and provenance	Packages And Provenance
Registry approval and package status	Contract Registry
Publishing and release workflow	Publishing
Runtime build provenance	Runtime Reproducible Builds
Platform monitoring	Monitoring And Alerting
General incident process	Incident Response
Backup and restore	Backup, Restore, And Recovery

Runtime Model

Smart-contract execution is consensus-significant. A successful contract call can write contract state, emit contract events, create call results, and affect the block state root. Every validator that executes the same block must reach the same result.

NOOSChain splits the work:

• the TypeScript node validates chain state, registry policy, package provenance, manifests, bucket permissions, transactions, and block effects;
• the Rust noos-contract-executor-wasmtime sidecar executes deterministic WASM under a protocol-pinned runtime profile;
• the active runtime requirements define the sidecar protocol, ABI version, executor version, Wasmtime version, required commands, required host imports, and fuel-metering requirement;
• local readiness checks decide whether this node is safe to receive contract traffic, especially when buckets or encrypted payloads are local surfaces.

There are two important boundaries:

Boundary	Consensus-visible?	Operator meaning
Runtime activation, rollout policy, package registry, contract status, contract state	Yes	Must change through transactions and replay deterministically.
Sidecar process, sidecar path, local payload availability, local readiness cache, operator token	No	Must be healthy before routing traffic, but cannot rewrite chain history.

If the active runtime requires Wasmtime and the sidecar is missing, unhealthy, or reports mismatched capabilities, the node must fail closed for contract execution. Do not bypass a runtime mismatch by editing local requirements.

Operator Responsibilities

Operators own:

• enabling or disabling smart-contract execution for the node role;
• deploying the correct Wasmtime sidecar binary;
• running runtime preflight before admitting production traffic;
• validating rollout policies and validator attestations before activation;
• monitoring contract calls, failures, fuel use, readiness, and release state;
• routing traffic away from nodes with failed local readiness;
• preserving evidence for runtime, package, bucket, caller, and contract-code incidents;
• using transactions for activation, release, deactivation, and policy changes.

Contract authors own WASM behavior, manifests, ABI design, package signatures, and migrations. Operators should cross-link to the author pages instead of trying to repair contract code directly on a live chain.

Decision Table

Situation	First action
Enable contract execution	Run runtime preflight, inspect activation, register rollout policy if required, submit transaction SET_CONTRACT_RUNTIME_ACTIVATION.
Deploy a sidecar binary	Build/test sidecar, drain one node, replace binary/path, run production preflight, roll node by node.
Check whether one contract is callable	Run contracts readiness <CONTRACT_ID> --refresh --json.
All contract calls fail	Check sidecar availability, runtime preflight, activation status, and global alerts.
One contract fails	Check recent call failures, readiness, ABI, package status, bucket access, and dependent callers.
Runtime mismatch	Treat as a binary/config/protocol mismatch; deploy the matching sidecar or plan a governed runtime upgrade.
Readiness stuck	Inspect bucket metadata, payload availability, permissions, registry status, and runtime readiness.
Unsafe contract	Stop callers, inspect dependencies, submit transaction DEACTIVATE_CONTRACT if required.
Node restored from backup	Verify runtime preflight and contract readiness before routing calls.

Configuration

Production contract nodes should make runtime behavior explicit:

NOOS_SMART_CONTRACTS_ENABLED=true
NOOS_CONTRACT_RUNTIME_PREFLIGHT=required
NOOS_CONTRACT_WASMTIME_SIDECAR_PATH=C:\nooschain\bin\noos-contract-executor-wasmtime.exe
NOOS_OPERATOR_TOKEN=<operator-token>

Linux/macOS example:

NOOS_SMART_CONTRACTS_ENABLED=true
NOOS_CONTRACT_RUNTIME_PREFLIGHT=required
NOOS_CONTRACT_WASMTIME_SIDECAR_PATH=/opt/noos/bin/noos-contract-executor-wasmtime
NOOS_OPERATOR_TOKEN=<operator-token>

NOOS_SMART_CONTRACTS_ENABLED=true tells production preflight that this node is expected to run the contract layer. When it is not set to true, preflight can warn instead of block depending on the selected profile.

NOOS_CONTRACT_RUNTIME_PREFLIGHT=required makes startup and production gates fail when the sidecar is missing, cannot start, lacks fuel metering, or reports capabilities that do not match the active protocol requirements. Use warn only in development or during controlled diagnostics.

NOOS_CONTRACT_WASMTIME_SIDECAR_PATH points to the sidecar binary. If unset, the node uses the bundled release binary under external/noos-contract-executor-wasmtime/target/release when available.

NOOS_OPERATOR_TOKEN protects operator observability and monitoring endpoints. Keep it out of command history and incident notes unless redacted.

Core Commands

Runtime and activation:

$env:NOOS_OPERATOR_TOKEN="<operator-token>"
npm run noos -- contracts runtime preflight --json
npm run noos -- contracts runtime activation --json
npm run noos -- contracts runtime rollout --json
npm run noos -- contracts runtime provenance --json

Readiness and alerts:

npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json
npm run noos -- contracts alerts --json
npm run noos -- contracts alerts <CONTRACT_ID> --json
npm run noos -- contracts monitoring snapshot --json
npm run noos -- contracts monitoring alerts --json
npm run noos -- contracts monitoring prometheus

Production gates:

npm run noos -- production preflight --profile validator
npm run noos -- production monitor-report --json
npm run noos -- chain verify
npm run noos -- chain replay-verify

Activation Gate

Production smart-contract execution is controlled by runtime activation state. Operators can stage packages and registry entries while the runtime is disabled, but deployment, instantiation, and calls are rejected unless activation permits them.

Activation is chain state. It is changed by submitting transaction SET_CONTRACT_RUNTIME_ACTIVATION; it is not changed by flipping a local env var.

Safe activation flow:

1. Deploy the sidecar binary to each validator.
2. Run local runtime preflight on each validator.
3. Confirm the expected runtime requirements hash.
4. Register a rollout policy if the environment requires rollout enforcement.
5. Collect validator attestations for the rollout policy.
6. Submit transaction SET_CONTRACT_RUNTIME_ACTIVATION.
7. Verify activation and production preflight after the activation block.

Check local status:

npm run noos -- contracts runtime preflight --json
npm run noos -- contracts runtime activation --json

Activation payload example:

{
  "status": "active",
  "runtime": "wasm-assemblyscript-v1",
  "protocolVersion": 1,
  "activationHeight": "12345",
  "requirements": {
    "sidecarProtocolVersion": 2,
    "abiVersion": "noos-contract-abi-v2",
    "executorVersion": "0.1.0",
    "wasmtimeVersion": "26.0.1",
    "requiredCommands": ["handshake", "validate", "call"],
    "requiredHostImports": [
      "noos.storage_get",
      "noos.storage_set",
      "noos.bucket_get_metadata",
      "noos.bucket_get_record_metadata",
      "noos.bucket_add_encrypted_record_metadata",
      "noos.emit_event",
      "noos.return_value",
      "noos.last_result_ptr",
      "noos.last_result_len",
      "noos.contract_call",
      "env.abort"
    ],
    "requireFuelMetering": true
  },
  "requirementsHash": "<sha256-canonical-requirements>",
  "preflightAttestation": {
    "ok": true,
    "requirementsHash": "<sha256-canonical-requirements>",
    "checkedAt": "2026-06-01T00:00:00.000Z",
    "nodeId": "node-validator-1"
  },
  "reason": "Enable production smart contracts after preflight",
  "metadata": {
    "changeControlId": "CHG-1234"
  }
}

To pause contract execution, submit transaction SET_CONTRACT_RUNTIME_ACTIVATION with status: "disabled" or status: "preflight_required". Registry review can continue while execution is paused.

Rollout Policy And Attestations

A runtime rollout policy proves that validators are checking the same runtime requirements and contract policy before activation opens the execution gate.

The rollout policy pins:

• runtime id;
• protocol version;
• canonical runtime requirements hash;
• contract policy hash;
• rollout status and metadata.

Inspect rollout state:

npm run noos -- contracts runtime rollout --json
npm run noos -- contracts runtime attestations --rollout-policy-id contract-runtime-rollout-v1 --json

Register a rollout policy by submitting transaction REGISTER_CONTRACT_RUNTIME_ROLLOUT_POLICY:

{
  "id": "contract-runtime-rollout-v1",
  "status": "active",
  "runtime": "wasm-assemblyscript-v1",
  "protocolVersion": 1,
  "requirementsHash": "<sha256-canonical-requirements>",
  "policyHash": "<sha256-canonical-contract-policy>",
  "metadata": {
    "changeControlId": "CHG-1234"
  }
}

Each validator then submits transaction SUBMIT_CONTRACT_RUNTIME_ROLLOUT_ATTESTATION:

{
  "id": "attest-node-validator-1-runtime-v1",
  "nodeId": "node-validator-1",
  "rolloutPolicyId": "contract-runtime-rollout-v1",
  "requirementsHash": "<sha256-canonical-requirements>",
  "policyHash": "<sha256-canonical-contract-policy>",
  "sidecarProtocolVersion": 2,
  "executorVersion": "0.1.0",
  "wasmtimeVersion": "26.0.1",
  "abiVersion": "noos-contract-abi-v2",
  "preflightOk": true,
  "preflightHash": "<sha256-canonical-preflight-summary>",
  "metadata": {
    "operator": "validator-1"
  }
}

When activating with rollout enforcement, include:

{
  "rolloutPolicyId": "contract-runtime-rollout-v1",
  "requireAllActiveValidatorsAttested": true,
  "minValidatorAttestations": 1
}

If a validator is missing or reports different hashes, activation should fail closed. Fix the validator or policy before retrying.

Sidecar Operations

Use this when deploying or replacing noos-contract-executor-wasmtime.

Before rollout:

npm run build:contract-wasmtime-sidecar
npm run test:contract-wasmtime-sidecar
npm run test:contract-wasmtime-sidecar-lifecycle
npm run test:contract-runtime-compatibility
npm run noos -- contracts runtime provenance --json

Rolling sequence:

1. Drain contract traffic from one node.
2. Stop the node cleanly.
3. Replace the sidecar binary or update NOOS_CONTRACT_WASMTIME_SIDECAR_PATH.
4. Start the node.
5. Run runtime and production preflight.
6. Exercise a known low-risk contract call or wait for normal traffic.
7. Watch monitoring for failures, runtime errors, fuel use, and host calls.
8. Roll to the next node only after the first node is stable.

Verification:

npm run noos -- contracts runtime preflight --json
npm run noos -- production preflight --profile validator
npm run noos -- production monitor-report --json

Rollback is a binary/config rollback, not a protocol relaxation:

1. Drain traffic.
2. Restore the previous sidecar binary and path.
3. Restart the node.
4. Re-run runtime and production preflight.
5. Verify failure rates return to baseline.

Do not edit local runtime requirements to silence CONTRACT_RUNTIME_CAPABILITY_MISMATCH. That changes the assumptions under which blocks are replayed.

Contract Readiness

Contract readiness answers: can this local node safely execute this contract now?

Readiness can be false even when the contract exists in consensus state. Common local blockers are missing sidecar readiness, missing bucket metadata, missing encrypted payloads, and local access-rule visibility.

Refresh readiness:

npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json

HTTP equivalent:

GET /contracts/<CONTRACT_ID>/readiness?refresh=true

Readiness considers:

Check	Why it matters
Runtime active and sidecar compatible	The node must be able to execute the active protocol-pinned runtime.
Contract status active	Inactive contracts must not execute.
Manifest and ABI available	The node needs method, bucket, call, and ABI metadata.
Bucket metadata available	Methods declare bucket requirements.
Contract principal permissions	Contracts need explicit bucket access.
Caller permissions	Users, organizations, or caller contracts may also need bucket access.
Encrypted payload availability	Methods that require local encrypted payloads must run on nodes that have them.
Registry/package policy	Package status and provenance may block unsafe contracts.

Common readiness repairs:

Symptom	Likely cause	Repair
CONTRACT_NOT_READY_ON_NODE	Local readiness is false.	Refresh readiness, then repair the reported bucket, payload, runtime, or permission blocker.
CONTRACT_NESTED_BUCKET_AVAILABILITY_UNSATISFIED	Target contract in a nested call lacks local bucket payloads.	Backfill payloads or route to a node where the target is ready.
Missing contract bucket permission	Contract principal lacks access.	Submit an authorized bucket access-rule transaction for the contract principal.
Missing caller bucket permission	User, organization, or caller contract lacks access.	Grant the required permission through the normal permission transaction flow.
Runtime mismatch in readiness	Sidecar capability mismatch.	Follow the runtime mismatch runbook below.

Nested calls use the target contract's readiness. A healthy caller can still fail if the target contract is not locally ready.

Monitoring

Use the JSON snapshot for dashboards:

npm run noos -- contracts monitoring snapshot --json

Use the alert feed for alert-manager style integrations:

npm run noos -- contracts monitoring alerts --json

Use Prometheus exposition for scraping:

npm run noos -- contracts monitoring prometheus

HTTP endpoints:

GET /node/observability/contracts
GET /node/observability/contracts/<CONTRACT_ID>
GET /node/observability/contracts/<CONTRACT_ID>/calls?status=failed&limit=50
GET /node/contracts/alerts
GET /contracts/<CONTRACT_ID>/alerts
GET /metrics/contracts

Alert mapping:

Metric or alert	Meaning	First runbook
noos_contract_sidecar_available == 0	Sidecar unavailable.	Sidecar unavailable.
noos_contract_sidecar_mismatch == 1	Runtime requirements mismatch.	Runtime mismatch.
noos_contract_readiness == 0	Local contract readiness is not active.	Stuck readiness.
High noos_contract_failure_rate	Calls are failing above baseline.	Failure spike.
Increasing noos_contract_fuel_exhaustions_total	Methods are exhausting fuel.	Fuel exhaustion.
Release stuck before completed	Release workflow needs operator review.	Publishing/release workflow.

Keep monitoring outputs with incident tickets. They are local evidence, not consensus state.

Production Alert Rules

Contract alert thresholds are available directly:

npm run noos -- contracts alerts --json
npm run noos -- contracts alerts <CONTRACT_ID> --json

Rule	Warning	Critical	Source
Contract failure rate	5% failed calls in latest 100-call sample	20% failed calls in latest 100-call sample	contract_call_metrics.status
Runtime errors	3 in 10 minutes	10 in 10 minutes	runtime_error_code and CONTRACT_RUNTIME_* failures
Fuel exhaustion	3 in 10 minutes	10 in 10 minutes	CONTRACT_EXECUTION_LIMIT_EXCEEDED failures
Readiness stuck	non-active for 5 minutes	non-active for 15 minutes	contract_readiness
Wasmtime sidecar unavailable	n/a	immediate	/node/contracts/runtime/preflight
Wasmtime sidecar mismatch	n/a	immediate	/node/contracts/runtime/preflight

Sidecar alerts are node-scoped. Failure-rate, runtime-error, fuel-exhaustion, and readiness alerts are contract-scoped.

Runbook: All Contract Calls Failing

Symptoms:

• every contract call fails;
• multiple contracts report runtime-related errors;
• monitoring shows sidecar unavailable or runtime mismatch.

Immediate containment:

1. Stop routing contract traffic to the affected node.
2. Keep consensus participation under the environment's validator policy, but do not admit contract traffic until runtime preflight is green.
3. Preserve monitoring output before restarting anything.

Commands:

npm run noos -- contracts runtime preflight --json
npm run noos -- contracts runtime activation --json
npm run noos -- contracts alerts --json
npm run noos -- production monitor-report --json

Likely causes:

• sidecar binary missing or not executable;
• sidecar path points to the wrong release;
• active runtime requirements changed but this node was not rolled;
• NOOS_CONTRACT_RUNTIME_PREFLIGHT was set to warn and the node admitted traffic despite mismatch;
• host CPU, memory, or process limits are preventing sidecar startup.

Repair:

1. Deploy the sidecar built from the matching NOOSChain release.
2. Set NOOS_CONTRACT_WASMTIME_SIDECAR_PATH to the correct absolute path.
3. Restart the node or sidecar supervisor path used by the deployment.
4. Re-run runtime and production preflight.
5. Re-admit traffic only after preflight and monitoring are green.

Runbook: Runtime Mismatch

Symptoms:

• calls fail with CONTRACT_RUNTIME_CAPABILITY_MISMATCH;
• readiness reports runtime mismatch;
• sidecar reports unexpected Wasmtime, executor, ABI, sidecar protocol, command, host import, or fuel-metering capability.

Triage:

npm run noos -- contracts runtime preflight --json
npm run noos -- contracts runtime provenance --json
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" "http://localhost:3000/node/observability/contracts/<CONTRACT_ID>/calls?status=failed&failureCode=CONTRACT_RUNTIME_CAPABILITY_MISMATCH&limit=20"

Repair:

1. Confirm the node release and sidecar binary came from the same release.
2. Check NOOS_CONTRACT_WASMTIME_SIDECAR_PATH.
3. Replace the sidecar with the matching binary.
4. Restart and run preflight.

If the desired runtime is intentionally different, stop. That is a protocol-pinned runtime upgrade requiring compatibility vectors, rollout policy, governance, and release planning.

Runbook: Sidecar Unavailable

Symptoms:

• runtime preflight reports unavailable sidecar;
• noos_contract_sidecar_available == 0;
• calls fail with CONTRACT_RUNTIME_UNAVAILABLE.

Triage:

npm run noos -- contracts runtime preflight --json
npm run noos -- production monitor-report --json

Checks:

• configured sidecar path exists;
• binary is executable by the node user;
• antivirus or service policy is not blocking process start;
• host has enough memory and process slots;
• sidecar can write any required temporary files.

Repair:

1. Drain contract traffic.
2. Fix path, permissions, or host limits.
3. Restart node or sidecar process.
4. Run runtime preflight and a low-risk call.

Runbook: Stuck Readiness

Symptoms:

• CALL_CONTRACT fails with CONTRACT_NOT_READY_ON_NODE;
• nested calls fail with CONTRACT_NESTED_BUCKET_AVAILABILITY_UNSATISFIED;
• readiness remains non-active after refresh.

Triage:

npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/payloads
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/buckets
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/bucket-access-rules
curl.exe http://localhost:3000/contracts/<CONTRACT_ID>/abi

Repair examples:

• If bucket metadata is missing, let the node sync before admitting calls.
• If ciphertext is missing and the method requires local encrypted payloads, run payload backfill or route calls to a node with payload availability.
• If the contract principal lacks bucket access, submit the appropriate bucket access-rule transaction.
• If a caller lacks access, grant the user, organization, or caller contract the required permission.
• If the manifest is wrong, publish a corrected package and migrate state.

Runbook: One Contract Failing

Symptoms:

• only one contract has high failure rate;
• one method started failing after a release or client change;
• runtime is healthy globally.

Triage:

curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/contracts
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" "http://localhost:3000/node/observability/contracts/<CONTRACT_ID>/calls?status=failed&limit=100"
curl.exe http://localhost:3000/contracts/<CONTRACT_ID>/abi

Classify the failures:

Failure code	Meaning	First action
CONTRACT_METHOD_FAILED	WASM returned non-zero status.	Check caller args, ABI, and recent client changes.
CONTRACT_EXECUTION_FAILED	WASM trapped or aborted.	Treat as contract bug until proven otherwise.
CONTRACT_EXECUTION_LIMIT_EXCEEDED	Fuel budget exhausted.	Check loops, argument size, and workload shape.
CONTRACT_HOST_CALL_LIMIT_EXCEEDED	Host-call budget exceeded.	Check repeated SDK calls and nested-call behavior.
CONTRACT_ARGS_TOO_LARGE	Encoded args exceeded protocol limit.	Fix caller payload size.
CONTRACT_EVENT_TOTAL_BYTES_LIMIT_EXCEEDED	Events exceeded byte budget.	Reduce event payload or split workflow.
CONTRACT_OUTBOUND_CALL_NOT_DECLARED	Caller manifest lacks outbound call edge.	Publish corrected caller manifest/version.
CONTRACT_NOT_ACTIVE	Contract was deactivated.	Route callers to replacement contract.

Repair:

1. Stop or fix bad callers if the issue is input-related.
2. Route away from nodes where readiness is false.
3. Ask contract owners for a fixed package when WASM behavior is faulty.
4. Plan migration if state must move to a replacement contract.
5. Verify monitor report, chain verify, and replay verify after repair.

Runbook: Fuel Exhaustion Spike

Symptoms:

• CONTRACT_EXECUTION_LIMIT_EXCEEDED appears repeatedly;
• noos_contract_fuel_exhaustions_total increases;
• failures correlate with large inputs or a new contract version.

Triage:

npm run noos -- contracts alerts <CONTRACT_ID> --json
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" "http://localhost:3000/node/observability/contracts/<CONTRACT_ID>/calls?status=failed&failureCode=CONTRACT_EXECUTION_LIMIT_EXCEEDED&limit=50"

Containment:

• rate-limit or stop the offending callers;
• route traffic to a safe replacement method if available;
• do not increase protocol fuel limits as an operator workaround.

Repair usually belongs to the contract owner: reduce loops, bound input sizes, split work across calls, or publish a new version. Operators verify that failure rates return to baseline.

Runbook: WASM Trap Spike

Symptoms:

• CONTRACT_EXECUTION_FAILED increases;
• failures cluster around one package version, method, or argument shape;
• runtime and sidecar preflight are green.

Triage:

curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" "http://localhost:3000/node/observability/contracts/<CONTRACT_ID>/calls?status=failed&failureCode=CONTRACT_EXECUTION_FAILED&limit=50"
npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json

Response:

1. Preserve failed transaction hashes, method names, args hashes if available, block height range, code hash, manifest hash, and descriptor hash.
2. Stop automated callers if they are amplifying the trap.
3. Ask contract owners to reproduce with the same package and inputs.
4. Publish a fixed version and migrate state if needed.
5. Deactivate the unsafe contract if continued calls are dangerous.

Contract Deactivation

Use this when a deployed contract should stop receiving production calls because it is vulnerable, deprecated, misconfigured, or failing dangerously.

Deactivation is a consensus transaction. Do not manually edit the contracts table on a live production chain.

Immediate containment:

1. Stop external clients and jobs that call the contract.
2. If registry enforcement is enabled, suspend or retire the package registry entry so new instantiations are blocked.
3. Identify contract-to-contract callers through manifest calls declarations.
4. Publish or select a replacement contract if users need service continuity.
5. Create a migration plan if state must move.

Before submitting transaction DEACTIVATE_CONTRACT, collect:

• contract id;
• code hash;
• manifest hash;
• package descriptor hash;
• current registry status;
• active callers and dependent contracts;
• pending migration jobs;
• recent failed transaction hashes;
• replacement contract id, if any.

Dependency checks:

npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json
npm run noos -- contracts deactivation-dependencies <CONTRACT_ID> --include-recent-calls --json
npm run noos -- contracts deactivation-dependencies <CONTRACT_ID> --policy-check --block-registry --require-replacement --replacement-contract-id <REPLACEMENT_CONTRACT_ID> --json

Submit transaction DEACTIVATE_CONTRACT:

npm run noos -- tx build-and-submit --type DEACTIVATE_CONTRACT --payload-json "{\"contractId\":\"<CONTRACT_ID>\",\"reason\":\"retired vulnerable package\",\"metadata\":{\"incidentId\":\"<INCIDENT_ID>\",\"replacementContractId\":\"<REPLACEMENT_CONTRACT_ID>\"}}" --signer-public-key "<ADMIN_PUBLIC_KEY_PEM>" --signer-private-key-path "<ADMIN_PRIVATE_KEY_PATH>" --yes

Expected behavior after deactivation:

• direct calls fail with CONTRACT_NOT_ACTIVE;
• nested calls targeting the contract fail before target WASM runs;
• no new state writes or events are produced by the inactive contract;
• replacement contract calls execute successfully;
• registry, package docs, and incident notes reference the replacement.

Emergency override should be reserved for incident response. It only works when the deterministic chain policy permits it, and the transaction must include overrideDependencyPolicy: true plus incidentId or changeControlId metadata.

Example: Enable Runtime On A New Validator

1. Configure the node:

   $env:NOOS_SMART_CONTRACTS_ENABLED="true"
   $env:NOOS_CONTRACT_RUNTIME_PREFLIGHT="required"
   $env:NOOS_CONTRACT_WASMTIME_SIDECAR_PATH="C:\nooschain\bin\noos-contract-executor-wasmtime.exe"
   $env:NOOS_OPERATOR_TOKEN="<operator-token>"

2. Run local preflight:

   npm run noos -- contracts runtime preflight --json
   npm run noos -- production preflight --profile validator

3. Check activation:

   npm run noos -- contracts runtime activation --json

4. If activation is already active, route contract traffic only after preflight is green. If activation is not active, follow the activation gate flow before expecting calls to succeed.

Example: Activation Blocked By One Validator

Symptoms:

• rollout policy exists;
• most validators attest;
• activation transaction fails because one active validator is missing or has a different requirements hash.

Actions:

npm run noos -- contracts runtime rollout --json
npm run noos -- contracts runtime attestations --rollout-policy-id contract-runtime-rollout-v1 --json

Fix the validator:

1. Deploy the matching sidecar.
2. Run runtime preflight.
3. Submit transaction SUBMIT_CONTRACT_RUNTIME_ROLLOUT_ATTESTATION with the correct hashes.
4. Retry transaction SET_CONTRACT_RUNTIME_ACTIVATION.

Do not lower rollout requirements just to make a broken validator pass unless change control explicitly removes that validator from the activation scope.

Example: Sidecar Path Misconfigured

Symptoms:

• preflight says the sidecar is unavailable;
• NOOS_CONTRACT_WASMTIME_SIDECAR_PATH points to a missing file;
• every call fails with CONTRACT_RUNTIME_UNAVAILABLE.

Repair:

$env:NOOS_CONTRACT_WASMTIME_SIDECAR_PATH="C:\nooschain\bin\noos-contract-executor-wasmtime.exe"
npm run noos -- contracts runtime preflight --json
npm run noos -- production preflight --profile validator

If the path is managed by service configuration, update the service env file, restart the node, and re-run the same checks.

Example: Missing Encrypted Payloads On One Node

Symptoms:

• contract readiness is active on node A but not node B;
• node B readiness mentions encrypted_payload_local;
• nested calls fail on node B with CONTRACT_NESTED_BUCKET_AVAILABILITY_UNSATISFIED.

Actions:

npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/payloads
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/buckets

Route calls to ready nodes while payload backfill or reconciliation runs. Do not submit a contract migration or package rollback for a local payload availability problem.

Example: Post-Restore Runtime Verification

After restoring a node, verify the chain first, then local contract runtime:

npm run noos -- chain verify
npm run noos -- chain replay-verify
npm run noos -- contracts runtime preflight --json
npm run noos -- contracts monitoring snapshot --json
npm run noos -- production preflight --profile validator

If chain replay passes but contract readiness is red, treat it as a local runtime, bucket, permission, or payload issue. Keep contract traffic away from the node until readiness is repaired.

Evidence Bundle

For any production smart-contract incident, archive:

• production monitor-report --json;
• contracts runtime preflight --json;
• contracts runtime activation --json;
• contracts monitoring snapshot --json;
• contracts monitoring alerts --json;
• /node/observability/contracts;
• /node/observability/contracts/<CONTRACT_ID>;
• recent /node/observability/contracts/<CONTRACT_ID>/calls;
• affected transaction hashes;
• contract ABI response;
• package descriptor and signatures;
• code hash, manifest hash, descriptor hash;
• bucket access and payload availability responses;
• sidecar binary path and provenance output;
• node release version and activation/rollout policy ids.

This evidence is enough to distinguish runtime failure, runtime mismatch, package governance, bucket availability, caller input, and contract-code bugs.

What Not To Do

• Do not manually edit contract, activation, rollout, registry, or package rows on a live chain.
• Do not bypass runtime mismatch by changing local runtime requirements.
• Do not increase fuel limits as an operator workaround for a failing contract.
• Do not route production calls to a node with red runtime preflight.
• Do not treat local readiness as consensus state.
• Do not deactivate a contract without checking dependent callers unless this is an approved emergency.
• Do not resolve an incident before replay, production preflight, and contract monitoring are green.