Smart Contract Operations Runbooks

These runbooks are for production operators responding to smart-contract runtime and contract-health incidents. They assume the node uses the Rust Wasmtime contract sidecar and the operator observability API is protected by NOOS_OPERATOR_TOKEN.

Start every incident by preserving context:

npm run noos -- production monitor-report --json
npm run noos -- node health --json

npm run noos -- production monitor-report --json
npm run noos -- node health --json

Keep the output with the incident ticket. It gives the chain head, node health, sidecar status, payload availability, and contract metrics at the time of triage.

Production Rollout Rehearsal

Prod-SC2.7 adds a copy-pasteable rehearsal for the production smart-contract rollout path. Run it before enabling a new runtime policy, registry policy, or contract release workflow in an environment:

npm run test:contract-production-rollout-rehearsal

npm run test:contract-production-rollout-rehearsal

The rehearsal covers package publish and registry enforcement, instantiation, bucket readiness failure and repair, successful calls, indexed events, structured return values, per-contract state writes, C2C, state migration, runtime capability mismatch detection, trap rollback, replay verification, and chain verification. A failure means the environment is not ready for production contract rollout until the failed stage is repaired and the rehearsal passes again.

The C2C stage runs the dedicated test:contract-to-contract-calls gate from inside the rehearsal so nested-call behavior stays pinned to the focused C2C suite while the rollout database remains replay-stable.

Multi-Node Production Rehearsal

Prod-SC3.4 extends the production rehearsal across two validator databases. Run it before treating a contract release process as network-ready:

npm run test:contract-production-multinode-rehearsal

npm run test:contract-production-multinode-rehearsal

The rehearsal performs the production path on node A, syncs node B from node A, and verifies both nodes converge. It covers runtime activation, package approval, v1 deployment and call, v2 release workflow, migration, source deactivation, rollback recording, release events, contract events, call results, replay verification, and chain verification.

Passing output means:

1. Node B executed the synced blocks and reached the same head hash and state root as node A.
2. Contract state, release status, release events, call results, migration output, source deactivation, and target activity match across nodes.
3. Operator APIs on the synced node can read the release plan, emitted contract event, and call result.
4. Replay succeeds independently on both nodes.

Local readiness and payload availability are still operator-local surfaces. They can appear differently in each node's plan, but they must not change the canonical block/state-root result.

External Monitoring Integration

Prod-SC3.5 exposes contract monitoring in formats that external systems can poll without screen-scraping CLI output.

Step 1: Set The Operator Token

$env:NOOS_OPERATOR_TOKEN="<operator-token>"

export NOOS_OPERATOR_TOKEN="<operator-token>"

Step 2: Check The JSON Snapshot

npm run noos -- contracts monitoring snapshot --json

npm run noos -- contracts monitoring snapshot --json

This is the easiest payload for the Web console or a custom dashboard. It contains runtime activation, preflight status, contract summaries, readiness, alerts, and release workflow status.

Step 3: Check The Alert Feed

npm run noos -- contracts monitoring alerts --json

npm run noos -- contracts monitoring alerts --json

Use this feed for alert-manager style integrations. Page on critical, create tickets on repeated warning, and keep details with the incident evidence.

Step 4: Scrape Prometheus Metrics

npm run noos -- contracts monitoring prometheus

npm run noos -- contracts monitoring prometheus

The HTTP endpoint is:

GET /metrics/contracts

Scrape it with the same operator bearer token used for other protected operator APIs. The exposition includes call totals, failures, failure rate, runtime errors, fuel exhaustion, fuel used, host calls, events, writes, readiness, runtime activation, sidecar availability/mismatch, and release status.

Step 5: Wire Alerts To Runbooks

Map the exported labels to runbook actions:

Metric or alert	First runbook
noos_contract_sidecar_available == 0	Sidecar rollout or runtime mismatch
noos_contract_sidecar_mismatch == 1	Runtime mismatch
noos_contract_readiness == 0	Stuck contract readiness
high noos_contract_failure_rate	Contract failure spikes
increasing noos_contract_fuel_exhaustions_total	Contract failure spikes
release status stuck before completed	Contract release workflow

Keep /monitoring/contracts and /metrics/contracts outputs with the incident ticket. They are local operator evidence, not consensus state.

Runtime Activation Gate

Prod-SC3.1 makes production smart-contract execution an explicit governance switch. Operators can stage registry entries while the runtime is disabled, but code deployment, contract instantiation, and contract calls are rejected unless the runtime activation status is active.

Check local preflight first:

$env:NOOS_OPERATOR_TOKEN="<operator-token>"
npm run noos -- contracts runtime preflight --json
npm run noos -- contracts runtime activation --json

export NOOS_OPERATOR_TOKEN="<operator-token>"
npm run noos -- contracts runtime preflight --json
npm run noos -- contracts runtime activation --json

Then submit a governance transaction of type SET_CONTRACT_RUNTIME_ACTIVATION. The activation payload must pin the protocol-required runtime requirements and include a successful preflight attestation when moving to active:

{
  "status": "active",
  "runtime": "wasm-assemblyscript-v1",
  "protocolVersion": 1,
  "activationHeight": "12345",
  "requirements": {
    "sidecarProtocolVersion": 2,
    "abiVersion": "noos-contract-abi-v2",
    "executorVersion": "0.1.0",
    "wasmtimeVersion": "26.0.1",
    "requiredCommands": ["handshake", "validate", "call"],
    "requiredHostImports": [
      "noos.storage_get",
      "noos.storage_set",
      "noos.bucket_get_metadata",
      "noos.bucket_get_record_metadata",
      "noos.bucket_add_encrypted_record_metadata",
      "noos.emit_event",
      "noos.return_value",
      "noos.last_result_ptr",
      "noos.last_result_len",
      "noos.contract_call",
      "env.abort"
    ],
    "requireFuelMetering": true
  },
  "requirementsHash": "<sha256-canonical-requirements>",
  "preflightAttestation": {
    "ok": true,
    "requirementsHash": "<sha256-canonical-requirements>",
    "checkedAt": "2026-06-01T00:00:00.000Z",
    "nodeId": "node-validator-1"
  },
  "reason": "Enable production smart contracts after preflight",
  "metadata": {
    "changeControlId": "CHG-1234"
  }
}

To pause smart-contract execution, submit the same transaction with status: "disabled" or status: "preflight_required". Registry operations remain available so packages can be reviewed and staged before reactivation.

Runtime Rollout Policy

Prod-SC3.2 adds the production rollout guard that should be used before moving the activation gate to active. The rollout policy pins the protocol runtime requirements hash and a contract policy hash; validator attestations then prove which active validators have checked the same rollout.

Inspect rollout state:

$env:NOOS_OPERATOR_TOKEN="<operator-token>"
npm run noos -- contracts runtime rollout --json
npm run noos -- contracts runtime attestations --rollout-policy-id contract-runtime-rollout-v1 --json

export NOOS_OPERATOR_TOKEN="<operator-token>"
npm run noos -- contracts runtime rollout --json
npm run noos -- contracts runtime attestations --rollout-policy-id contract-runtime-rollout-v1 --json

Submit a rollout policy with REGISTER_CONTRACT_RUNTIME_ROLLOUT_POLICY:

{
  "id": "contract-runtime-rollout-v1",
  "status": "active",
  "runtime": "wasm-assemblyscript-v1",
  "protocolVersion": 1,
  "requirements": {
    "sidecarProtocolVersion": 2,
    "abiVersion": "noos-contract-abi-v2",
    "executorVersion": "0.1.0",
    "wasmtimeVersion": "26.0.1",
    "requiredCommands": ["handshake", "validate", "call"],
    "requiredHostImports": [
      "noos.storage_get",
      "noos.storage_set",
      "noos.bucket_get_metadata",
      "noos.bucket_get_record_metadata",
      "noos.bucket_add_encrypted_record_metadata",
      "noos.emit_event",
      "noos.return_value",
      "noos.last_result_ptr",
      "noos.last_result_len",
      "noos.contract_call",
      "env.abort"
    ],
    "requireFuelMetering": true
  },
  "requirementsHash": "<sha256-canonical-requirements>",
  "policyHash": "<sha256-canonical-contract-policy>",
  "metadata": {
    "changeControlId": "CHG-1234"
  }
}

Each validator should run local preflight and submit SUBMIT_CONTRACT_RUNTIME_ROLLOUT_ATTESTATION:

{
  "id": "attest-node-validator-1-runtime-v1",
  "nodeId": "node-validator-1",
  "rolloutPolicyId": "contract-runtime-rollout-v1",
  "requirementsHash": "<sha256-canonical-requirements>",
  "policyHash": "<sha256-canonical-contract-policy>",
  "sidecarProtocolVersion": 2,
  "executorVersion": "0.1.0",
  "wasmtimeVersion": "26.0.1",
  "abiVersion": "noos-contract-abi-v2",
  "preflightOk": true,
  "preflightHash": "<sha256-canonical-preflight-summary>",
  "metadata": {
    "operator": "validator-1"
  }
}

Finally, activate with rollout enforcement by adding these fields to SET_CONTRACT_RUNTIME_ACTIVATION:

{
  "rolloutPolicyId": "contract-runtime-rollout-v1",
  "requireAllActiveValidatorsAttested": true,
  "minValidatorAttestations": 1
}

If a validator is missing or reports different hashes, activation fails with a stable rollout failure code and no contract deployment/call gate is opened.

Contract Release Workflow

Prod-SC3.3 turns package approval, deployment, migration, deactivation, and rollback into one operator-visible release workflow.

Start by creating the release payload:

npm run noos -- contracts release create-payload --id release-report-store-0.2.0 --release-type upgrade --package-registry-id pkg-report-store-0.2.0 --package-name report-store --package-version 0.2.0 --code-hash "<V2_CODE_HASH>" --manifest-hash "<V2_MANIFEST_HASH>" --descriptor-hash "<V2_DESCRIPTOR_HASH>" --source-contract-id contract-report-store --target-contract-id contract-report-store-v2 --audit-metadata-json '{"audit":"internal-pass"}' --approval-metadata-json '{"changeControlId":"CHG-1234"}' --metadata-json '{"operator":"release-team"}' --out release-report-store-0.2.0.json

npm run noos -- contracts release create-payload --id release-report-store-0.2.0 --release-type upgrade --package-registry-id pkg-report-store-0.2.0 --package-name report-store --package-version 0.2.0 --code-hash "<V2_CODE_HASH>" --manifest-hash "<V2_MANIFEST_HASH>" --descriptor-hash "<V2_DESCRIPTOR_HASH>" --source-contract-id contract-report-store --target-contract-id contract-report-store-v2 --audit-metadata-json '{"audit":"internal-pass"}' --approval-metadata-json '{"changeControlId":"CHG-1234"}' --metadata-json '{"operator":"release-team"}' --out release-report-store-0.2.0.json

Submit it:

npm run noos -- tx build-and-submit --type CREATE_CONTRACT_RELEASE --payload-file release-report-store-0.2.0.json --signer-public-key "<admin-public-key>" --signer-private-key-path ".secrets/admin.private.pem" --yes

Check the plan after each stage:

$env:NOOS_OPERATOR_TOKEN="<operator-token>"
npm run noos -- contracts release plan --id release-report-store-0.2.0 --json
npm run noos -- contracts release events --id release-report-store-0.2.0 --json

Advance stages with explicit governance transactions:

npm run noos -- contracts release advance-payload --id release-report-store-0.2.0 --expected-current-status draft --next-status package_approved --reason "Registry package approved" --out release-advance.json
npm run noos -- tx build-and-submit --type ADVANCE_CONTRACT_RELEASE --payload-file release-advance.json --signer-public-key "<admin-public-key>" --signer-private-key-path ".secrets/admin.private.pem" --yes

Use this order for an upgrade:

1. package_approved after the registry entry is approved and audit metadata is present.
2. deployed after DEPLOY_CONTRACT_CODE stores the exact codeHash.
3. instantiated after INSTANTIATE_CONTRACT creates the target contract with the exact codeHash and manifestHash.
4. readiness_checked after the operator plan shows local readiness is active on the node that will execute calls.
5. migration_planned after the migration proposal/job is ready.
6. migration_completed after the migration job is finalized or the migration receipt exists.
7. traffic_ready after callers are ready to move to the target contract.
8. source_deactivated after DEACTIVATE_CONTRACT marks the source inactive.
9. completed after the release has no remaining blockers.

For a new contract with no source contract, skip migration and source deactivation: go from readiness_checked to traffic_ready, then completed.

Rollback is also explicit:

npm run noos -- contracts release rollback-payload --id release-report-store-0.2.0 --expected-current-status completed --reason "Rollback after production validation" --rollback-target-release-id release-report-store-0.1.0 --out release-rollback.json
npm run noos -- tx build-and-submit --type ROLLBACK_CONTRACT_RELEASE --payload-file release-rollback.json --signer-public-key "<admin-public-key>" --signer-private-key-path ".secrets/admin.private.pem" --yes

Run ROLLBACK_CONTRACT_RELEASE once to move to rollback_started, and again from rollback_started to rolled_back after the rollback actions are complete. If state moved forward during the failed release, create and finalize a reverse migration before marking rollback complete.

The release transition checks use only consensus state. Local readiness, payload availability, and operator observations appear in contracts release plan, but they are not hidden replay inputs.

Quick Decision Tree

1. Are all contract calls failing?
   • Check runtime sidecar rollout and runtime mismatch first.
2. Is only one contract failing?
   • Check its call metrics, recent failure codes, readiness, ABI, package registry status, and dependent callers.
3. Are failures readiness-related?
   • Follow the stuck readiness runbook and confirm bucket payload availability.
4. Are failures execution-related?
   • Classify by failure code, method, block range, and runtime error message.
5. Is the contract unsafe to keep callable?
   • Suspend the package registry entry when registry policy is enforced, stop dependent traffic, publish a fixed version, and plan migration. Do not manually edit contract rows while the chain is live.

Production Alert Rules

Prod-SC2.5 defines formal contract alert thresholds that operators can poll directly:

npm run noos -- contracts alerts --json
npm run noos -- contracts alerts <CONTRACT_ID> --json

npm run noos -- contracts alerts --json
npm run noos -- contracts alerts <CONTRACT_ID> --json

Rule	Warning	Critical	Source
Contract failure rate	5% failed calls in latest 100-call sample	20% failed calls in latest 100-call sample	contract_call_metrics.status
Runtime errors	3 in 10 minutes	10 in 10 minutes	runtime_error_code and CONTRACT_RUNTIME_* failures
Fuel exhaustion	3 in 10 minutes	10 in 10 minutes	CONTRACT_EXECUTION_LIMIT_EXCEEDED failures
Readiness stuck	non-active for 5 minutes	non-active for 15 minutes	contract_readiness
Wasmtime sidecar unavailable	n/a	immediate	/node/contracts/runtime/preflight
Wasmtime sidecar mismatch	n/a	immediate	/node/contracts/runtime/preflight

Each rule returns status: "ok" or status: "firing", severity, details, and a recommended operator action. Sidecar alerts are node-scoped; failure-rate, runtime-error, fuel-exhaustion, and readiness alerts are contract-scoped.

Useful Reads

Use these reads before mutating anything:

$env:NOOS_OPERATOR_TOKEN="<operator-token>"
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/contracts
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/contracts/<CONTRACT_ID>
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" "http://localhost:3000/node/observability/contracts/<CONTRACT_ID>/calls?status=failed&limit=50"
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/contracts/alerts
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/contracts/<CONTRACT_ID>/alerts
curl.exe http://localhost:3000/contracts/<CONTRACT_ID>/abi
curl.exe "http://localhost:3000/contracts/<CONTRACT_ID>/events?limit=50&order=desc"

export NOOS_OPERATOR_TOKEN="<operator-token>"
curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/contracts
curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/contracts/<CONTRACT_ID>
curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" "http://localhost:3000/node/observability/contracts/<CONTRACT_ID>/calls?status=failed&limit=50"
curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" http://localhost:3000/node/contracts/alerts
curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" http://localhost:3000/contracts/<CONTRACT_ID>/alerts
curl http://localhost:3000/contracts/<CONTRACT_ID>/abi
curl "http://localhost:3000/contracts/<CONTRACT_ID>/events?limit=50&order=desc"

For payload and bucket availability:

curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/payloads
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/buckets
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/bucket-access-rules

curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/payloads
curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/buckets
curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/bucket-access-rules

Sidecar Rollout

Use this when deploying a new noos-contract-executor-wasmtime binary.

Before Rollout

1. Build and verify the binary from the release commit:

npm run build:contract-wasmtime-sidecar
npm run test:contract-wasmtime-sidecar
npm run test:contract-wasmtime-sidecar-lifecycle
npm run test:contract-runtime-compatibility

npm run build:contract-wasmtime-sidecar
npm run test:contract-wasmtime-sidecar
npm run test:contract-wasmtime-sidecar-lifecycle
npm run test:contract-runtime-compatibility

2. Confirm the expected runtime requirements in the release notes or in src/contracts/runtime-requirements.ts.
3. Confirm the sidecar binary path configured for the node. If NOOS_CONTRACT_WASMTIME_SIDECAR_PATH is unset, the node uses the bundled release binary under external/noos-contract-executor-wasmtime/target/release.
4. Run production preflight before admitting the node:

npm run noos -- production preflight --profile validator

npm run noos -- production preflight --profile validator

Rollout Sequence

1. Drain external traffic from one node.
2. Stop the node process cleanly.
3. Replace the sidecar binary or update NOOS_CONTRACT_WASMTIME_SIDECAR_PATH.
4. Start the node.
5. Run:

npm run noos -- production monitor-report --json
npm run noos -- production preflight --profile validator

npm run noos -- production monitor-report --json
npm run noos -- production preflight --profile validator

6. Exercise a known low-risk contract call or wait for normal traffic.
7. Watch /node/observability/contracts for failures, runtime errors, fuel use, and host-call counts.
8. Roll to the next node only after the first node is stable.

Rollback

Rollback is a binary/config rollback, not a protocol relaxation.

1. Drain traffic.
2. Restore the previous sidecar binary and path.
3. Restart the node.
4. Re-run production preflight.
5. Verify failed-call rates return to baseline.

Do not bypass CONTRACT_RUNTIME_CAPABILITY_MISMATCH by editing local runtime requirements. That changes consensus execution semantics and can break replay.

Runtime Mismatch

Symptoms:

• calls fail with CONTRACT_RUNTIME_CAPABILITY_MISMATCH
• readiness reports runtime mismatch
• the sidecar reports an unexpected Wasmtime, executor, ABI, protocol, command, host import, or fuel-metering capability

Triage:

1. Run the contract runtime production preflight:

npm run noos -- contracts runtime preflight --json

npm run noos -- contracts runtime preflight --json

The same result is available to operator tooling:

GET /node/contracts/runtime/preflight

For production smart-contract nodes, set:

NOOS_SMART_CONTRACTS_ENABLED=true
NOOS_CONTRACT_RUNTIME_PREFLIGHT=required
NOOS_CONTRACT_WASMTIME_SIDECAR_PATH=<absolute path to noos-contract-executor-wasmtime>

With required, startup refuses to continue if the sidecar is missing, cannot start, lacks fuel metering, or reports capabilities that do not match the active protocol requirements.

2. Collect the exact failure:

curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" "http://localhost:3000/node/observability/contracts/<CONTRACT_ID>/calls?status=failed&failureCode=CONTRACT_RUNTIME_CAPABILITY_MISMATCH&limit=20"

curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" "http://localhost:3000/node/observability/contracts/<CONTRACT_ID>/calls?status=failed&failureCode=CONTRACT_RUNTIME_CAPABILITY_MISMATCH&limit=20"

3. Confirm the node release and sidecar binary came from the same release.
4. Check NOOS_CONTRACT_WASMTIME_SIDECAR_PATH.
5. Rebuild or redeploy the matching sidecar.
6. Restart the node and run production preflight.

If the desired runtime is intentionally different, this is not an operator repair. It requires a protocol-pinned runtime upgrade, compatibility vectors, docs, tests, and governance/rollout planning.

Stuck Contract Readiness

Symptoms:

• CALL_CONTRACT fails with CONTRACT_NOT_READY_ON_NODE
• nested calls fail with CONTRACT_NESTED_BUCKET_AVAILABILITY_UNSATISFIED
• readiness details mention missing bucket metadata, missing access, missing payloads, or runtime mismatch

Triage:

1. Read contract ABI and manifest-facing requirements:

npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json

npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json

The HTTP equivalent is:

GET /contracts/<CONTRACT_ID>/readiness?refresh=true

The readiness response includes callable, contractStatus, status, reason, details, checkedAt, and source. If callable is false, use the reason and details fields as the first remediation target.

curl.exe http://localhost:3000/contracts/<CONTRACT_ID>/abi

curl http://localhost:3000/contracts/<CONTRACT_ID>/abi

2. Check local payload and bucket observability:

curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/payloads
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/buckets

curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/payloads
curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/buckets

3. Check contract and caller access rules:

curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/bucket-access-rules

curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/bucket-access-rules

Common causes and fixes:

Cause	Fix
Manifest references a bucket missing on this node	Sync chain state before admitting calls.
Contract principal lacks bucket permission	Submit an authorized bucket access rule transaction for the contract principal.
Caller lacks required bucket permission	Grant the user/organization/contract caller the required bucket permission.
Method requires encrypted_payload_local but ciphertext is missing	Run payload backfill/reconciliation or route calls to a node that has the payloads.
Runtime readiness reports mismatch	Follow the runtime mismatch runbook.
Manifest is wrong	Publish a new contract version with a corrected manifest and migrate state if needed.

Nested calls use the target contract's readiness. A healthy caller can still fail if the target's bucket ciphertext is unavailable on the local node.

Contract Failure Spikes

Symptoms:

• /node/observability/contracts shows increased failures
• a method's failure rate changes suddenly
• runtime errors cluster around one block range or deployment

Triage:

1. Identify scope:

curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/contracts
curl.exe -H "Authorization: Bearer $env:NOOS_OPERATOR_TOKEN" "http://localhost:3000/node/observability/contracts/<CONTRACT_ID>/calls?status=failed&limit=100"

curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" http://localhost:3000/node/observability/contracts
curl -H "Authorization: Bearer $NOOS_OPERATOR_TOKEN" "http://localhost:3000/node/observability/contracts/<CONTRACT_ID>/calls?status=failed&limit=100"

2. Group by failure code and method.
3. Pull affected transaction details:

curl.exe http://localhost:3000/transactions/<TX_HASH>

curl http://localhost:3000/transactions/<TX_HASH>

4. Preserve package descriptor, code hash, manifest hash, contract id, method, block height range, and failed transaction hashes.

Common classifications:

Failure code	Meaning	First action
CONTRACT_METHOD_FAILED	WASM returned non-zero status.	Check caller args, ABI, and recent client changes.
CONTRACT_EXECUTION_FAILED	WASM trapped or aborted.	Treat as contract bug until proven otherwise.
CONTRACT_EXECUTION_LIMIT_EXCEEDED	Wasmtime fuel budget was exhausted.	Check loops, args size, and recent workload shape.
CONTRACT_HOST_CALL_LIMIT_EXCEEDED	Host call count exceeded protocol limit.	Check method behavior and repeated SDK calls.
CONTRACT_ARGS_TOO_LARGE	Encoded args exceeded protocol limit.	Fix caller payload size.
CONTRACT_EVENT_TOTAL_BYTES_LIMIT_EXCEEDED	Events exceeded protocol byte budget.	Reduce emitted data or split workflow.
CONTRACT_NOT_READY_ON_NODE	Local readiness failed.	Follow stuck readiness runbook.
CONTRACT_OUTBOUND_CALL_NOT_DECLARED	Caller manifest lacks outbound call edge.	Publish corrected caller manifest/version.
CONTRACT_NESTED_BUCKET_AVAILABILITY_UNSATISFIED	Target bucket ciphertext is not available locally.	Backfill payloads or route to ready nodes.
CONTRACT_RUNTIME_UNAVAILABLE	Sidecar missing, stopped, or not executable.	Follow sidecar rollout/runtime repair.
CONTRACT_RUNTIME_CAPABILITY_MISMATCH	Sidecar does not match protocol-pinned requirements.	Follow runtime mismatch runbook.

Response:

1. If failures are client-caused, stop or fix callers.
2. If failures are readiness-caused, repair node readiness or route traffic.
3. If failures are contract-code-caused, publish a fixed version and plan state migration.
4. If the contract is unsafe, move to the deactivation runbook.
5. After repair, verify replay and chain health:

npm run noos -- production monitor-report --json

npm run noos -- production monitor-report --json

Contract Deactivation

Use this when a deployed contract should stop receiving production calls because it is vulnerable, deprecated, misconfigured, or failing dangerously.

Deactivation is a first-class consensus transaction. Do not manually edit the contracts table on a live production chain. Manual database edits are not a consensus operation and can break replay, state roots, and other nodes.

Immediate Containment

1. Stop external clients and jobs that submit calls to the contract.
2. If registry enforcement is enabled, suspend or retire the package registry entry so new instantiations are blocked.
3. If the contract is a target of contract-to-contract calls, identify caller manifests through their calls declarations and stop those entrypoints too.
4. Publish a replacement contract version if users need continued service.
5. Create a migration plan if state must move to the replacement.

Pre-Deactivation Checklist

Before submitting DEACTIVATE_CONTRACT, collect:

• contract id
• code hash
• manifest hash
• package descriptor hash
• current registry status
• active callers and dependent contracts
• pending migration jobs
• recent failed transaction hashes
• target replacement contract id, if any

Run the dependency inspector and archive the result:

npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json
npm run noos -- contracts deactivation-dependencies <CONTRACT_ID> --include-recent-calls --json
npm run noos -- contracts deactivation-dependencies <CONTRACT_ID> --policy-check --block-registry --require-replacement --replacement-contract-id <REPLACEMENT_CONTRACT_ID> --json

npm run noos -- contracts readiness <CONTRACT_ID> --refresh --json
npm run noos -- contracts deactivation-dependencies <CONTRACT_ID> --include-recent-calls --json
npm run noos -- contracts deactivation-dependencies <CONTRACT_ID> --policy-check --block-registry --require-replacement --replacement-contract-id <REPLACEMENT_CONTRACT_ID> --json

Treat risk: "high" or non-empty blockingWarnings as a change-control stop: coordinate caller contracts, finish or cancel active migration jobs, and review registry status before submitting the deactivation transaction.

If the chain enables enforced deactivation dependency policy, also treat policy.wouldBlockDeactivation: true as a consensus failure preview. The deactivation transaction will fail until the blocking state is fixed or an explicit emergency override is used under policy.

Submit Deactivation

Use an account with chain:admin:

npm run noos -- tx build-and-submit --type DEACTIVATE_CONTRACT --payload-json "{\"contractId\":\"<CONTRACT_ID>\",\"reason\":\"retired vulnerable package\",\"metadata\":{\"incidentId\":\"<INCIDENT_ID>\",\"replacementContractId\":\"<REPLACEMENT_CONTRACT_ID>\"}}" --signer-public-key "<ADMIN_PUBLIC_KEY_PEM>" --signer-private-key-path "<ADMIN_PRIVATE_KEY_PATH>" --yes

npm run noos -- tx build-and-submit --type DEACTIVATE_CONTRACT --payload-json '{"contractId":"<CONTRACT_ID>","reason":"retired vulnerable package","metadata":{"incidentId":"<INCIDENT_ID>","replacementContractId":"<REPLACEMENT_CONTRACT_ID>"}}' --signer-public-key "<ADMIN_PUBLIC_KEY_PEM>" --signer-private-key-path "<ADMIN_PRIVATE_KEY_PATH>" --yes

SDK equivalent:

await client.contracts.deactivate({
  signer,
  nonce: "auto",
  submit: true,
  contractId: "<CONTRACT_ID>",
  reason: "retired vulnerable package",
  metadata: { incidentId: "<INCIDENT_ID>", replacementContractId: "<REPLACEMENT_CONTRACT_ID>" },
});

Emergency override should be reserved for incident response. It only works when the deterministic chain policy permits it, and the transaction must include overrideDependencyPolicy: true plus incidentId or changeControlId metadata.

Expected Post-Deactivation Behavior

After the deactivation transaction executes, verify:

• direct calls fail with CONTRACT_NOT_ACTIVE
• nested calls targeting the contract fail before target WASM runs
• no new state writes or events are produced by the inactive contract
• replacement contract calls execute successfully
• registry, package docs, and operator incident notes reference the replacement

Evidence Bundle

For any production smart-contract incident, archive:

• production monitor-report --json
• /node/observability/contracts
• /node/observability/contracts/<CONTRACT_ID>
• recent /node/observability/contracts/<CONTRACT_ID>/calls
• affected transaction hashes
• contract ABI response
• package descriptor and signatures
• code hash, manifest hash, descriptor hash
• relevant bucket access and payload availability responses
• sidecar binary version and configured path

This is the minimum bundle needed to reconstruct whether the incident was runtime, manifest, caller, bucket availability, package governance, or contract code.