Join our community of builders on Discord!

Drain & Graceful Exit

Taking a Lightchain AI worker offline is not the same as stopping its container. The worker has obligations on-chain — jobs it already completed carry a dispute window during which the disputer can still raise a challenge — and the on-chain registry refuses to release your staked LCAI while any of those obligations remain. Calling deregister before that settles reverts with ActiveJobsExist and leaves the stake locked. This page is the operator runbook for shutting a worker down cleanly: stop it, wait out the dispute window, settle its jobs with release, then deregister. It applies to both testnet and mainnet — substitute the env values for your network.

Why the order matters

WorkerRegistry.deregisterWorker() reverts with ActiveJobsExist whenever getActiveJobCount(worker) > 0. After every completed job, the worker stays "active" for the on-chain dispute window (a parameter on AIConfig, which can be set per model). During that window the disputer can still raise a challenge, so the contract holds the slot open. Once the dispute window passes, a call to JobRegistry.releaseJobs(...) settles the job: it decrements activeJobsCount and credits the fee to the worker's balance in JobRegistry. A running worker sends that call itself — its release scheduler probes every 5 minutes (RELEASE_PROBE_INTERVAL) and runs a full cycle within RELEASE_INTERVAL (default 8 h) of each job's expiry. You can't simply keep the worker running until the count reaches 0. In dispatcher-free mode a running worker keeps claiming new sessions, and every new job starts a new dispute window. So the worker has to stop first, and the release is then run by hand from a one-off container, with the same image and keys.

Prerequisites

The examples below assume the setup from Run a Worker — Testnet or Mainnet: the env file ~/lightchain-worker/worker.env and the keys directory ~/lightchain-worker/keys. Set these in your shell:
CodeBASH
This shell function runs one lightchain-worker command in a one-off container with your env file and keys. It works while the worker container is stopped:
CodeBASH
cast (from Foundry) is used for read-only chain queries throughout. Installed with the CLI (Run a Worker on Testnet with the CLI)? Use the lcw function from that guide, stop the worker with sudo systemctl stop lightchain-worker, and read the metrics with curl -s http://127.0.0.1:9101/metrics. The steps are otherwise the same.

Exit sequence

Stop, wait, release, optionally withdraw, deregister, remove. Steps 1–3 are mandatory; step 4 is optional housekeeping; steps 5–6 retire the worker.

1. Stop the worker

A running worker keeps claiming sessions until the process stops. lightchain-worker drain doesn't change that: it sets a marker that only the retired dispatcher read, and nothing checks it before a claim. Stop it:
CodeBASH
On SIGTERM the worker claims no new session and starts no new job, and the jobs it is already running keep going until they finish or SHUTDOWN_TIMEOUT runs out (240 s in the install guides, 30 s when unset). A job still running then is cancelled: it times out on-chain and the timeout is recorded as an offence (see Slashing & Rehabilitation). Keep docker stop -t above SHUTDOWN_TIMEOUT, or Docker kills the worker before its jobs finish. Worker images before release v0.0.5 end a running job at SIGTERM; pull latest first. Jobs submitted later in sessions the worker already claimed still target it. While it is offline they time out the same way. Keep the container (no docker rm yet): step 2 reads its logs, and docker start lightchain-worker brings it back if you change your mind. Its release_state.json, the list of jobs waiting for release, is on the mounted volume (RELEASE_STATE_PATH=/data/release_state.json), and the one-off container in step 3 reads it from there.

2. Wait out the dispute window

A job can be released once the dispute window has passed since it completed. Read the window:
CodeBASH
and find when the last job completed:
CodeBASH
Wait until that time plus the dispute window has passed.

3. Release

CodeBASH
release first brings the ledger up to date from the chain, then sends one releaseJobs transaction for the jobs that are past the window. It prints, for example:
CodeTEXT
What the last two lines mean:
  • pending after cycle: 0 — everything is settled. Go on to the count check below.
  • 50 released with jobs still pending — one run releases at most 50 jobs. Run it again.
  • 0 released with jobs still pending — those jobs are still inside their dispute window, or under dispute. Run it again once the window has passed.
  • no jobs ready for release — the ledger holds nothing to release.
  • release cycle paused or release cycle blocked: backoff active until chain time … — the contract was paused on the last attempt. Run it again after that time.
Then confirm the count is 0:
CodeBASH
If the count stays above 0 although release has nothing left to release, some jobs never completed. See the recovery scenarios.

4. (Optional) Withdraw earnings

Released fees accumulate as the worker's balance in JobRegistry. Check what's claimable, then transfer it to the worker address:
CodeBASH
From there you can sweep it to your designated payout wallet as described in Rewards and fund handling in the install guides (Testnet, Mainnet).

5. Deregister

Once activeJobsCount is 0, deregister unlocks your staked LCAI and removes the worker from the registry:
CodeBASH
If this reverts with ActiveJobsExist, go back to step 3. A failed deregister doesn't change on-chain state; you can fix the condition and re-run.

6. Remove the container

Only after deregister has succeeded:
CodeBASH

Quick reference

ConcernCommand
Jobs running right nowdocker exec lightchain-worker wget -qO- http://127.0.0.1:9101/metrics | grep '^worker_active_jobs '
Check activeJobsCountcast call $WORKER_REGISTRY_ADDRESS "getActiveJobCount(address)(uint256)" $WORKER_ADDR --rpc-url $RPC_URL
Check dispute window lengthcast call $AI_CONFIG_ADDRESS "getDisputeWindow()(uint256)" --rpc-url $RPC_URL (seconds)
Release jobs past the windowlcw release
Check claimable earningslcw balance
Change your mind and keep servingdocker start lightchain-worker

Recovery scenarios

"I ran deregister and got ActiveJobsExist." Your worker is still registered — the failed tx didn't burn anything. Run lcw release (after the dispute window), check getActiveJobCount, then retry deregister. "activeJobsCount never drops." release can only settle jobs that reached Completed. If your worker went offline part-way through a job, that job is frozen in Submitted or Acknowledged past its deadline, and no amount of waiting will clear it. See Stuck Jobs — clearing these carries a penalty, so check the cost before acting. "I removed the container (docker rm) before deregistering." Nothing is lost when RELEASE_STATE_PATH=/data/release_state.json is set, as in the current install guides: the ledger is on the mounted volume. Without it, the ledger was inside the container. Your claimable funds and registration still exist on-chain, and lcw release rebuilds the ledger from the chain before it releases. That scan covers the whole chain and takes a few minutes.

Important behaviors

  • A running worker keeps claiming. It does not check the drain marker before a claim, so stopping the process is the only way to take it out of rotation.
  • Stopping lets running jobs finish, up to a limit. SIGTERM stops new work at once and gives the jobs already running up to SHUTDOWN_TIMEOUT to finish (worker release v0.0.5 on). Keep docker stop -t above it.
  • The release scheduler needs a running worker. A stopped worker settles nothing on its own; lcw release does it from a one-off container.
  • deregister is retryable. A failed deregister (e.g. ActiveJobsExist) doesn't change on-chain state. Fix the underlying condition and re-run; nothing is consumed by the revert.

Related pages

Previous PageDispatcher-free ModeNext PageStuck Jobs