Skip to content

Recycle a worker whose ateom container was replaced under it + telemetry coverage - #1473

Draft
Krisztian F (krisztianfekete) wants to merge 1 commit into
agent-substrate:mainfrom
krisztianfekete:feat/worker-sandbox-gone
Draft

Recycle a worker whose ateom container was replaced under it + telemetry coverage#1473
Krisztian F (krisztianfekete) wants to merge 1 commit into
agent-substrate:mainfrom
krisztianfekete:feat/worker-sandbox-gone

Conversation

@krisztianfekete

Copy link
Copy Markdown
Collaborator

An ateom container that OOMs or crashes is restarted in place by the kubelet. Every actor on that worker stayed RUNNING pointing at a sandbox that no longer existed. ResumeActor short-circuits on RUNNING, so nothing would ever recover it, and the worker stayed claimed. Since the multi-actor worker API that is every actor on the worker, not one.

Before After
Actor RUNNING, unroutable, never recovers CRASHED with WORKER_SANDBOX_GONE and the kubelet's own reason
Worker claimed forever, effectively gone from the pool recycled in place, capacity kept, schedulable again
Crash-looping ateom keeps taking actors and destroying them drained, pod handed back to its Deployment
Telemetry nothing fires ate.actor.crashes + one trace joining the cause to every actor it crashed

How

WorkerStatus.ateom_container_id records the container a Worker is running; when the syncer sees a different one, a new RecycleWorker RPC releases every bound actor and hands the Worker back schedulable. The record is kept rather than re-registered on purpose: an ateom reports capacity once per process and CreateWorker leaves it unset, so a re-registered Worker would be permanently unschedulable.

An OOMKilled worker container is infrastructure, not workload. The limit is the operator's and the cgroup covers the sandbox runtime and every actor on the worker. Also fixes finalizeRunning committing RUNNING over an actor the
release path had already crashed.

Operational notes

Roll out ateapi before atecontroller. atecontroller now needs delete on core/pods to replace a crash-looping worker.

Local testing

Killing a live ateom under a running actor produced the crash with the right reason and trace, left the worker
schedulable, and a new actor was placed back on it; driving one into CrashLoopBackOff drained it and its Deployment replaced it.

  • Tests pass
  • Appropriate changes to documentation are included in the PR

@krisztianfekete
Krisztian F (krisztianfekete) force-pushed the feat/worker-sandbox-gone branch 2 times, most recently from d9a4133 to fe6d629 Compare September 4, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant