Skip to content

fix k3s shutdown cleanup - #816

Draft
bussyjd wants to merge 1 commit into
mainfrom
agent/k3s-clean-shutdown
Draft

fix k3s shutdown cleanup#816
bussyjd wants to merge 1 commit into
mainfrom
agent/k3s-clean-shutdown

Conversation

@bussyjd

@bussyjd bussyjd commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • locate the official k3s-killall.sh beside the configured K3s binary, beside a resolved binary symlink, or in the standard installer fallback locations
  • run the helper with Obol's custom K3S_DATA_DIR, including when the PID file is missing or stale
  • remove the incomplete pkill containerd-shim fallback and surface cleanup failures while retaining PID evidence
  • stop Destroy from deleting global K3s directories or invoking the system-wide uninstaller; persistent data remains governed by stack purge --force

Root cause

K3s intentionally leaves containers running when its service stops. Its official killall helper performs the corresponding containerd, CNI, mount, and iptables cleanup while preserving cluster data:

Obol previously looked only for /usr/local/bin/k3s-killall.sh. The supported backend setup uses INSTALL_K3S_BIN_DIR, which installs the helper beside the K3s binary. Missing that helper caused shutdown to fall back to killing shim processes without removing K3s networking state.

The previous destroy path also invoked the system-wide K3s uninstaller and removed global directories. That exceeds Obol's ownership boundary and bypasses the CLI's existing --force control for persistent-data removal.

Impact

obol stack down now delegates runtime cleanup to the version-matched K3s helper and preserves cluster data for restart. Stale PID state no longer prevents orphan cleanup. obol stack purge preserves K3s data unless the operator explicitly passes --force, and it no longer removes system K3s assets.

Validation

No live K3s instance was exercised.

  • go test ./cmd/obol ./internal/tunnel ./internal/stack -count=1
  • go test -race ./internal/stack -run 'TestK3s' -count=1
  • golangci-lint run --new-from-rev=main ./internal/stack/...
  • go build ./cmd/obol
  • focused tests use fake cleanup helpers to verify discovery, symlink resolution, K3S_DATA_DIR propagation, missing-PID cleanup, failure retention, and persistent-data preservation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant