Skip to content

Harden hypervisor process liveness checks - #363

Open
yummybomb wants to merge 22 commits into
hypeship/generalize-vgpu-devicefrom
hypeship/hypervisor-liveness
Open

Harden hypervisor process liveness checks#363
yummybomb wants to merge 22 commits into
hypeship/generalize-vgpu-devicefrom
hypeship/hypervisor-liveness

Conversation

@yummybomb

@yummybomb yummybomb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Layer 1 of the vendor VFIO vGPU stack (generalize-vgpu-devicethisvendor-vfio-backendvendor-vfio-vgpu). Pure hypervisor-process hardening with no vGPU-specific code; reviewable in isolation.

The upper layers guard vGPU release decisions on "is this instance's hypervisor still alive", so the liveness answer has to be trustworthy first:

  • Unify liveness checks on ProcessExists — one exported, EPERM-aware, zombie-filtering definition instead of scattered bare kill(pid, 0) probes. EPERM means the process exists but cannot be signaled; treating it as dead would be wrong.
  • Wait for non-child hypervisor exit before finishing kill — after a hypeman restart the hypervisor is not our child, so Wait4 returns ECHILD immediately and the kill loop finished before the process had exited. Poll for actual exit in that case.
  • Verify process identity before teardown — a bare PID probe treats any process that reused a stored hypervisor PID as the owning VMM. On Linux, require confirmed socket ownership or a matching PID/start-time token scoped to the current host boot, including the graceful stop/delete wait path.

Testing

  • go build ./..., go vet clean
  • go test -race ./lib/instances/ targeted suites pass (TestCreateInstanceWithNetwork requires image pulls + iptables and fails in this environment on the unmodified base as well)

Note

High Risk
Changes fail-closed kill/delete/stop and Linux /proc socket resolution; mistakes could leave VMs running or SIGKILL the wrong process, and delete now errors instead of best-effort teardown when kill cannot be confirmed.

Overview
Hardens how hypeman decides which process owns a VM’s control socket and when it is safe to SIGKILL or tear down instance resources.

Linux socket PID resolution now returns (pid, confirmed, err), distinguishes confirmed FD ownership from cmdline-only matches, filters /proc/net/unix to listening sockets (__SO_ACCEPTCON), handles multiple owners via ResolveProcessPIDForOwner, and surfaces ErrNoOwningProcess.

Process identity is persisted as HypervisorStartTime and HypervisorBootID alongside HypervisorPID. resolveLiveHypervisorPID, HypervisorProcessExists, and exported ProcessExists (EPERM-aware, zombie-filtering) gate stop, delete, and PID refresh so reused PIDs or cmdline-only matches cannot satisfy destructive paths.

Delete/stop behavior is fail-closed: failed hypervisor kill aborts delete (metadata retained); killHypervisor and graceful shutdown wait on the confirmed socket owner and poll up to 30s after SIGKILL via improved WaitForProcessExit for non-child VMMs.

Identity fields are cleared on stop/standby/fork/snapshot rollback paths; broad Linux integration tests cover PID reuse, stale metadata, and unconfirmed ownership.

Reviewed by Cursor Bugbot for commit 2c2bb69. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread lib/hypervisor/socket_pid_linux.go
@yummybomb
yummybomb force-pushed the hypeship/hypervisor-liveness branch 2 times, most recently from 76b9f42 to 78fc483 Compare August 7, 2026 20:52
@yummybomb
yummybomb force-pushed the hypeship/hypervisor-liveness branch from 78fc483 to f8fbe79 Compare August 8, 2026 01:05
Comment thread lib/instances/delete.go Outdated
Comment thread lib/instances/delete.go Outdated
Comment thread lib/instances/query.go
@yummybomb
yummybomb force-pushed the hypeship/hypervisor-liveness branch from 9c1fbaf to 1a1ff46 Compare August 10, 2026 15:37
@yummybomb
yummybomb force-pushed the hypeship/hypervisor-liveness branch from 7dbe8e7 to be6c8ff Compare August 10, 2026 18:26
@yummybomb
yummybomb force-pushed the hypeship/hypervisor-liveness branch from be6c8ff to fdd7b9a Compare August 10, 2026 18:34
@yummybomb
yummybomb force-pushed the hypeship/hypervisor-liveness branch from fdd7b9a to bba964c Compare August 10, 2026 18:58
@yummybomb
yummybomb force-pushed the hypeship/hypervisor-liveness branch from bba964c to 201852a Compare August 10, 2026 19:08
kill(pid, 0) returning EPERM means the process exists but cannot be
signaled, and a zombie PID passes a bare kill(0) probe. Export the
EPERM-aware, zombie-filtering processExists helper so every hypervisor
liveness check shares one definition.
After a hypeman restart the hypervisor is not our child, so Wait4
returns ECHILD immediately and the kill loop finished before the
process had exited. Poll for actual process exit in that case.
A bare liveness probe treats any process that reused a stored
hypervisor PID as the owning VMM. Require the PID to own the
instance's hypervisor socket on Linux before reporting it alive.
Accepted server-side sockets appear in /proc/net/unix with the same
bound path as the listener, so any connected API client made
socketRefForPath report multiple inodes and pid-reuse protection fell
back to unconfirmed while the control socket was in use. Only entries
with __SO_ACCEPTCON identify the owning process; duplicate listeners
from unlink-and-rebind still resolve as unconfirmed.
Require confirmed socket ownership before any destructive kill: a
command-line match is no longer sufficient to SIGKILL the stored PID.
When ownership of a live stored PID cannot be confirmed, or the process
does not exit after SIGKILL, killHypervisor now returns an error and
keeps the socket in place, and delete aborts before releasing the vGPU,
network, devices, or metadata. The restart policy is already blocked at
that point, so the retained instance can be deleted again safely.
Comment thread lib/instances/stop.go
@yummybomb
yummybomb force-pushed the hypeship/hypervisor-liveness branch from 201852a to 6b710d7 Compare August 10, 2026 19:32

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 2c2bb69. Configure here.

Comment thread lib/instances/create.go
return fallbackPID
}
pid, err := hypervisor.ResolveProcessPID(socketPath)
pid, _, err := hypervisor.ResolveProcessPID(socketPath)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unconfirmed PID becomes trusted identity

Medium Severity

resolveRuntimeHypervisorPID ignores the new confirmed flag from ResolveProcessPID, so a command-line-only match can be passed to setHypervisorProcessIdentity. Later, resolveLiveHypervisorPID short-circuits on that PID/start-time/boot-id token without checking socket ownership, which can target the wrong process on stop/delete while the real socket owner keeps running.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 2c2bb69. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant