Harden hypervisor process liveness checks - #363
Conversation
f9e6580 to
3c8b92f
Compare
3c8b92f to
915f0ef
Compare
915f0ef to
a187984
Compare
76b9f42 to
78fc483
Compare
78fc483 to
f8fbe79
Compare
ea4603c to
f17f33e
Compare
9c1fbaf to
1a1ff46
Compare
7dbe8e7 to
be6c8ff
Compare
be6c8ff to
fdd7b9a
Compare
fdd7b9a to
bba964c
Compare
bba964c to
201852a
Compare
kill(pid, 0) returning EPERM means the process exists but cannot be signaled, and a zombie PID passes a bare kill(0) probe. Export the EPERM-aware, zombie-filtering processExists helper so every hypervisor liveness check shares one definition.
After a hypeman restart the hypervisor is not our child, so Wait4 returns ECHILD immediately and the kill loop finished before the process had exited. Poll for actual process exit in that case.
A bare liveness probe treats any process that reused a stored hypervisor PID as the owning VMM. Require the PID to own the instance's hypervisor socket on Linux before reporting it alive.
Accepted server-side sockets appear in /proc/net/unix with the same bound path as the listener, so any connected API client made socketRefForPath report multiple inodes and pid-reuse protection fell back to unconfirmed while the control socket was in use. Only entries with __SO_ACCEPTCON identify the owning process; duplicate listeners from unlink-and-rebind still resolve as unconfirmed.
Require confirmed socket ownership before any destructive kill: a command-line match is no longer sufficient to SIGKILL the stored PID. When ownership of a live stored PID cannot be confirmed, or the process does not exit after SIGKILL, killHypervisor now returns an error and keeps the socket in place, and delete aborts before releasing the vGPU, network, devices, or metadata. The restart policy is already blocked at that point, so the retained instance can be deleted again safely.
201852a to
6b710d7
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2c2bb69. Configure here.
| return fallbackPID | ||
| } | ||
| pid, err := hypervisor.ResolveProcessPID(socketPath) | ||
| pid, _, err := hypervisor.ResolveProcessPID(socketPath) |
There was a problem hiding this comment.
Unconfirmed PID becomes trusted identity
Medium Severity
resolveRuntimeHypervisorPID ignores the new confirmed flag from ResolveProcessPID, so a command-line-only match can be passed to setHypervisorProcessIdentity. Later, resolveLiveHypervisorPID short-circuits on that PID/start-time/boot-id token without checking socket ownership, which can target the wrong process on stop/delete while the real socket owner keeps running.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 2c2bb69. Configure here.


Summary
Layer 1 of the vendor VFIO vGPU stack (
generalize-vgpu-device← this ←vendor-vfio-backend←vendor-vfio-vgpu). Pure hypervisor-process hardening with no vGPU-specific code; reviewable in isolation.The upper layers guard vGPU release decisions on "is this instance's hypervisor still alive", so the liveness answer has to be trustworthy first:
ProcessExists— one exported, EPERM-aware, zombie-filtering definition instead of scattered barekill(pid, 0)probes. EPERM means the process exists but cannot be signaled; treating it as dead would be wrong.Wait4returns ECHILD immediately and the kill loop finished before the process had exited. Poll for actual exit in that case.Testing
go build ./...,go vetcleango test -race ./lib/instances/targeted suites pass (TestCreateInstanceWithNetworkrequires image pulls + iptables and fails in this environment on the unmodified base as well)Note
High Risk
Changes fail-closed kill/delete/stop and Linux /proc socket resolution; mistakes could leave VMs running or SIGKILL the wrong process, and delete now errors instead of best-effort teardown when kill cannot be confirmed.
Overview
Hardens how hypeman decides which process owns a VM’s control socket and when it is safe to SIGKILL or tear down instance resources.
Linux socket PID resolution now returns
(pid, confirmed, err), distinguishes confirmed FD ownership from cmdline-only matches, filters/proc/net/unixto listening sockets (__SO_ACCEPTCON), handles multiple owners viaResolveProcessPIDForOwner, and surfacesErrNoOwningProcess.Process identity is persisted as
HypervisorStartTimeandHypervisorBootIDalongsideHypervisorPID.resolveLiveHypervisorPID,HypervisorProcessExists, and exportedProcessExists(EPERM-aware, zombie-filtering) gate stop, delete, and PID refresh so reused PIDs or cmdline-only matches cannot satisfy destructive paths.Delete/stop behavior is fail-closed: failed hypervisor kill aborts delete (metadata retained);
killHypervisorand graceful shutdown wait on the confirmed socket owner and poll up to 30s after SIGKILL via improvedWaitForProcessExitfor non-child VMMs.Identity fields are cleared on stop/standby/fork/snapshot rollback paths; broad Linux integration tests cover PID reuse, stale metadata, and unconfirmed ownership.
Reviewed by Cursor Bugbot for commit 2c2bb69. Bugbot is set up for automated code reviews on this repo. Configure here.