Skip to content

Integrate vendor VFIO vGPUs into the instance lifecycle - #321

Open
yummybomb wants to merge 30 commits into
hypeship/vendor-vfio-backendfrom
hypeship/vendor-vfio-vgpu
Open

Integrate vendor VFIO vGPUs into the instance lifecycle#321
yummybomb wants to merge 30 commits into
hypeship/vendor-vfio-backendfrom
hypeship/vendor-vfio-vgpu

Conversation

@yummybomb

@yummybomb yummybomb commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Top layer of the vendor VFIO vGPU stack (generalize-vgpu-device #322hypervisor-liveness #363vendor-vfio-backend #364this). The backend itself is in #364; this layer wires it into the instance lifecycle and adds the release guards that make reusable-VF assignments safe:

  • Live-instance claim guard on release — an assignment goes stale when its release succeeds but the metadata save does not (or start fails between the release and its first save). Since vendor VFIO reuses the same VF path across assignments, honoring a stale claim could clear a VF out from under the live instance that owns it now. Every release consults live instance metadata (liveness verified via Harden hypervisor process liveness checks #363's socket-ownership check) and drops stale metadata without touching the device.
  • Assignment durability — assignments are tagged with the owning instance ID, persisted before booting a started instance without carrying over the previous VMM identity, and retained in metadata when rollback release fails in create/start so later release paths can still find the device.
  • Fail-closed startup reconciliation — the protected set of live VFs is built from ListInstancesForReconcile, which fails on any unreadable metadata instead of silently skipping it. Recent assignments receive bounded startup protection when their PID is absent or stale; when the inventory is unavailable, vendor VFIO reconciliation is skipped entirely while mdev reconciliation still runs.
  • Hypervisor scope — hypervisor selection remains a caller policy, preserving the framework layer's existing mdev behavior. The production rollout continues to place vGPU instances on QEMU.

Testing

  • go build ./..., go vet ./... clean
  • go test -race passes for lib/instances targeted suites, lib/devices, lib/resources, lib/builds (TestCreateInstanceWithNetwork needs image pulls + iptables and fails in this environment on the unmodified base as well)

Note

High Risk
Changes GPU assignment/release, startup reconciliation, and fail-closed metadata scans on paths shared by live VMs; incorrect liveness or protection logic could drop or retain the wrong VF assignment.

Overview
Wires vendor VFIO into create/start/stop/delete (replacing the previous “not integrated” gate) and treats vGPU assignments as durable host state tracked with GPUAssignedAt, instance-scoped create/destroy hooks, and metadata retention when rollback release fails.

Release safety for reused VF paths: vendor VFIO releases scan other instances’ metadata and verify hypervisor liveness before destroying a device; stale claims are dropped without touching hardware, while ambiguous or unreadable inventory fails closed so the requester keeps its assignment. Mdev releases skip that scan.

Create/start failure handling: failed rollbacks can surface VGPUCleanupPendingError to the API as vgpu_cleanup_pending (with retained vs reconcile-only guidance). Start clears stale VMM identity before assigning a new vGPU and persists assignments before boot continues.

Startup: API boot replaces bare mdev reconcile with reconcileVGPUs, building a protected VF set from ListInstancesForReconcile (strict on bad metadata), live PIDs, and a bounded grace window for assignments without a PID; it may reschedule reconcile when protection expires. Snapshot restore keeps the instance’s current vGPU fields (including GPUAssignedAt) instead of resurrecting snapshot-embedded assignments.

Reviewed by Cursor Bugbot for commit 6e4e796. Bugbot is set up for automated code reviews on this repo. Configure here.

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 199b1d1 to a93d009 Compare July 28, 2026 21:32
@yummybomb
yummybomb changed the base branch from main to hypeship/generalize-vgpu-device July 28, 2026 21:32
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch 4 times, most recently from d893fce to de4742a Compare July 29, 2026 15:15
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from de4742a to 18e047f Compare July 29, 2026 16:09
@yummybomb

yummybomb commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Trimmed over-engineering after an architectural review against the parent task (host kernel 6.8 / Ubuntu 24.04 support). Most of the removed weight came from earlier review-round additions rather than the original design:

@yummybomb
yummybomb marked this pull request as ready for review August 5, 2026 19:47
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 7ef1f8d to bf21162 Compare August 5, 2026 19:57
Comment thread lib/instances/start.go Outdated
Comment thread integration/vgpu_test.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from bf21162 to 83c53ea Compare August 5, 2026 20:24
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 83c53ea to 79eef07 Compare August 5, 2026 20:39
Comment thread cmd/api/main.go
Comment thread lib/devices/vendor_vfio_linux.go Outdated
Comment thread lib/instances/query.go
Comment thread lib/instances/create.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from ab2b0e8 to 19b9602 Compare August 6, 2026 17:08
Comment thread lib/instances/start.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from d92201d to f177335 Compare August 6, 2026 19:06
@yummybomb
yummybomb changed the base branch from hypeship/generalize-vgpu-device to hypeship/vendor-vfio-backend August 6, 2026 19:08
@yummybomb yummybomb changed the title Support vendor VFIO vGPU devices Integrate vendor VFIO vGPUs into the instance lifecycle Aug 6, 2026
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from f177335 to 1d875f2 Compare August 6, 2026 19:26
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 1d875f2 to 23fb2f5 Compare August 6, 2026 19:40
The claim guard compared raw GPUDevicePath, which is empty on records
persisted before the framework migration; a live claimant with only a
legacy GPUMdevUUID was invisible to the check. Normalize the inventory
side with storedVGPUDevicePath, matching the release subject.
…r seam

Replace the go:linkname shadow of devices.hostVendorVFIO with createVGPU/destroyVGPU manager fields, and wrap failed creates whose rollback release also failed in VGPUCleanupPendingError so the API can point callers at the retained instance record.
Enable vendor VFIO dispatch in CreateVGPU now that the lifecycle
persists assignments durably and guards releases.

Protect nil-PID claims in the release guard: the hypervisor PID is only
persisted after the claimant boots, so a matching assignment without a
PID must be treated as live, matching the startup reconcile protection.

Scan raw metadata instead of hydrating instances for the claim check.
Hydration derives state through hypervisor queries for every instance
on the host, which every vendor VFIO release would pay; the guard only
needs the stored assignment, PID, and socket. Unreadable metadata still
fails the release closed.

Report pending vGPU cleanup even when retaining the rollback record
fails: the destroy already failed, so the caller must learn about the
outstanding assignment either way.
The vgpu_cleanup_pending response replaced the original create error
with cleanup guidance, leaving the cause only in server logs. Prefix
the message with the wrapped error so callers see why creation failed
as well as how to recover.
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 2316ae5 to ef0a018 Compare August 10, 2026 19:53

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f2ca548. Configure here.

Comment thread cmd/api/main.go
ListInstancesForReconcile hydrated every instance (socket stat, UFFD
health, /vm.info per instance) before the API served and again on each
grace retry, while the protected-set scan only reads stored metadata
fields. List raw metadata fail-closed instead, matching the release
claim scan, and drop the now-unused loadInstances parameterization.

Snapshot restore preserved the source's vGPU assignment path fields but
not GPUAssignedAt, so a retained assignment lost its crash-recovery
grace timestamp across a restore. Carry the timestamp with the rest of
the assignment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant