Integrate vendor VFIO vGPUs into the instance lifecycle - #321
Open
yummybomb wants to merge 30 commits into
Open
Conversation
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
July 28, 2026 21:32
199b1d1 to
a93d009
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
4 times, most recently
from
July 29, 2026 15:15
d893fce to
de4742a
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
July 29, 2026 16:09
de4742a to
18e047f
Compare
Contributor
Author
|
Trimmed over-engineering after an architectural review against the parent task (host kernel 6.8 / Ubuntu 24.04 support). Most of the removed weight came from earlier review-round additions rather than the original design: |
yummybomb
marked this pull request as ready for review
August 5, 2026 19:47
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 5, 2026 19:57
7ef1f8d to
bf21162
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 5, 2026 20:24
bf21162 to
83c53ea
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 5, 2026 20:39
83c53ea to
79eef07
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 6, 2026 17:08
ab2b0e8 to
19b9602
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 6, 2026 19:06
d92201d to
f177335
Compare
yummybomb
changed the base branch from
hypeship/generalize-vgpu-device
to
hypeship/vendor-vfio-backend
August 6, 2026 19:08
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 6, 2026 19:26
f177335 to
1d875f2
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 6, 2026 19:40
1d875f2 to
23fb2f5
Compare
The claim guard compared raw GPUDevicePath, which is empty on records persisted before the framework migration; a live claimant with only a legacy GPUMdevUUID was invisible to the check. Normalize the inventory side with storedVGPUDevicePath, matching the release subject.
…r seam Replace the go:linkname shadow of devices.hostVendorVFIO with createVGPU/destroyVGPU manager fields, and wrap failed creates whose rollback release also failed in VGPUCleanupPendingError so the API can point callers at the retained instance record.
Enable vendor VFIO dispatch in CreateVGPU now that the lifecycle persists assignments durably and guards releases. Protect nil-PID claims in the release guard: the hypervisor PID is only persisted after the claimant boots, so a matching assignment without a PID must be treated as live, matching the startup reconcile protection. Scan raw metadata instead of hydrating instances for the claim check. Hydration derives state through hypervisor queries for every instance on the host, which every vendor VFIO release would pay; the guard only needs the stored assignment, PID, and socket. Unreadable metadata still fails the release closed. Report pending vGPU cleanup even when retaining the rollback record fails: the destroy already failed, so the caller must learn about the outstanding assignment either way.
The vgpu_cleanup_pending response replaced the original create error with cleanup guidance, leaving the cause only in server logs. Prefix the message with the wrapped error so callers see why creation failed as well as how to recover.
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 10, 2026 19:53
2316ae5 to
ef0a018
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f2ca548. Configure here.
ListInstancesForReconcile hydrated every instance (socket stat, UFFD health, /vm.info per instance) before the API served and again on each grace retry, while the protected-set scan only reads stored metadata fields. List raw metadata fail-closed instead, matching the release claim scan, and drop the now-unused loadInstances parameterization. Snapshot restore preserved the source's vGPU assignment path fields but not GPUAssignedAt, so a retained assignment lost its crash-recovery grace timestamp across a restore. Carry the timestamp with the rest of the assignment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Top layer of the vendor VFIO vGPU stack (
generalize-vgpu-device#322 ←hypervisor-liveness#363 ←vendor-vfio-backend#364 ← this). The backend itself is in #364; this layer wires it into the instance lifecycle and adds the release guards that make reusable-VF assignments safe:ListInstancesForReconcile, which fails on any unreadable metadata instead of silently skipping it. Recent assignments receive bounded startup protection when their PID is absent or stale; when the inventory is unavailable, vendor VFIO reconciliation is skipped entirely while mdev reconciliation still runs.Testing
go build ./...,go vet ./...cleango test -racepasses forlib/instancestargeted suites,lib/devices,lib/resources,lib/builds(TestCreateInstanceWithNetworkneeds image pulls + iptables and fails in this environment on the unmodified base as well)Note
High Risk
Changes GPU assignment/release, startup reconciliation, and fail-closed metadata scans on paths shared by live VMs; incorrect liveness or protection logic could drop or retain the wrong VF assignment.
Overview
Wires vendor VFIO into create/start/stop/delete (replacing the previous “not integrated” gate) and treats vGPU assignments as durable host state tracked with
GPUAssignedAt, instance-scoped create/destroy hooks, and metadata retention when rollback release fails.Release safety for reused VF paths: vendor VFIO releases scan other instances’ metadata and verify hypervisor liveness before destroying a device; stale claims are dropped without touching hardware, while ambiguous or unreadable inventory fails closed so the requester keeps its assignment. Mdev releases skip that scan.
Create/start failure handling: failed rollbacks can surface
VGPUCleanupPendingErrorto the API asvgpu_cleanup_pending(with retained vs reconcile-only guidance). Start clears stale VMM identity before assigning a new vGPU and persists assignments before boot continues.Startup: API boot replaces bare mdev reconcile with
reconcileVGPUs, building a protected VF set fromListInstancesForReconcile(strict on bad metadata), live PIDs, and a bounded grace window for assignments without a PID; it may reschedule reconcile when protection expires. Snapshot restore keeps the instance’s current vGPU fields (includingGPUAssignedAt) instead of resurrecting snapshot-embedded assignments.Reviewed by Cursor Bugbot for commit 6e4e796. Bugbot is set up for automated code reviews on this repo. Configure here.