Skip to content

Add the vendor VFIO vGPU device backend - #364

Open
yummybomb wants to merge 19 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend
Open

Add the vendor VFIO vGPU device backend#364
yummybomb wants to merge 19 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend

Conversation

@yummybomb

@yummybomb yummybomb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Layer 2 of the vendor VFIO vGPU stack (generalize-vgpu-devicehypervisor-livenessthisvendor-vfio-vgpu). Self-contained in lib/devices + lib/resources; nothing in the instance lifecycle calls it yet (that's the top layer).

Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's nvidia/current_vgpu_type and passed to QEMU as a plain VFIO PCI device. This adds that backend behind the framework dispatch introduced in #322:

  • Discovery & placement — profile discovery from the capacity-dependent creatable_vgpu_types catalogs, least-loaded-GPU VF selection, create/verify/rollback. Profile availability counts free VFs currently advertising each type as a best-effort snapshot; creating one assignment may change sibling catalogs.
  • Release guards — unlike mdev (fresh UUID per assignment), vendor VFIO reuses the same VF path across assignments, so a stale release could clear a later owner's vGPU. Release is guarded by an in-process owner map (covers the window before QEMU opens the device) and an open-VFIO-handle scan (refuses to clear a VF a running VM holds).
  • Reconciliation — clears orphaned assignments on startup, skips VFs in the caller-supplied protected set, and fails closed (skips vendor VFIO entirely) when that set is unavailable, while mdev reconciliation still runs.
  • Integration test — branched by discovered framework, extended to cover release on stop and reacquisition on start.

Testing

  • go build ./..., go vet clean
  • go test -race ./lib/devices/ ./lib/resources/ pass

Note

Medium Risk
Touches host GPU sysfs assignment and release with VFIO in-use checks, but vendor VFIO instance creation remains disabled until a follow-up integration layer lands.

Overview
Adds NVIDIA vendor VFIO vGPU support for newer hosts (assign via current_vgpu_type / creatable_vgpu_types) alongside the existing mdev path, unified behind DiscoverVGPU and framework-aware profile listing, create/destroy, and reconciliation.

vendor_vfio_linux.go implements sysfs discovery, profile availability from free VFs, least-loaded-GPU placement, create/verify/rollback, and teardown that refuses to clear a VF when open VFIO handles exist or a stale instance does not own the assignment. ReconcileVGPUs clears orphaned vendor assignments while honoring a protected device-path set.

CreateVGPU still errors on vendor VFIO until instance lifecycle integration lands; destroy/reconcile and resources reporting already route through the new stack. GetGPUStatus now takes context, probes vGPU before falling back to passthrough, and documents profile availability as a best-effort VF snapshot.

The vGPU integration test is framework-aware (sysfs checks, stop/release, start/reacquire) but skips vendor VFIO hosts until create is wired. Docs/OpenAPI reflect mdev vs vendor VFIO and ephemeral assignment lifecycle.

Reviewed by Cursor Bugbot for commit 9769a33. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread lib/devices/vendor_vfio_linux.go Outdated
Comment thread lib/resources/gpu.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 2bb8e86 to 7fc3b49 Compare August 6, 2026 19:26
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 7fc3b49 to f661e63 Compare August 6, 2026 19:40
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from f661e63 to d1207d0 Compare August 7, 2026 14:02
Comment thread lib/resources/gpu.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from d1207d0 to 5b47670 Compare August 7, 2026 15:04
Comment thread lib/devices/vendor_vfio_linux.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 5b47670 to ffbf8a0 Compare August 7, 2026 20:52
Comment thread lib/devices/vendor_vfio_linux.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from ffbf8a0 to 78801e7 Compare August 8, 2026 01:05
Comment thread integration/vgpu_test.go Outdated
@yummybomb

Copy link
Copy Markdown
Contributor Author

added 7f5233f: report GPUProfile.Available as the count of free VFs advertising the profile type (creatable-instance units), matching mdev's summed available_instances, the OpenAPI description, and the integration test's decrement assertion. note the gpuProfileSlots metric steps up on vendor VFIO hosts (per-GPU → per-free-VF units). go test ./lib/devices ./lib/resources green; the hardware integration test was not run locally.

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 7f5233f to 0baacb3 Compare August 10, 2026 07:26
@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
-->

✱ stlc build

go code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

112 files generated at 6927b75 (pushed)

go get github.com/kernel/hypeman-go-staging@6927b75ff19ed53011cc2047f9d61c23cc1cc694
typescript code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

134 files generated at 6a2532f (pushed)

Diagnostics: 💡 0 new / 5 total note
LevelCodeMessageTargets
Build metadata
Buildbd_766sHtSJ-forked-cradle
Timestamp2026-08-10T20:45:58.247Z
stlc8413509
Spec hashc36f98252380
Config hash9125eb22cb4c

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-08-10 20:46:15 UTC

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from eb00c67 to 0fb3c6a Compare August 10, 2026 18:34
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 0fb3c6a to 5beb0fb Compare August 10, 2026 18:58
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 5beb0fb to ea22b71 Compare August 10, 2026 19:08
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from ea22b71 to 49cb877 Compare August 10, 2026 19:32
Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are
assigned by writing a type ID to a VF's nvidia/current_vgpu_type and
passed to QEMU as a plain VFIO PCI device. Add a vendor VFIO backend
behind the existing framework dispatch: profile discovery from the
capacity-dependent creatable catalogs, least-loaded VF placement,
create/verify/rollback, and release.

Because the same VF path is reused across assignments (unlike mdev
UUIDs), release is guarded: an in-process owner map covers the window
before QEMU opens the device, and an open-VFIO-handle scan refuses to
clear a VF a running VM still holds. Reconciliation clears orphaned
assignments on startup, skipping VFs protected by the caller and
failing closed when the protected set is unavailable.

Branch the vGPU integration test by discovered framework and extend it
to cover release on stop and reacquisition on start.
Sort GPUs with unaccountable load last instead of rejecting placement, and stop reporting passthrough capacity when vGPU discovery fails.
The instance lifecycle already routes create/start/stop/delete through
CreateVGPU/DestroyVGPU, so dispatching vendor VFIO creates here would
activate the backend before assignment durability and release guards
exist. Reject vendor VFIO creates for now; destroy stays wired so
existing assignments remain releasable. The integration test skips on
vendor VFIO hosts at this layer and no longer asserts the transitional
stop-retention behavior.
Counting every free VF advertising a type overreports concurrent
capacity: sibling VFs share their parent GPU's framebuffer, so one 48Q
assignment revokes the type from every other VF on that GPU. Bound each
GPU's contribution by both its free VFs and how many times the profile
framebuffer fits into the GPU's remaining framebuffer, using the largest
still-creatable profile as a lower bound on what remains.
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 49cb877 to 1438015 Compare August 10, 2026 19:53
@yummybomb yummybomb closed this Aug 10, 2026
@yummybomb yummybomb reopened this Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant