fix: normalize firmware reference value collection across baremetal/Azure - #134
Merged
butler54 merged 3 commits intoAug 31, 2026
Merged
Conversation
…ilures Fixes validatedpatterns#113 scripts/collect-firmware-refvals.sh changes: - --tee now defaults to 'both' instead of 'tdx': runs veritas once per TEE and merges the resulting reference values into a single output. TDX/SNP reference values use disjoint, TEE-prefixed keys, so the merge is a plain JSON union with no collision risk -- this matches how trustee-chart's RVPS template already consumes the firmwareReferenceValues/pcrStash secrets. Applies to both the azure and baremetal platform branches. Explicit --tee tdx/--tee snp remains available for single-arch collection. - Switched from running veritas via the pinned quay.io/openshift_sandboxed_containers/coco-tools:0.5.1 container to a host-installed 'veritas' CLI (pip install osc-veritas[snp]). Traced the pinned image's requirements.txt to confirm it vendors veritas v0.1.2 (tagged before --skip-tlog/--cosign-pub-key/ --mirror-registry existed, and before coco-tools published any newer tag) -- so the container could not be fixed without us building and maintaining our own image. osc-veritas is published on PyPI as of 0.1.3rc1, confirmed via a real install that it exposes --skip-tlog. - Default to --skip-tlog for the Azure branch. Verified in veritas's own container.py that the Red Hat cosign public key is hardcoded (not overridable), and --rekor-url only changes which Rekor server is queried for a transparency-log entry. Red Hat signs and logs these images against its own private Rekor instance (rekor-server-default.apps.rosa.rekor-prod...), so pointing at a different instance (e.g. public Sigstore) would not find that entry and fail -- it is not a viable workaround. --skip-tlog is the only flag that avoids the private instance while still verifying the actual cosign signature. New --verify-tlog flag opts back into full verification. - New prerequisite checks: 'veritas' on PATH (clear pip install instructions if missing), and 'cosign' >= 2.0 for the Azure branch only (bare metal verifies via 'oc adm release info --verify', not cosign/Rekor at all). - Deleted scripts/get-pcr.sh: the legacy, manual predecessor to veritas-based collection. It depended on the same private Rekor instance but never actually completed verification (its cosign verify call was already commented out/dead code) -- strictly worse than the current path, and explicitly superseded per docs/firmware-reference-values.md's own description. docs/firmware-reference-values.md, README.md, rhdp/README.md updated to reflect the new prerequisites (veritas via pip, cosign for Azure), the --tee both default, and the host-install-instead-of-container tradeoff (a tracking issue for reverting to the container once coco-tools ships a newer veritas will be filed separately). Verified with a full scenario-test harness using stub veritas/cosign binaries: --tee both merge correctness (azure and baremetal), single --tee opt-out, --hw-xfam-allow only added for the tdx run (not snp), --skip-tlog on by default for azure only, --verify-tlog opt-out, invalid --tee/--platform rejected, veritas/cosign-missing and cosign-too-old prerequisite errors, and cosign not required at all for the baremetal branch.
…tern trustee-chart's values.yaml has a kbs.azure.* block (snpLaunchMeasurement, smtEnabled, tsmeEnabled, abiMajor, abiMinor, singleSocket, smtAllowed) describing the SEV-SNP policy Azure enforces for a given confidential VM series. These are static platform constants, not measurements veritas can collect from any artifact -- and coco-pattern previously had no override wiring for them at all, silently inheriting whatever trustee-chart's own defaults happened to be. Add overrides/values-trustee-azure.yaml, wired into values-azure.yaml's trustee extraValueFiles, pinning the current values explicitly so they're visible/versioned in this repository instead of hidden inside a dependency chart. The 'Azure External Reference Values' section added to docs/firmware-reference-values.md in the previous commit covers the current caveat that the corresponding Rego policy checks are commented out upstream (dormant, not yet enforced) and points to the follow-up snpguest-based collection script issue.
Surveyed scripts/*.sh for consistency with the existing Makefile convention (thin wrapper-to-script for anything nontrivial, vs. embedded shell directly in the Makefile for simple one-off logic like detect-hardware/cache-keys/pck-register). - Inlined get-pccs-node.sh (10 lines, a single oc query) directly into a new 'make get-pccs-node' target; deleted the script. - Added missing wrapper targets for scripts that had none before, keeping the scripts themselves (too much python-heredoc/multi-step logic to reasonably inline): 'make gen-secrets', 'make check-pck-expiry', 'make check-collateral-expiry'. scripts/collect-snp-vcek-urls.sh, download-snp-vcek.sh, gen-snp-vcek-overrides.sh, collect-dcap-collateral.sh, collect-firmware-refvals.sh, and airgap-post-install.sh are left as scripts (already wrapped, and complex enough to warrant staying separate).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #113. Addresses the veritas/Rekor reliability issue reported in production, the Azure "external" reference values gap, and normalizes several scripts into the Makefile.
1. Multi-arch reference value collection (Fixes #113)
scripts/collect-firmware-refvals.sh's--teenow defaults tobothinstead oftdx: it runsveritasonce per TEE and merges the results into a single output. TDX/SNP reference values use disjoint, TEE-prefixed keys (confirmed viatrustee-chart's RVPS template, which already passes through whichever TEE-specific keys are present), so the merge is a safe JSON union. Applies to both theazureandbaremetalplatform branches — this also meansmake collect-firmware-refvals/make collect-azure-refvalsnow naturally produce complete reference values for heterogeneous/mixed-TEE deployments (e.g.values-baremetal-hub.yaml'skbs.tdx.enabled+kbs.snp.enabled) without any manual merge step. Explicit--tee tdx/--tee snpremains available for single-arch collection.2. Rekor verification failures
Traced the actual failure to veritas's own
container.py: Red Hat's cosign public key is hardcoded (not overridable via any flag), and--rekor-urlonly changes which Rekor server is queried for a transparency-log entry. Red Hat signs and logs these images against its own private Rekor instance, so pointing at a different one (e.g. public Sigstore) would not find that entry and fail — not a viable workaround.The only flag that actually avoids this while still verifying the real signature is
--skip-tlog, which is not present in the veritas version (v0.1.2) vendored by the currently-pinnedquay.io/openshift_sandboxed_containers/coco-tools:0.5.1container (confirmed via that image'srequirements.txt;--skip-tlogwas added to veritas afterv0.1.2was tagged, and no newercoco-toolstag exists yet).Fix: switched to a host-installed
veritasCLI (pip install "osc-veritas[snp]==0.1.3rc1"— published on PyPI, confirmed via a real install +veritas --helpthat--skip-tlogis present) instead of the container. Default--skip-tlogon the Azure branch; new--verify-tlogflag opts back into full verification. New prerequisite checks forveritasandcosign >= 2.0(Azure only — bare metal verifies viaoc adm release info --verify, not cosign/Rekor at all).This host-install is a deliberate, temporary deviation — filed #132 to track moving back to the container once
coco-toolsships a release with a newer veritas, with the reasoning for why container-based is preferred long-term (encapsulation, less host-environment drift, reproducibility).Also deleted
scripts/get-pcr.sh: the legacy manual predecessor to veritas-based collection, which depended on the same private Rekor instance but never actually completed verification (itscosign verifycall was already dead/commented-out code) — strictly worse than the current path, and explicitly superseded per the docs' own description.3. Azure "external" reference values
trustee-chart'skbs.azure.*block (snpLaunchMeasurement,smtEnabled,tsmeEnabled,abiMajor,abiMinor,singleSocket,smtAllowed) — static Azure-platform/VM-series constants, not something veritas collects — had no override wiring in coco-pattern at all, silently inheriting whatevertrustee-chart's own defaults happened to be. Addedoverrides/values-trustee-azure.yaml(wired intovalues-azure.yaml) pinning these explicitly, plus documentation of what they are, the current VM-series mapping, and the caveat that the corresponding policy checks are currently commented out upstream (dormant, not enforced).Filed #133 to track the actual collection/verification tooling (SSH into podvm +
snpguest --openhcl) — scoped out since it needs live Azure hardware to validate, which wasn't available while authoring this PR.4. Makefile/script normalization
Surveyed
scripts/*.shfor consistency with the existing convention (thin wrapper-to-script for nontrivial scripts, vs. embedded shell directly in the Makefile for simple one-offs likedetect-hardware/cache-keys):get-pccs-node.sh(10 lines, singleocquery) into a newmake get-pccs-nodetarget; deleted the script.make gen-secrets,make check-pck-expiry,make check-collateral-expiry(scripts stay — too much python-heredoc/multi-step logic to reasonably inline).Test plan
veritas/cosignbinaries:--tee bothmerge correctness (both azure and baremetal), single---teeopt-out,--hw-xfam-allowonly added for the TDX run (not SNP),--skip-tlogon by default for azure only,--verify-tlogopt-out, invalid--tee/--platformrejected, missing-veritas/missing-or-old-cosignprerequisite errors, cosign not required for baremetal.bash -non the modified script;make -ndry-run on all new/changed Makefile targets; ranmake get-pccs-nodefor real (fails gracefully without cluster access, as expected).values-azure.yaml/overrides/values-trustee-azure.yaml.veritas/cosigninvocations against real Azure/bare-metal clusters — not possible in this sandbox (no cluster/network image pull access). Will need validation against a real Azure cluster before merge.Follow-ups filed
coco-toolsships a newer release (with rationale for why that's the preferred long-term state)snpguest --openhcl