Skip to content

fix: normalize firmware reference value collection across baremetal/Azure - #134

Merged
butler54 merged 3 commits into
validatedpatterns:mainfrom
butler54:feat/refval-collection-normalize
Aug 31, 2026
Merged

fix: normalize firmware reference value collection across baremetal/Azure#134
butler54 merged 3 commits into
validatedpatterns:mainfrom
butler54:feat/refval-collection-normalize

Conversation

@butler54

Copy link
Copy Markdown
Collaborator

Summary

Fixes #113. Addresses the veritas/Rekor reliability issue reported in production, the Azure "external" reference values gap, and normalizes several scripts into the Makefile.

1. Multi-arch reference value collection (Fixes #113)

scripts/collect-firmware-refvals.sh's --tee now defaults to both instead of tdx: it runs veritas once per TEE and merges the results into a single output. TDX/SNP reference values use disjoint, TEE-prefixed keys (confirmed via trustee-chart's RVPS template, which already passes through whichever TEE-specific keys are present), so the merge is a safe JSON union. Applies to both the azure and baremetal platform branches — this also means make collect-firmware-refvals/make collect-azure-refvals now naturally produce complete reference values for heterogeneous/mixed-TEE deployments (e.g. values-baremetal-hub.yaml's kbs.tdx.enabled + kbs.snp.enabled) without any manual merge step. Explicit --tee tdx/--tee snp remains available for single-arch collection.

2. Rekor verification failures

Traced the actual failure to veritas's own container.py: Red Hat's cosign public key is hardcoded (not overridable via any flag), and --rekor-url only changes which Rekor server is queried for a transparency-log entry. Red Hat signs and logs these images against its own private Rekor instance, so pointing at a different one (e.g. public Sigstore) would not find that entry and fail — not a viable workaround.

The only flag that actually avoids this while still verifying the real signature is --skip-tlog, which is not present in the veritas version (v0.1.2) vendored by the currently-pinned quay.io/openshift_sandboxed_containers/coco-tools:0.5.1 container (confirmed via that image's requirements.txt; --skip-tlog was added to veritas after v0.1.2 was tagged, and no newer coco-tools tag exists yet).

Fix: switched to a host-installed veritas CLI (pip install "osc-veritas[snp]==0.1.3rc1" — published on PyPI, confirmed via a real install + veritas --help that --skip-tlog is present) instead of the container. Default --skip-tlog on the Azure branch; new --verify-tlog flag opts back into full verification. New prerequisite checks for veritas and cosign >= 2.0 (Azure only — bare metal verifies via oc adm release info --verify, not cosign/Rekor at all).

This host-install is a deliberate, temporary deviation — filed #132 to track moving back to the container once coco-tools ships a release with a newer veritas, with the reasoning for why container-based is preferred long-term (encapsulation, less host-environment drift, reproducibility).

Also deleted scripts/get-pcr.sh: the legacy manual predecessor to veritas-based collection, which depended on the same private Rekor instance but never actually completed verification (its cosign verify call was already dead/commented-out code) — strictly worse than the current path, and explicitly superseded per the docs' own description.

3. Azure "external" reference values

trustee-chart's kbs.azure.* block (snpLaunchMeasurement, smtEnabled, tsmeEnabled, abiMajor, abiMinor, singleSocket, smtAllowed) — static Azure-platform/VM-series constants, not something veritas collects — had no override wiring in coco-pattern at all, silently inheriting whatever trustee-chart's own defaults happened to be. Added overrides/values-trustee-azure.yaml (wired into values-azure.yaml) pinning these explicitly, plus documentation of what they are, the current VM-series mapping, and the caveat that the corresponding policy checks are currently commented out upstream (dormant, not enforced).

Filed #133 to track the actual collection/verification tooling (SSH into podvm + snpguest --openhcl) — scoped out since it needs live Azure hardware to validate, which wasn't available while authoring this PR.

4. Makefile/script normalization

Surveyed scripts/*.sh for consistency with the existing convention (thin wrapper-to-script for nontrivial scripts, vs. embedded shell directly in the Makefile for simple one-offs like detect-hardware/cache-keys):

  • Inlined get-pccs-node.sh (10 lines, single oc query) into a new make get-pccs-node target; deleted the script.
  • Added missing wrapper targets for scripts that had none: make gen-secrets, make check-pck-expiry, make check-collateral-expiry (scripts stay — too much python-heredoc/multi-step logic to reasonably inline).

Test plan

  • Full scenario-test harness with stub veritas/cosign binaries: --tee both merge correctness (both azure and baremetal), single---tee opt-out, --hw-xfam-allow only added for the TDX run (not SNP), --skip-tlog on by default for azure only, --verify-tlog opt-out, invalid --tee/--platform rejected, missing-veritas/missing-or-old-cosign prerequisite errors, cosign not required for baremetal.
  • bash -n on the modified script; make -n dry-run on all new/changed Makefile targets; ran make get-pccs-node for real (fails gracefully without cluster access, as expected).
  • YAML validity of values-azure.yaml / overrides/values-trustee-azure.yaml.
  • markdownlint / textlint clean on all touched docs (matching the CI-pinned tool versions from prior sessions).
  • Live veritas/cosign invocations against real Azure/bare-metal clusters — not possible in this sandbox (no cluster/network image pull access). Will need validation against a real Azure cluster before merge.

Follow-ups filed

…ilures

Fixes validatedpatterns#113

scripts/collect-firmware-refvals.sh changes:

- --tee now defaults to 'both' instead of 'tdx': runs veritas once per
  TEE and merges the resulting reference values into a single output.
  TDX/SNP reference values use disjoint, TEE-prefixed keys, so the
  merge is a plain JSON union with no collision risk -- this matches
  how trustee-chart's RVPS template already consumes the
  firmwareReferenceValues/pcrStash secrets. Applies to both the azure
  and baremetal platform branches. Explicit --tee tdx/--tee snp
  remains available for single-arch collection.

- Switched from running veritas via the pinned
  quay.io/openshift_sandboxed_containers/coco-tools:0.5.1 container to
  a host-installed 'veritas' CLI (pip install osc-veritas[snp]).
  Traced the pinned image's requirements.txt to confirm it vendors
  veritas v0.1.2 (tagged before --skip-tlog/--cosign-pub-key/
  --mirror-registry existed, and before coco-tools published any newer
  tag) -- so the container could not be fixed without us building and
  maintaining our own image. osc-veritas is published on PyPI as of
  0.1.3rc1, confirmed via a real install that it exposes --skip-tlog.

- Default to --skip-tlog for the Azure branch. Verified in veritas's
  own container.py that the Red Hat cosign public key is hardcoded
  (not overridable), and --rekor-url only changes which Rekor server
  is queried for a transparency-log entry. Red Hat signs and logs
  these images against its own private Rekor instance
  (rekor-server-default.apps.rosa.rekor-prod...), so pointing at a
  different instance (e.g. public Sigstore) would not find that entry
  and fail -- it is not a viable workaround. --skip-tlog is the only
  flag that avoids the private instance while still verifying the
  actual cosign signature. New --verify-tlog flag opts back into full
  verification.

- New prerequisite checks: 'veritas' on PATH (clear pip install
  instructions if missing), and 'cosign' >= 2.0 for the Azure branch
  only (bare metal verifies via 'oc adm release info --verify', not
  cosign/Rekor at all).

- Deleted scripts/get-pcr.sh: the legacy, manual predecessor to
  veritas-based collection. It depended on the same private Rekor
  instance but never actually completed verification (its cosign
  verify call was already commented out/dead code) -- strictly worse
  than the current path, and explicitly superseded per
  docs/firmware-reference-values.md's own description.

docs/firmware-reference-values.md, README.md, rhdp/README.md updated
to reflect the new prerequisites (veritas via pip, cosign for Azure),
the --tee both default, and the host-install-instead-of-container
tradeoff (a tracking issue for reverting to the container once
coco-tools ships a newer veritas will be filed separately).

Verified with a full scenario-test harness using stub veritas/cosign
binaries: --tee both merge correctness (azure and baremetal), single
--tee opt-out, --hw-xfam-allow only added for the tdx run (not snp),
--skip-tlog on by default for azure only, --verify-tlog opt-out,
invalid --tee/--platform rejected, veritas/cosign-missing and
cosign-too-old prerequisite errors, and cosign not required at all
for the baremetal branch.
…tern

trustee-chart's values.yaml has a kbs.azure.* block (snpLaunchMeasurement,
smtEnabled, tsmeEnabled, abiMajor, abiMinor, singleSocket, smtAllowed)
describing the SEV-SNP policy Azure enforces for a given confidential
VM series. These are static platform constants, not measurements
veritas can collect from any artifact -- and coco-pattern previously
had no override wiring for them at all, silently inheriting whatever
trustee-chart's own defaults happened to be.

Add overrides/values-trustee-azure.yaml, wired into values-azure.yaml's
trustee extraValueFiles, pinning the current values explicitly so
they're visible/versioned in this repository instead of hidden inside
a dependency chart. The 'Azure External Reference Values' section
added to docs/firmware-reference-values.md in the previous commit
covers the current caveat that the corresponding Rego policy checks
are commented out upstream (dormant, not yet enforced) and points to
the follow-up snpguest-based collection script issue.
Surveyed scripts/*.sh for consistency with the existing Makefile
convention (thin wrapper-to-script for anything nontrivial, vs.
embedded shell directly in the Makefile for simple one-off logic like
detect-hardware/cache-keys/pck-register).

- Inlined get-pccs-node.sh (10 lines, a single oc query) directly into
  a new 'make get-pccs-node' target; deleted the script.
- Added missing wrapper targets for scripts that had none before,
  keeping the scripts themselves (too much python-heredoc/multi-step
  logic to reasonably inline): 'make gen-secrets', 'make
  check-pck-expiry', 'make check-collateral-expiry'.

scripts/collect-snp-vcek-urls.sh, download-snp-vcek.sh,
gen-snp-vcek-overrides.sh, collect-dcap-collateral.sh,
collect-firmware-refvals.sh, and airgap-post-install.sh are left as
scripts (already wrapped, and complex enough to warrant staying
separate).
@butler54
butler54 requested a review from a team August 31, 2026 06:01
@butler54
butler54 merged commit 1d356ac into validatedpatterns:main Aug 31, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: Support multiple architectures for TEEs reference value generation

1 participant