Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .github/ISSUE_TEMPLATE/bug-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,30 @@ assignees: ''
---

### Description

A clear and concise description of the bug.

### Steps to Reproduce
1. Install the pattern

1. Install the pattern
2. Go to X
3. See Y


### Expected Behavior

Explain what you expected to happen.

### Screenshots

Add screenshots if applicable.

### Environment

- Hardware / cloud provide (e.g. Azure; Intel Barel metal; AMD Bare metal)
- CPU Specs and/or cloud instance types.
- Cluster topology (one cluster; SNO; multi-cluster)
- Connected or disconnected

### Additional Context

Add any other context about the problem here.
6 changes: 5 additions & 1 deletion .github/ISSUE_TEMPLATE/documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,17 @@ assignees: ''
---

### Documentation Update
What part of the documentation needs to be updated or added? Is it for the validated patterns docs site or the repo docs.

What part of the documentation needs to be updated or added? Is it for the validated patterns docs site or the repository docs.

### Why Is This Needed?

Explain the importance of this update.

### Suggested Changes

Provide a detailed description of the changes.

### Additional Context

Include any related resources.
5 changes: 5 additions & 0 deletions .github/ISSUE_TEMPLATE/feature-request.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,21 @@ assignees: ''
---

### Feature Description

What feature would you like to see?

### Why Is This Needed?

Explain the problem or need for this feature.

### Suggested Solutions

Describe how this feature could be implemented.

### Proposed Acceptance criteria

Discrete acceptance criteria, suitable for evaluation by a LLM

### Additional Context

Add any relevant screenshots, links, or resources.
4 changes: 4 additions & 0 deletions .github/workflows/superlinter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@ jobs:
VALIDATE_MARKDOWN_PRETTIER: false
VALIDATE_KUBERNETES_KUBECONFORM: false
VALIDATE_PYTHON_PYLINT: false
# Pyink and Black conflict (super-linter warns against enabling both).
# This repo formats Python with Black (.flake8 targets Black's line
# length), and no other validatedpatterns/* repo enables Pyink.
VALIDATE_PYTHON_PYINK: false
VALIDATE_SHELL_SHFMT: false
VALIDATE_YAML: false
VALIDATE_YAML_PRETTIER: false
Expand Down
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,13 @@ Breaking change from v5. Upgrades to OSC 1.13 / Trustee 1.2, adds full airgap (d
- **8.0** — OSC 1.13 / Trustee 1.2 upgrade. KBS TOML rewrite, RVPS format alignment, operator CSV pins, trustee-chart v0.10.0. Airgap deployment support: single mirror-registry architecture, `oc-mirror` v2, `airgap-post-install.sh` bootstrap, `DEPLOY-RUNBOOK.md` operational guide. Requires OCP 4.22+.

**Key changes from v5:**

- **Operator upgrade:** OSC 1.12 → 1.13, Trustee 1.1 → 1.2. Breaking API changes in KBS configuration and RVPS reference value format.
- **Airgap support:** Full disconnected deployment on bare metal via a single mirror-registry. Includes `oc-mirror` imageset configs, `airgap-post-install.sh` for bootstrap (CatalogSources, IDMS/ITMS, NeverContactSource policy normalisation, ArgoCD CA injection, git HTTP server), and a step-by-step operational runbook at `airgap/DEPLOY-RUNBOOK.md`.
- **Airgap support:** Full disconnected deployment on bare metal via a single mirror-registry. Includes `oc-mirror` imageset configs, `airgap-post-install.sh` for bootstrap (CatalogSources, IDMS/ITMS, NeverContactSource policy normalisation, ArgoCD CA injection, Git HTTP server), and a step-by-step operational runbook at `airgap/DEPLOY-RUNBOOK.md`.
- **Chart architecture:** trustee-chart moved to OCI Helm artifact (`quay.io/validatedpatterns/trustee:0.10.0`). Kyverno chart vendored under `charts/vendor/`. External charts (sandboxed-containers, sandboxed-policies) remain OCI.
- **TDX MachineConfig:** QGS socket port `socket_port=0` now deployed via MachineConfig drop-in (replaces manual `sed` workaround).
- **DCAP collateral:** `collect-dcap-collateral.sh` uses `pcsclient.py fetch -p E5` with `jq` fixup for QeIdentity (Red Hat OSC 1.13 disconnected TDX procedure).
- **Experimental:** KubeVirt TDX confidential VMs (`charts/all/kubevirtconfidential/`, `charts/all/kubevirtvm/`). Disabled by default. Requires Intel TDX hardware and KubeVirt post-v1.8.4 for full QGS attestation. See chart READMEs for details.
- **Experimental:** KubeVirt TDX confidential VMs (`charts/all/kubevirtconfidential/`, `charts/all/kubevirtvm/`). Disabled by default. Requires Intel TDX hardware and KubeVirt post-v1.8.4 for full QGS attestation. See chart readmes for details.

### Previous versions

Expand All @@ -53,14 +54,16 @@ Breaking change from v5. Upgrades to OSC 1.13 / Trustee 1.2, adds full airgap (d
### Airgap (disconnected) deployment

For air-gapped bare metal environments, see [`airgap/DEPLOY-RUNBOOK.md`](airgap/DEPLOY-RUNBOOK.md) for the full operational procedure. The runbook covers:
- Phase 0: One-time jump host setup (mirror registry, git HTTP server, CA certificates)

- Phase 0: One-time jump host setup (mirror registry, Git HTTP server, CA certificates)
- Phase A–B: Mirror wipe and `oc-mirror` v2 re-mirror
- Phase C: Agent-based OCP install
- Phase D: Pattern bootstrap (`airgap-post-install.sh`)
- Phase E: Intel DCAP collateral and TDX attestation
- Phase F: Verification and pass/fail checklist

**Prerequisites for airgap:**

- A jump host with internet access (for `oc-mirror`) and network access to the target cluster
- `docker.io/library/registry:2` container running as the mirror registry (setup documented in Phase 0)
- `scripts/git-http-server.py` serving pattern repos over smart HTTP (required by the patterns-operator's go-git client)
Expand Down Expand Up @@ -127,7 +130,7 @@ These scripts generate the cryptographic material and attestation reference valu
- Options: `intel-tdx`, `amd-snp`, `intel-tdx-gpu`, `amd-snp-gpu`
3. Run `bash scripts/gen-secrets.sh` to generate KBS keys and PCCS secrets
4. `./pattern.sh make install`
6. Wait for the cluster to reboot nodes (MachineConfig updates for TDX/SEV-SNP kernel parameters and vsock)
5. Wait for the cluster to reboot nodes (MachineConfig updates for TDX/SEV-SNP kernel parameters and vsock)

> **Note:** Bare metal support is currently tested on SNO (Single Node OpenShift) configurations. Multi-node bare metal clusters are expected to work but have not been validated yet.

Expand Down
14 changes: 9 additions & 5 deletions airgap/DEPLOY-RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ EOF

### 0-3: Local Git Server Initialisation (ONE-TIME per jump host)

The local git HTTP server makes pattern repos available to the cluster's patterns-operator
The local Git HTTP server makes pattern repos available to the cluster's patterns-operator
during the deploy. `scripts/airgap-post-install.sh --sync-repos-only` creates bare repos
and starts the service; subsequent runs auto-sync via the D-1 step.

Expand Down Expand Up @@ -387,7 +387,6 @@ python3 -c "import json; [print(k) for k in json.load(open('pull-secret.json'))[
echo "A-1: State verification complete at $(date)" 2>&1 | tee -a "$LOG"
```


### A-3: oc-mirror Workspace Wipe

> **DESTRUCTIVE — confirm A-1 shows correct git branch and mirror-registry state before proceeding.**
Expand Down Expand Up @@ -743,7 +742,6 @@ ls -lh ~/public_html/node-02-42208.iso 2>&1 | tee -a "$LOG" && \
echo "C-1: Config generation and ISO build complete at $(date)" 2>&1 | tee -a "$LOG"
```


### C-3: Mount ISO via iDRAC

```bash
Expand Down Expand Up @@ -835,6 +833,7 @@ echo "C-6: KUBECONFIG configured at $(date)" 2>&1 | tee -a "$LOG"
### D-1: Run airgap-post-install.sh

> `airgap-post-install.sh` handles all of the following in one pass:
>
> - Disables OperatorHub default catalogs
> - Applies CatalogSources from oc-mirror cluster-resources (deleting stale ones)
> - Applies ALL oc-mirror cluster-resources: IDMS, ITMS, ClusterCatalog (OLM v1), signature ConfigMap
Expand Down Expand Up @@ -956,6 +955,7 @@ echo "D-4: pattern.sh complete at $(date)" 2>&1 | tee -a "$LOG"
> If you delay, ArgoCD may come up and fail to pull VP OCI Helm charts before credentials are loaded.
>
> Steps D-5 performs once vp-gitops appears:
>
> 1. Inject mirror-registry private CA into ArgoCD's TLS trust store
> 2. `make load-bootstrap` — pre-seeds ArgoCD with mirror-registry OCI Helm credentials
> (from the `bootstrap_secrets` block in values-secret.yaml; no vault required)
Expand Down Expand Up @@ -1391,7 +1391,7 @@ echo "=== Run Complete: $(date) ===" 2>&1 | tee -a "$LOG"
| Criterion | Expected | Actual |
|-----------|----------|--------|
| patterns-operator CSV Succeeded | Via OLM from mirror-registry community catalog | |
| Pattern CR reconciles | targetRepo = HTTP git URL (not GitHub) | |
| Pattern CR reconciles | targetRepo = HTTP Git URL (not GitHub) | |
| Vault init without manual SA creation | Sync-wave fix still working | |
| 15/15 (or current count) ArgoCD apps Synced+Healthy | All green | |
| KBS attestation affirming | POST /attest 200 | |
Expand All @@ -1402,7 +1402,7 @@ echo "=== Run Complete: $(date) ===" 2>&1 | tee -a "$LOG"
| secret.txt contains KBS resource | Fetched via CDH from KBS | |
| All images from mirror-registry | No upstream refs | |
| MCH Running | ACM + MCE healthy | |
| QGS socket_port=0 WITHOUT manual sed | D-06 — MCO fix working (new row) |
| QGS socket_port=0 WITHOUT manual sed | D-06 — MCO fix working (new row) | |
| No mirrorSourcePolicy conflicts after D-1.5 | PASS: all IDMS/ITMS have NeverContactSource (new row) | |
| scripts/deprecated/ untouched — 3 files, none invoked | D-07 — no deprecated scripts called (new row) | |
| 2-line global catalogSource overlay active in values-baremetal-airgap.yaml | D-07 — Phase 24 C-05 migration active (new row) | |
Expand Down Expand Up @@ -1492,10 +1492,12 @@ SELinux note: htpasswd file requires `container_file_t` label. If recreating, ru
`chcon -t container_file_t ~/mirror-registry-config/htpasswd`

CA trust for container tools (podman, oc-mirror, skopeo):

```bash
mkdir -p ~/.config/containers/certs.d/${MIRROR_REGISTRY}
cp ~/mirror-registry-certs/ca.crt ~/.config/containers/certs.d/${MIRROR_REGISTRY}/ca.crt
```

This is idempotent and is run automatically in A-0. It enables `--dest-tls-verify=true` in oc-mirror
and removes the need for `--tls-verify=false` in podman login.

Expand Down Expand Up @@ -1553,12 +1555,14 @@ export MREG_PASS=$(cat ~/.coco-pattern/mirror-registry-password)
```

Auto-set during run:

```bash
export KUBECONFIG=<set in C-6 from build dir>
export MIRROR_REGISTRY=MIRROR_REGISTRY_HOST:8443
```

Persistent in `~/.envrc`:

```bash
export LABCTL_IDRAC_USER=chbutler
export LABCTL_IDRAC_PASSWORD="..."
Expand Down
2 changes: 1 addition & 1 deletion charts/all/kubevirtconfidential/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,6 @@ When enabled, the chart creates:
1. **HyperConverged CR patch** -- enables the `WorkloadEncryptionSEV` feature gate
(which also covers TDX on Intel platforms)
2. **SELinux MachineConfig** -- installs a custom SELinux policy module
(`kubevirt-qgs`) allowing QEMU processes to connect to the QGS unix socket
(`kubevirt-qgs`) allowing QEMU processes to connect to the QGS Unix socket
3. **RHEL 9 ImageStream import job** -- imports the RHEL 9 guest image into the
internal registry for VM boot via DataImportCron
2 changes: 1 addition & 1 deletion charts/all/kubevirtvm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,5 +44,5 @@ When enabled with `global.kubevirt.vm.enabled: "true"`:

When additionally enabled with `global.kubevirt.vm.tdx.enabled: "true"`:

2. **TDX Confidential VM** -- a RHEL 9 VM with TDX launch security enabled,
1. **TDX Confidential VM** -- a RHEL 9 VM with TDX launch security enabled,
validating end-to-end confidential compute inside KubeVirt
26 changes: 18 additions & 8 deletions docs/disconnected-deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Deploy the CoCo validated pattern on a disconnected OpenShift cluster using mirr

## Architecture

```
```text
Internet ──► Jump Host ──► Disconnected Network
│ │
quay.io (source) quay-mirror (dest)
Expand All @@ -21,10 +21,11 @@ Internet ──► Jump Host ──► Disconnected Network
```

**Content flows:**

1. `oc-mirror` mirrors OCP release + operator catalogs + container images to Quay
2. `airgap-post-install.sh` mirrors OCI Helm charts (oc-mirror can't handle these)
3. `git-http-server.py` serves pattern git repos over smart HTTP (go-git requires this)
4. Pattern operator clones from git server, ArgoCD pulls Helm charts from Quay
3. `git-http-server.py` serves pattern Git repos over smart HTTP (go-git requires this)
4. Pattern operator clones from Git server, ArgoCD pulls Helm charts from Quay

## Step-by-Step Deployment

Expand All @@ -41,6 +42,7 @@ make airgap-mirror \
```

**What gets mirrored:**

- OCP 4.21.24 release images
- Red Hat operators: gitops, sandboxed-containers, trustee, cert-manager, ACM, LVM, CNV, NFD, intel-device-plugins
- Certified operators: gpu-operator
Expand All @@ -49,7 +51,7 @@ make airgap-mirror \

**Known limitation:** `oc-mirror --v2` cannot mirror OCI Helm chart artifacts via `additionalImages`. The post-install script handles these separately.

**Known limitation:** `oc-mirror --v2` fails on images published as OCI image indexes (`application/vnd.oci.image.index.v1+json`) with "Manifest list must be converted" error. Currently only affects `patterns-operator`. See [patterns-operator#774](https://github.com/validatedpatterns/patterns-operator/issues/774). Workaround: deploy operator manually (automated in bootstrap script).
**Known limitation:** `oc-mirror --v2` fails on images published as OCI image indices (`application/vnd.oci.image.index.v1+json`) with "Manifest list must be converted" error. Currently only affects `patterns-operator`. See [patterns-operator#774](https://github.com/validatedpatterns/patterns-operator/issues/774). Workaround: deploy operator manually (automated in bootstrap script).

### 2. Install OpenShift

Expand Down Expand Up @@ -100,15 +102,15 @@ make airgap-post-install

| Step | Action |
|------|--------|
| 1 | Validate prerequisites (oc, git, KUBECONFIG, MIRROR_REGISTRY) |
| 1 | Validate prerequisites (oc, Git, KUBECONFIG, MIRROR_REGISTRY) |
| 2 | Disable default CatalogSources (prevent OLM reaching internet) |
| 3 | Create mirrored CatalogSources (redhat, certified, community) |
| 4 | Create ITMS for tag-based image pulls (ubi-minimal, VP images) |
| 5 | Mirror OCI Helm charts that oc-mirror can't handle |
| 6 | Fix oc-mirror manifest list failures (skopeo fallback) |
| 7 | Add mirror CA cert to ArgoCD TLS config |
| 8 | Enable OVN routingViaHost (opt-in, lab networks only) |
| 9 | Set up smart HTTP git server for pattern repos |
| 9 | Set up smart HTTP Git server for pattern repos |
| 10 | Create patterns-operator-config ConfigMap (GitOps channel override) |

### 5. Install patterns-operator
Expand Down Expand Up @@ -195,35 +197,43 @@ The following Quay repos **must be public** for ArgoCD to pull OCI Helm charts (
| `make airgap-mirror` | Mirror content to disconnected registry |
| `make airgap-post-install` | Run full post-install bootstrap |
| `make airgap-deploy-pattern` | Deploy Pattern CR directly |
| `make airgap-sync-repos` | Sync git working copies to bare HTTP repos |
| `make airgap-sync-repos` | Sync Git working copies to bare HTTP repos |

## Troubleshooting

### "manifest unknown" during OCP install

The `openshift-install` binary version must match the mirrored release. Rebuild the ISO if certificates expired (>24 hours old).

### ArgoCD apps stuck at "Unknown" sync

Check if Helm chart repos are public on Quay. Verify with:

```bash
curl -sk https://quay.example.com:443/v2/mirror/validatedpatterns/clustergroup/tags/list
# Should return 200 with JSON, not 401
```

### "unexpected EOF" from patterns-operator git clone
### "unexpected EOF" from patterns-operator Git clone

The patterns-operator uses go-git which doesn't support Apache dumb HTTP. Use `git-http-server.py` (smart HTTP via `git-http-backend` CGI) on port 8080.

### GitOps operator subscription wrong channel

The patterns-operator hardcodes the GitOps channel. The bootstrap script creates a `patterns-operator-config` ConfigMap to override it to `latest` (the only channel available in the mirrored catalog).

### Vault ImagePullBackOff

The VP Helm chart references `registry.connect.redhat.com/hashicorp/vault:VERSION-ubi`. Add an ITMS entry for `registry.connect.redhat.com/hashicorp` and mirror the image:

```bash
oc image mirror --insecure=true \
registry.connect.redhat.com/hashicorp/vault:1.21.4-ubi \
quay.example.com:443/mirror/hashicorp/vault:1.21.4-ubi
```

### OVN routingViaHost

Only needed when the OVN default gateway can't route to the jump host subnet. This is lab-specific — in a properly routed network, pods can reach the git/registry servers without it. Set `ENABLE_ROUTINGVIAHOST=true` before running the bootstrap.

## Known Issues
Expand Down
31 changes: 31 additions & 0 deletions rhdp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,34 @@ The wrapper handles: cluster provisioning, secret generation, PCR reference valu

1. `bash ./rhdp/wrapper-cluster-only.sh eastasia`
2. Provisions the cluster without installing secrets or the pattern

## Re-running against an existing install directory

All three wrapper scripts (and `rhdp/rhdp-cluster-define.py` directly) refuse
to touch an install directory (e.g. `openshift-install`,
`openshift-install-hub`) that already has cluster state (`metadata.json`)
from a previous run. This prevents accidentally wiping the local record of a
still-live cluster and silently replacing it with a new one.

If you see an error about existing cluster install state, you have two
options:

1. **Destroy the existing cluster's cloud resources yourself first**, then
re-run the wrapper normally:

```shell
openshift-install destroy cluster --dir=./openshift-install
```

2. **Pass `--recreate`** if you've already confirmed the cloud resources are
gone (or were never fully created):

```shell
bash ./rhdp/wrapper.sh --recreate eastasia
```

**`--recreate` does NOT call `openshift-install destroy cluster` for
you.** It only wipes the local install directory so a fresh install can
proceed. If the previous cluster's cloud resources are still live, they
will be orphaned (left running in Azure, unmanaged) — verify and clean
those up manually via the Azure portal/CLI if needed.
Loading
Loading