diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index d07bf1b6c8..707e1ca674 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -395,12 +395,83 @@ The shared state directory should preserve `sandbox_gid` inheritance `@openshell-sidecar-ssh`; the network sidecar verifies its peer PID before bridging gateway relay requests. No `ssh.sock` file should appear in the shared state directory. + +If `topology = "cni-sidecar"` is rendered, the gateway should render +the same process container and long-running network sidecar as sidecar mode, but +there should be no `openshell-network-init` init container in sandbox pods. +Instead, the chart must install the privileged `openshell-cni` DaemonSet and the +sandbox pod should carry `openshell.ai/cni=enabled`, +`openshell.ai/network-enforcement-mode=cni-sidecar`, and +`openshell.ai/proxy-uid=` annotations. The CNI DaemonSet copies +`/openshell-cni` into the host CNI binary directory and patches an existing CNI +`.conflist`; if sandbox pods bypass network enforcement or fail during pod +network setup, inspect the DaemonSet logs, the host CNI config, and whether the +cluster actually invokes chained CNI plugins for the sandbox runtime class. + +The CNI installer is a **cluster singleton** with a fixed identity +(`openshell-cni`). It is not necessarily in the release you are debugging — the +owner is the release with `cni.enabled=true`; other gateway releases set +`cni.external=true` and reuse it. Locate the owner and its resources cluster-wide: + +```bash +kubectl get daemonset -A -l app.kubernetes.io/name=openshell-cni +kubectl get clusterrole,clusterrolebinding openshell-cni +``` + +A gateway release using `cni-sidecar` with neither `cni.enabled` nor +`cni.external` fails to render (template error), so if the gateway installed but +sandboxes are unenforced, confirm the singleton exists and is Ready. + +**Namespace allowlist (silent unenforced sandboxes).** The plugin only inspects +pods whose namespace is in its `sandboxNamespaces` allowlist. That allowlist is +built automatically from Helm-owned marker ConfigMaps +(`openshell.ai/cni-registration=true`) — one per cni-sidecar release, in its +sandbox namespace — unioned with the static `cni.sandboxNamespaces`. If a sandbox +reaches 2/2 but its egress is NOT blocked, confirm its namespace is registered and +in the installed allowlist: + +```bash +# Registration markers (their namespaces are what the singleton enforces): +kubectl get configmaps -A -l openshell.ai/cni-registration=true +# Installed plugin config's aggregated sandboxNamespaces: +kubectl -n exec ds/openshell-cni -c install-cni -- \ + sh -c 'cat /host/run/multus/cni/net.d/vendor-cni-chain/openshell-cni.conf 2>/dev/null \ + || cat /host/etc/cni/net.d/*.conflist' +``` + +**Gateway stuck in Init (`wait-cni-coverage`).** A cni-sidecar gateway pod has a +`wait-cni-coverage` init container that blocks until every `cni-ready` node's +`openshell.ai/cni-sandbox-namespaces` annotation includes the gateway's namespace. +If the gateway is stuck initializing, the singleton has not yet acknowledged the +namespace on all nodes — check the per-node coverage: +`kubectl get nodes -o custom-columns=NAME:.metadata.name,COVERAGE:.metadata.annotations.openshell\.ai/cni-sandbox-namespaces` +and confirm a marker ConfigMap exists for that namespace. + +**Node reboot / boot taint.** In `multus-chain` mode the chain file lives under +`/run` (tmpfs) and is wiped on reboot while the `cni-ready` label persists. +Clusters that apply the optional boot-time taint (`openshell.ai/cni-not-ready`, +see `deploy/helm/openshell/examples/`) will show it on a node until the installer +removes it; a node stuck with the taint means the installer has not reached ready +there. Check with `kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints`. + +A per-node scheduling gate can also keep cni-sidecar sandbox pods `Pending`. The +CNI DaemonSet labels each node `openshell.ai/cni-ready=true` after installing the +plugin, and the gateway sets a required `nodeAffinity` on that label. If a +sandbox pod stays `Pending` with an "unmatched nodeAffinity" event, check that +the CNI DaemonSet is Ready on schedulable nodes and that the label is present +(`kubectl get nodes -L openshell.ai/cni-ready`). A missing label means the +installer has not completed its first patch or a reconcile tick cleared it after +a plugin-restore failure. + Inspect all three when sandbox registration or egress enforcement fails: ```bash kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^\[openshell\.drivers\.kubernetes\]|^topology\s*=' kubectl -n get pod -o jsonpath='{range .spec.initContainers[*]}{.name}{" "}{.command}{"\n"}{end}' kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}' +kubectl -n get pod -o jsonpath='{.metadata.annotations}' +kubectl -n openshell get daemonset,pod -l app.kubernetes.io/component=cni +kubectl -n openshell logs daemonset/openshell-cni -c install-cni --tail=200 kubectl -n logs -c openshell-network-init --tail=200 kubectl -n logs -c openshell-supervisor-network --tail=200 kubectl -n logs -c agent --tail=200 @@ -446,6 +517,7 @@ openshell logs | HTTP request returns `middleware_failed` or `middleware_denied` | Selected stage failed or explicitly denied the admitted request | Sandbox OCSF logs; policy-local middleware config; service availability; `on_error` | | Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or configured with a reserved/mismatched name | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs | | Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials | +| CNI-sidecar sandbox pods fail network setup | OpenShell CNI DaemonSet did not patch the node CNI conflist, cannot read pods, or the runtime class does not invoke the chained plugin | `kubectl -n openshell logs daemonset/openshell-cni -c install-cni`, chart `cni.*` values, host CNI config | | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled | | HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://` | | HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https:// --oidc-issuer `, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster | diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index adb40a9575..bc5d795115 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -70,13 +70,20 @@ mise run helm:skaffold:run:sidecar mise run helm:skaffold:run:sidecar-mtls ``` -Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm +**Supervisor CNI-sidecar topology** (build once and leave running): +```bash +mise run helm:skaffold:run:cni-sidecar +``` + +These commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm chart. The sidecar profile renders an `openshell-network-init` init container for nftables setup and an `openshell-supervisor-network` runtime sidecar for proxying. Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and `DAC_READ_SEARCH`; relaxed mode can run it as the configured proxy UID. The sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores -`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install +`server.disableTls=false` inline for Skaffold. The cni-sidecar profile enables +the privileged OpenShell CNI DaemonSet and uses the sidecar runtime model +without the pod-local network init container. The `pkiInitJob` hook (a pre-install Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first install. Envoy Gateway opt-in; see the Optional Add-ons section below. @@ -87,6 +94,30 @@ The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or create the Secret named `openshell-ha-pg` with a `uri` key, then run `mise run helm:skaffold:run` or `mise run helm:skaffold:dev`. +### Kubernetes e2e profiles + +Run the default Kubernetes e2e environment: + +```bash +mise run e2e:kubernetes +``` + +Run the sidecar topology e2e environment: + +```bash +mise run e2e:kubernetes:sidecar +``` + +Run the CNI-sidecar topology e2e environment: + +```bash +mise run e2e:kubernetes:cni-sidecar +``` + +The cni-sidecar e2e task applies `ci/values-cni-sidecar.yaml` through +`OPENSHELL_E2E_KUBE_EXTRA_VALUES` and requires OpenShell CNI installer +permissions on the target cluster's nodes. + ### TLS behaviour `ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run @@ -149,6 +180,12 @@ For a sidecar-profile deployment: mise run helm:skaffold:delete:sidecar ``` +For a cni-sidecar-profile deployment: + +```bash +mise run helm:skaffold:delete:cni-sidecar +``` + ### Delete the cluster entirely ```bash @@ -274,6 +311,7 @@ for dependencies still declared in `Chart.yaml`. | `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) | | `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay | | `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev | +| `deploy/helm/openshell/ci/values-cni-sidecar.yaml` | Supervisor CNI-sidecar topology overlay for Kubernetes e2e/dev; enables the OpenShell CNI DaemonSet | | `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay | | `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev | | `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) | diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index d0d600e58b..1788d60492 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -246,6 +246,9 @@ jobs: fi mkdir -p "$stage" install -m 0755 "$found" "$stage/$binary" + if [[ "${{ inputs.component }}" == "supervisor" ]]; then + PREBUILT_ARCH="${{ matrix.arch }}" tasks/scripts/stage-prebuilt-binaries.sh cni + fi ls -lh "$stage/" - name: Build ${{ inputs.component }} image diff --git a/AGENTS.md b/AGENTS.md index 540a8d9207..44fa97c924 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,6 +33,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files | `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary | | `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing | | `crates/openshell-policy/` | Policy engine | Filesystem, network, process, and inference constraints | +| `crates/openshell-cni/` | Chained CNI plugin | Node-level egress bypass-prevention for the cni-sidecar topology; also sets the per-node readiness label | | `crates/openshell-router/` | Privacy router | Privacy-aware LLM routing | | `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage | | `crates/openshell-gateway-interceptors/` | Gateway interceptors | Intercepts and transforms configured gRPC requests at the gateway routing boundary | diff --git a/Cargo.lock b/Cargo.lock index 528309a970..6036c06680 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3862,6 +3862,20 @@ dependencies = [ "url", ] +[[package]] +name = "openshell-cni" +version = "0.0.0" +dependencies = [ + "base64 0.22.1", + "libc", + "miette", + "reqwest 0.12.28", + "serde", + "serde_json", + "serde_yml", + "tempfile", +] + [[package]] name = "openshell-core" version = "0.0.0" diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 55d7e7a29d..2cf3b6f6bd 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -241,6 +241,108 @@ already unprivileged. Sidecar pods use a shared process namespace so the network sidecar can resolve workload process and binary identity through `/proc/`. +The cni-sidecar topology keeps the sidecar runtime model and its shared-state +boundary, but removes the privileged `openshell-network-init` init container and +its `NET_ADMIN`. Instead, the privileged OpenShell CNI DaemonSet installs the +pod-network bypass-prevention rules during CNI `ADD` using nftables or iptables. +The driver annotates sandbox pods so the chained CNI plugin can read the proxy +UID and enforcement mode. The network sidecar keeps the same privilege profile +as the other sidecar topologies: in the default binary-aware mode it runs as UID +0 with `SYS_PTRACE` and `DAC_READ_SEARCH` (but no `NET_ADMIN`) to resolve +cross-UID `/proc`, and only stays non-root with no added capabilities when +`process_binary_aware_network_policy` is disabled. The agent container stays +non-root with no added Linux capabilities in either mode. Because the node CNI — +not an in-pod container — programs the firewall, no container in the sandbox pod +holds `NET_ADMIN`. + +The CNI installer supports two modes. `conflist` (default) appends the +`openshell-cni` plugin to an existing CNI `.conflist` (k3s / vanilla). On +OpenShift (Multus / OVN-Kubernetes) there is no appendable `.conflist`, so +`multus-chain` writes a standalone plugin `.conf` into the Multus +`vendor-cni-chain` auxiliary-chain directory and stores plugin credentials in a +persistent `stateDir`. Neither mode modifies a CNO-managed file. + +The installer patches the chained plugin at startup and then re-verifies it on a +reconcile tick, re-patching when the plugin is missing. This keeps enforcement in +place across CNI config rewrites (for example a CNO reconcile) and DaemonSet +restarts, bounding any such gap to one reconcile interval. The chained plugin +lives in the host CNI config and survives installer pod restarts, so an ordinary +DaemonSet restart or rolling update never strips enforcement: the `preStop` hook +removes it only when the owning DaemonSet is actually being deleted (helm +uninstall), and fences the node before doing so. + +A per-node scheduling gate closes the cold-start race. Once the chained plugin is +installed, the installer labels its node `openshell.ai/cni-ready=true`. Each +reconcile tick fences before it repairs: the instant enforcement is not +verifiably in place it clears the label, attempts repair, and only re-marks the +node ready once the plugin is healthy again. The gateway sets a required +`nodeAffinity` on that label for every cni-sidecar sandbox pod, so a pod cannot +schedule onto a node before that node's egress enforcement is active, and a node +whose enforcement later breaks stops accepting new sandbox pods. The label is set +through a minimal cluster-scoped grant (`nodes` `get`/`patch`, plus `get` on the +installer's own DaemonSet) bound to the dedicated CNI ServiceAccount. + +The persistent label cannot by itself cover a node reboot that wipes tmpfs-backed +enforcement (`multus-chain` stores the chain file under `/run`). A boot-time +`NoSchedule` taint (`openshell.ai/cni-not-ready`) **narrows** that window but does +not fully close it: operator node config applies it at boot (only node config +runs before the scheduler; a MachineConfig example ships under +`deploy/helm/openshell/examples/`), the CNI DaemonSet tolerates it, and the +installer removes it once enforcement is ready (never re-adding it, so a transient +unready does not over-repel). A small residual race remains because the taint is +applied after the kubelet starts, so the kubelet can briefly mark the node +schedulable before the taint lands; `--register-with-taints` covers the initial +join but not reboot re-registration. `conflist` mode keeps the plugin on +persistent disk and is unaffected by reboot, and is the way to avoid the window +entirely. One residual +limitation: an ungraceful DaemonSet pod deletion (no `preStop`) leaves a stale +`cni-ready=true` until the pod is rescheduled and the next reconcile tick +re-evaluates it. + +The CNI installer is a **cluster singleton**. Its chained plugin enforces pods +that carry the OpenShell annotations (`openshell.ai/cni=enabled` plus the +proxy-UID and enforcement-mode annotations, set only by a gateway on its own +sandbox pods) **and** whose namespace is in the plugin's `sandboxNamespaces` +allowlist. Pods in other namespaces are passed through without a Kubernetes API +lookup, so the allowlist bounds the blast radius of the per-pod annotation read. +The allowlist is built **automatically**: each `cni-sidecar` gateway release +ships a Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true`) in its +sandbox namespace, and the installer's reconcile aggregates every marker's +namespace (via `list-sandbox-namespaces`) — unioned with the optional static +`cni.sandboxNamespaces` — into the plugin config. Because the marker is a normal +Helm resource, uninstalling a release or changing its `sandboxNamespace` removes +the marker. Deregistration is **drain-gated and monotonic**: the reconcile also +unions in every namespace that still contains an OpenShell-managed sandbox pod +(`openshell.ai/managed-by=openshell`), so removing a marker does not drop +enforcement while sandboxes are still running (which would fail-open their +recreated pods) — the namespace is pruned only once it is drained (no marker and +no sandbox pods). An additional `cni.external` +release is discovered within one reconcile, with no manual allowlist edit. To +eliminate the discovery-window race, the installer publishes on each node the CSV +of namespaces it currently enforces (`openshell.ai/cni-sandbox-namespaces` +annotation, via `set-node-coverage`), and every gateway runs a `wait-coverage` +init container that blocks until each enforcement-ready node acknowledges the +gateway's namespace — so a newly-registered release does not serve sandboxes until +enforcement is confirmed cluster-wide. The installer resources use a fixed +release-independent name, the plugin config carries a fixed `openshell` owner, and +a `configVersion` (over the aggregated allowlist and config) binds readiness so a +stale-version entry is repaired before the node is re-marked ready. Install the +singleton (`cni.enabled=true`) in one release per cluster; additional releases set +`cni.enabled=false` + `cni.external=true`. The installer treats any `openshell-cni` +chained entry as its own and upgrades it in place (the conflist patch preserves +all other plugins). The `pods get` and `configmaps list` grants are cluster-scoped +so the one installer can discover marker ConfigMaps and read sandbox pods in any +allowlisted namespace. + +On OpenShift, binary-aware network policy also requires a purpose-built +SecurityContextConstraints for sandbox pods: the network sidecar runs as UID 0 +with `SYS_PTRACE` and `DAC_READ_SEARCH` to inspect cross-UID `/proc`, which +`restricted-v2` forbids. `sandboxServiceAccount.openshift.binaryAwareSCC` creates +a minimal SCC (the `restricted-v2` baseline plus only those two capabilities, UID +0, and the `image` volume type) and binds it to the sandbox ServiceAccount. +Disabling `processBinaryAwareNetworkPolicy` drops the capability requirement and +lets the stock `restricted-v2` SCC apply. + ## Images The gateway image and Helm chart are built from this repository. Sandbox images diff --git a/crates/openshell-cni/BUILD.bazel b/crates/openshell-cni/BUILD.bazel new file mode 100644 index 0000000000..78705e644c --- /dev/null +++ b/crates/openshell-cni/BUILD.bazel @@ -0,0 +1,41 @@ +load("@crates//:defs.bzl", "aliases", "all_crate_deps") +load("@rules_rs//rs:rust_binary.bzl", "rust_binary") +load("@rules_rs//rs:rust_library.bzl", "rust_library") +load("@rules_rs//rs:rust_test.bzl", "rust_test") +load("@rules_rust//rust:defs.bzl", "rustfmt_test") + +rust_library( + name = "openshell-cni", + srcs = glob( + ["src/**/*.rs"], + exclude = ["src/main.rs"], + ), + aliases = aliases(), + visibility = ["//visibility:public"], + deps = all_crate_deps(normal = True), +) + +rust_binary( + name = "openshell-cni_bin", + srcs = ["src/main.rs"], + aliases = aliases(), + binary_name = "openshell-cni", + visibility = ["//visibility:public"], + deps = all_crate_deps(normal = True) + [":openshell-cni"], +) + +rust_test( + name = "openshell-cni_test", + crate = ":openshell-cni", + deps = all_crate_deps(normal_dev = True), +) + +rustfmt_test( + name = "rustfmt_test", + targets = [ + ":openshell-cni", + ":openshell-cni_bin", + ":openshell-cni_test", + ], + visibility = ["//crates:__pkg__"], +) diff --git a/crates/openshell-cni/Cargo.toml b/crates/openshell-cni/Cargo.toml new file mode 100644 index 0000000000..835d1ef4c6 --- /dev/null +++ b/crates/openshell-cni/Cargo.toml @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-cni" +description = "OpenShell chained CNI plugin for Kubernetes sidecar network enforcement" +version.workspace = true +edition.workspace = true +license.workspace = true +repository.workspace = true +rust-version.workspace = true + +[dependencies] +base64 = { workspace = true } +miette = { workspace = true } +reqwest = { workspace = true, features = ["blocking"] } +serde = { workspace = true } +serde_json = { workspace = true } +serde_yml = { workspace = true } +tempfile = "3" + +[target.'cfg(target_os = "linux")'.dependencies] +libc = "0.2" + +[dev-dependencies] +tempfile = "3" + +[lints] +workspace = true diff --git a/crates/openshell-cni/src/lib.rs b/crates/openshell-cni/src/lib.rs new file mode 100644 index 0000000000..2dc371ae97 --- /dev/null +++ b/crates/openshell-cni/src/lib.rs @@ -0,0 +1,1824 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use base64::Engine; +use miette::{Context, IntoDiagnostic, Result}; +use serde::Deserialize; +use serde_json::Value; +use std::collections::BTreeMap; +use std::io::{Read, Write}; +use std::path::{Path, PathBuf}; +#[cfg(target_os = "linux")] +use std::process::Command; + +const DEFAULT_CNI_VERSION: &str = "1.0.0"; +const SUPPORTED_CNI_VERSIONS: &[&str] = &["0.3.0", "0.3.1", "0.4.0", "1.0.0"]; +const DEFAULT_KUBECONFIG_PATH: &str = "/etc/cni/net.d/openshell-cni-kubeconfig"; +const OPENSHELL_CNI_ENABLED_ANNOTATION: &str = "openshell.ai/cni"; +const OPENSHELL_CNI_PROXY_UID_ANNOTATION: &str = "openshell.ai/proxy-uid"; +const OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION: &str = + "openshell.ai/network-enforcement-mode"; +const CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE: &str = "cni-sidecar"; +/// Node label set by the CNI installer once the chained plugin is in place and +/// cleared when it is not, so the gateway can gate sandbox scheduling on +/// per-node egress-enforcement readiness. Must stay in sync with the identical +/// constant in `openshell-driver-kubernetes` (used for the sandbox pod +/// nodeAffinity) and with the CNI `DaemonSet`'s node-patch RBAC. +const NODE_READY_LABEL: &str = "openshell.ai/cni-ready"; +/// Node taint applied at boot (via an operator-provided `MachineConfig` / node +/// config) so a rebooted node repels workloads until egress enforcement is +/// re-established — the persistent `cni-ready` label cannot reflect a reboot that +/// wipes tmpfs-backed enforcement, so a boot-time taint narrows that window +/// (a small residual reboot race remains — see the `MachineConfig` example). The +/// installer only REMOVES this taint (once enforcement is ready); it never adds +/// it, so a transient unready never over-repels a running node. Must stay in sync +/// with the taint key used by the `MachineConfig` and the `DaemonSet` toleration. +const NODE_NOT_READY_TAINT_KEY: &str = "openshell.ai/cni-not-ready"; +const NODE_NOT_READY_TAINT_EFFECT: &str = "NoSchedule"; +/// Namespace label a gateway release stamps on its sandbox namespace so the CNI +/// singleton discovers and enforces it automatically — no manual allowlist edit +/// per release. The installer aggregates every namespace containing a marker +/// `ConfigMap` with this label into the plugin's `sandboxNamespaces`. The marker is +/// a Helm-owned resource, so uninstalling a release (or changing its sandbox +/// namespace) removes the marker and de-registers the namespace automatically. +const CNI_REGISTRATION_LABEL: &str = "openshell.ai/cni-registration"; +/// Node annotation the installer sets to the sorted CSV of the namespaces it +/// currently enforces on that node, so the gateway can wait for cluster-wide +/// acknowledgement of its namespace before it serves sandboxes. +const NODE_COVERAGE_ANNOTATION: &str = "openshell.ai/cni-sandbox-namespaces"; +/// Label selector identifying OpenShell-managed sandbox pods (set by the gateway +/// driver). Used to keep a namespace enforced while its sandboxes are still +/// running even after its registration marker is removed (drain-gated prune). +const SANDBOX_POD_LABEL_SELECTOR: &str = "openshell.ai/managed-by=openshell"; +#[allow(dead_code)] +const OPENSHELL_TABLE: &str = "openshell_sidecar_bypass"; +#[allow(dead_code)] +const OPENSHELL_IPTABLES_CHAIN: &str = "OPENSHELL_OUTPUT"; +#[cfg(target_os = "linux")] +const NFT_SEARCH_PATHS: &[&str] = &[ + "/usr/sbin/nft", + "/sbin/nft", + "/usr/bin/nft", + "/bin/nft", + "/opt/cni/bin/nft", + "/bin/aux/nft", +]; +#[cfg(target_os = "linux")] +const IPTABLES_SEARCH_PATHS: &[&str] = &[ + "/usr/sbin/iptables", + "/sbin/iptables", + "/usr/bin/iptables", + "/bin/iptables", + "/opt/cni/bin/iptables", + "/bin/aux/iptables", +]; +#[cfg(target_os = "linux")] +const IP6TABLES_SEARCH_PATHS: &[&str] = &[ + "/usr/sbin/ip6tables", + "/sbin/ip6tables", + "/usr/bin/ip6tables", + "/bin/ip6tables", + "/opt/cni/bin/ip6tables", + "/bin/aux/ip6tables", +]; + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +struct CniConfig { + cni_version: Option, + #[serde(default)] + prev_result: Option, + #[serde(default)] + openshell: OpenShellConfig, +} + +#[derive(Debug, Default, Deserialize)] +#[serde(rename_all = "camelCase")] +struct OpenShellConfig { + kubeconfig: Option, + log_file: Option, + #[serde(default)] + sandbox_namespaces: Vec, +} + +#[derive(Debug, Clone)] +struct CniEnv { + command: String, + netns: Option, + args: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct PodRef { + namespace: String, + name: String, +} + +#[derive(Debug, Deserialize)] +struct PodResponse { + metadata: PodMetadata, +} + +#[derive(Debug, Deserialize)] +struct PodMetadata { + #[serde(default)] + annotations: BTreeMap, +} + +#[derive(Debug, Deserialize)] +struct DaemonSetResponse { + metadata: ObjectMeta, +} + +#[derive(Debug, Default, Deserialize)] +struct ObjectMeta { + #[serde(rename = "deletionTimestamp")] + deletion_timestamp: Option, +} + +#[derive(Debug, Deserialize)] +struct KubeConfig { + #[serde(rename = "current-context")] + current_context: String, + clusters: Vec, + contexts: Vec, + users: Vec, +} + +#[derive(Debug, Deserialize)] +struct NamedCluster { + name: String, + cluster: ClusterConfig, +} + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "kebab-case")] +struct ClusterConfig { + server: String, + certificate_authority_data: Option, + certificate_authority: Option, +} + +#[derive(Debug, Deserialize)] +struct NamedContext { + name: String, + context: ContextConfig, +} + +#[derive(Debug, Deserialize)] +struct ContextConfig { + cluster: String, + user: String, +} + +#[derive(Debug, Deserialize)] +struct NamedUser { + name: String, + user: UserConfig, +} + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "kebab-case")] +struct UserConfig { + token: Option, + token_file: Option, +} + +struct Runtime; + +trait PodReader { + fn pod_annotations(&self, kubeconfig: &Path, pod: &PodRef) -> Result>; +} + +trait RuleInstaller { + fn install(&self, netns: &Path, proxy_uid: u32) -> Result; + /// Read-only validation that the bypass-prevention rules are present. Must NOT + /// modify live rules (CNI CHECK runs on a running pod; a destructive reinstall + /// that fails would leave it unenforced). + fn verify(&self, netns: &Path) -> Result<()>; + fn cleanup(&self, netns: &Path) -> Result<()>; +} + +/// Returns true when `/proc/net/if_inet6` contents report at least one +/// non-loopback IPv6 address. Link-local (`fe80::/10`) counts because it can +/// still reach on-link peers and node services; only loopback (`::1`) is +/// excluded. Empty contents (IPv6 disabled in the kernel) return false. +fn if_inet6_has_non_loopback_ipv6(contents: &str) -> bool { + contents.lines().any(|line| { + // Format: <32-hex-addr> + line.split_whitespace() + .next() + .is_some_and(|addr| addr != "00000000000000000000000000000001") + }) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct InstallReport { + backend: &'static str, +} + +/// CNI spec reserves error codes >= 100 for plugin-specific failures. +const CNI_PLUGIN_ERROR_CODE: u32 = 100; + +/// Parses the `node-ready` subcommand flags into the desired readiness state. +/// `--set` marks the node ready; `--clear` fences it. +fn parse_node_ready_args(args: &[String]) -> Result { + let mut ready = None; + for arg in args { + match arg.as_str() { + "--set" => ready = Some(true), + "--clear" => ready = Some(false), + other => return Err(miette::miette!("unknown node-ready argument '{other}'")), + } + } + ready.ok_or_else(|| miette::miette!("node-ready requires --set or --clear")) +} + +/// Entry point for `openshell-cni node-ready`, the CNI installer's readiness gate. +/// +/// Invoked by the installer `DaemonSet` to gate sandbox scheduling on per-node +/// enforcement readiness. `--set` labels the node ready once the chained plugin +/// is installed; `--clear` removes the label on shutdown or repair failure so +/// new sandbox pods will not schedule on a node where egress enforcement is +/// absent. Reads the target node from `NODE_NAME` and authenticates with the +/// pod's in-cluster service account. +pub fn node_ready(args: &[String]) -> Result<()> { + let ready = parse_node_ready_args(args)?; + let node = std::env::var("NODE_NAME") + .map_err(|_| miette::miette!("NODE_NAME env var is required for node-ready"))?; + let client = KubeApiClient::from_in_cluster()?; + if ready { + // Set the label first (still fenced by the boot taint if present), then + // remove the boot taint so scheduling opens only once both agree. + client.patch_node_label(&node, NODE_READY_LABEL, Some("true"))?; + client.remove_node_taint(&node, NODE_NOT_READY_TAINT_KEY, NODE_NOT_READY_TAINT_EFFECT) + } else { + // Clear the label to fence new sandbox pods. Do NOT add the taint here: + // the taint is boot-managed (it would over-repel all workloads on a + // transient unready); the label gate is sandbox-specific. + client.patch_node_label(&node, NODE_READY_LABEL, None) + } +} + +/// Entry point for `openshell-cni list-sandbox-namespaces`. +/// +/// Run by the installer reconcile to compute the enforcement allowlist. Prints, +/// one per line, the union of: +/// * namespaces containing a Helm-owned registration marker `ConfigMap` +/// (`openshell.ai/cni-registration=true`) — active registrations; and +/// * namespaces that still contain an OpenShell-managed sandbox pod +/// (`openshell.ai/managed-by=openshell`). +/// +/// The second set makes the allowlist **monotonic while in use**: removing a +/// release's marker does not immediately drop enforcement for a namespace whose +/// sandboxes are still running (which would fail-open their recreated pods). The +/// namespace is pruned only once it is drained — no marker and no sandbox pods. +pub fn list_sandbox_namespaces() -> Result<()> { + let client = KubeApiClient::from_in_cluster()?; + let mut namespaces = client.list_registration_namespaces(CNI_REGISTRATION_LABEL)?; + namespaces.extend(client.list_sandbox_pod_namespaces(SANDBOX_POD_LABEL_SELECTOR)?); + namespaces.sort(); + namespaces.dedup(); + for ns in namespaces { + println!("{ns}"); + } + Ok(()) +} + +/// Entry point for `openshell-cni set-node-coverage `. +/// +/// Run by the installer reconcile to publish, on its own node, the sorted CSV of +/// namespaces it currently enforces, so gateways can wait for cluster-wide +/// acknowledgement of their namespace before serving sandboxes. +pub fn set_node_coverage(args: &[String]) -> Result<()> { + let csv = args.first().map_or("", String::as_str); + let node = std::env::var("NODE_NAME") + .map_err(|_| miette::miette!("NODE_NAME env var is required for set-node-coverage"))?; + let client = KubeApiClient::from_in_cluster()?; + client.patch_node_annotation(&node, NODE_COVERAGE_ANNOTATION, csv) +} + +/// Entry point for `openshell-cni wait-coverage `, run as a gateway +/// init container so the gateway does not serve sandboxes until every +/// enforcement-ready node acknowledges the namespace. +/// +/// Blocks until at least one node carries the `cni-ready` label and every such +/// node's coverage annotation includes the namespace. Times out (non-zero exit, +/// failing the init container fail-closed) after a bounded wait. +pub fn wait_coverage(args: &[String]) -> Result<()> { + let namespace = args + .first() + .ok_or_else(|| miette::miette!("wait-coverage requires a namespace argument"))?; + let client = KubeApiClient::from_in_cluster()?; + // ~5 minutes of 5s polls; fail closed if enforcement never converges. + for _ in 0..60 { + match client.namespace_covered_on_all_ready_nodes( + namespace, + NODE_READY_LABEL, + NODE_COVERAGE_ANNOTATION, + ) { + Ok(true) => return Ok(()), + Ok(false) => {} + Err(error) => eprintln!("wait-coverage: transient error: {error:?}"), + } + std::thread::sleep(std::time::Duration::from_secs(5)); + } + Err(miette::miette!( + "namespace {namespace} not enforced on all cni-ready nodes within timeout" + )) +} + +/// Entry point for `openshell-cni daemonset-active`. +/// +/// Used by the installer `preStop` hook to distinguish an ordinary pod +/// restart/rolling update from a real teardown (helm uninstall / `DaemonSet` +/// delete). +/// +/// Returns `Ok(())` (exit 0) whenever enforcement should be **preserved** — the +/// owning `DaemonSet` still exists and is not being deleted, OR the state cannot +/// be determined (missing env, API error). Enforcement lives in the host CNI +/// config and survives pod restarts, so preserving is the fail-safe default. +/// Returns `Err` (non-zero) only when the `DaemonSet` is positively confirmed +/// gone or terminating, signalling the caller that cleanup is appropriate. +pub fn daemonset_active() -> Result<()> { + let (Ok(name), Ok(namespace)) = (std::env::var("DS_NAME"), std::env::var("DS_NAMESPACE")) + else { + // Cannot identify the owning DaemonSet; preserve enforcement. + return Ok(()); + }; + let Ok(client) = KubeApiClient::from_in_cluster() else { + return Ok(()); + }; + match client.daemonset_terminating(&namespace, &name) { + Ok(true) => Err(miette::miette!( + "owning DaemonSet {namespace}/{name} is gone or terminating" + )), + // Active, or an API error we cannot interpret: preserve enforcement. + Ok(false) | Err(_) => Ok(()), + } +} + +pub fn run() -> Result<()> { + let env = CniEnv::from_process(); + let mut input = String::new(); + let result = (|| -> Result> { + std::io::stdin() + .read_to_string(&mut input) + .into_diagnostic() + .wrap_err("failed to read CNI config from stdin")?; + let runtime = Runtime; + handle_command(&input, &env, &runtime, &runtime) + })(); + + match result { + Ok(output) => { + if let Some(output) = output { + println!("{}", serde_json::to_string(&output).into_diagnostic()?); + } + Ok(()) + } + Err(error) => { + log_cni_error(&input, &env, &error); + // Per the CNI spec a failing plugin must print a structured error object + // on stdout and exit non-zero; the runtime parses this to surface the + // failure instead of treating stderr text as an opaque crash. + emit_cni_error(&input, &error); + Err(error) + } + } +} + +/// Builds the CNI-spec error object (`cniVersion`, `code`, `msg`, `details`). The +/// version echoes the request when parseable so the runtime accepts the reply. +fn cni_error_payload(input: &str, error: &miette::Report) -> Value { + let cni_version = serde_json::from_str::(input) + .ok() + .and_then(|config| config.cni_version) + .unwrap_or_else(|| DEFAULT_CNI_VERSION.to_string()); + serde_json::json!({ + "cniVersion": cni_version, + "code": CNI_PLUGIN_ERROR_CODE, + "msg": "OpenShell CNI plugin error", + "details": one_line_error(error), + }) +} + +fn emit_cni_error(input: &str, error: &miette::Report) { + if let Ok(serialized) = serde_json::to_string(&cni_error_payload(input, error)) { + println!("{serialized}"); + } +} + +fn log_cni_error(input: &str, env: &CniEnv, error: &miette::Report) { + let Ok(config) = serde_json::from_str::(input) else { + return; + }; + log_cni_info(&config, env, &format!("error={}", one_line_error(error))); +} + +fn log_cni_info(config: &CniConfig, env: &CniEnv, message: &str) { + let Some(log_file) = config.openshell.log_file.as_deref() else { + return; + }; + if log_file.is_empty() { + return; + } + + let pod = env.pod_ref().map_or_else( + || "-".to_string(), + |pod| format!("{}/{}", pod.namespace, pod.name), + ); + let Ok(mut file) = std::fs::OpenOptions::new() + .create(true) + .append(true) + .open(log_file) + else { + return; + }; + let _ = writeln!(file, "command={} pod={} {}", env.command, pod, message); +} + +fn one_line_error(error: &miette::Report) -> String { + format!("{error:?}") + .lines() + .map(str::trim) + .filter(|line| !line.is_empty()) + .collect::>() + .join(" | ") +} + +fn handle_command( + input: &str, + env: &CniEnv, + pod_reader: &impl PodReader, + installer: &impl RuleInstaller, +) -> Result> { + match env.command.as_str() { + "VERSION" => Ok(Some(version_response())), + "DEL" => { + if let Some(netns) = env.netns.as_deref() { + let _ = installer.cleanup(netns); + } + Ok(None) + } + "ADD" => { + let config: CniConfig = serde_json::from_str(input).into_diagnostic()?; + if let Some(workload) = workload_from_config(&config, env, pod_reader)? { + let netns = env.netns.as_deref().ok_or_else(|| { + miette::miette!("CNI_NETNS is required for OpenShell CNI ADD") + })?; + let report = installer.install(netns, workload.proxy_uid)?; + log_cni_info( + &config, + env, + &format!( + "status=installed backend={} proxy_uid={}", + report.backend, workload.proxy_uid + ), + ); + } + Ok(Some(pass_through_result(&config))) + } + "CHECK" => { + let config: CniConfig = serde_json::from_str(input).into_diagnostic()?; + if let Some(workload) = workload_from_config(&config, env, pod_reader)? { + let netns = env.netns.as_deref().ok_or_else(|| { + miette::miette!("CNI_NETNS is required for OpenShell CNI CHECK") + })?; + // CHECK is read-only: verify the rules are present without touching + // them, so a check never leaves a running pod momentarily unenforced. + installer.verify(netns)?; + let report = InstallReport { + backend: "verified", + }; + log_cni_info( + &config, + env, + &format!( + "status=installed backend={} proxy_uid={}", + report.backend, workload.proxy_uid + ), + ); + } + Ok(None) + } + other => Err(miette::miette!("unsupported CNI_COMMAND '{other}'")), + } +} + +#[derive(Debug, Clone, Copy)] +struct WorkloadConfig { + proxy_uid: u32, +} + +fn workload_from_config( + config: &CniConfig, + env: &CniEnv, + pod_reader: &impl PodReader, +) -> Result> { + let Some(pod) = env.pod_ref() else { + return Ok(None); + }; + // sandbox_namespaces is the allowlist of namespaces whose pods we inspect. + // A pod outside it is passed through immediately WITHOUT a Kubernetes API + // lookup — a blast-radius guard so control-plane/RBAC problems cannot block + // unrelated workloads' pod creation. Within the allowlist, enforcement is + // gated by the per-pod OpenShell annotations below (set only by a gateway on + // its own sandbox pods). The singleton installer populates this from + // cni.sandboxNamespaces (default: the release's sandbox namespace). An empty + // list means "any namespace" and is intentionally avoided by the chart. + if !config.openshell.sandbox_namespaces.is_empty() + && !config + .openshell + .sandbox_namespaces + .iter() + .any(|namespace| namespace == &pod.namespace) + { + return Ok(None); + } + let kubeconfig = config + .openshell + .kubeconfig + .as_deref() + .unwrap_or(DEFAULT_KUBECONFIG_PATH); + let annotations = pod_reader.pod_annotations(Path::new(kubeconfig), &pod)?; + if annotations + .get(OPENSHELL_CNI_ENABLED_ANNOTATION) + .map(String::as_str) + != Some("enabled") + { + return Ok(None); + } + if annotations + .get(OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION) + .map(String::as_str) + != Some(CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE) + { + return Ok(None); + } + let proxy_uid = annotations + .get(OPENSHELL_CNI_PROXY_UID_ANNOTATION) + .ok_or_else(|| miette::miette!("OpenShell CNI pod is missing proxy UID annotation"))? + .parse::() + .into_diagnostic() + .wrap_err("invalid OpenShell CNI proxy UID annotation")?; + Ok(Some(WorkloadConfig { proxy_uid })) +} + +fn pass_through_result(config: &CniConfig) -> Value { + config.prev_result.clone().unwrap_or_else(|| { + serde_json::json!({ + "cniVersion": config.cni_version.as_deref().unwrap_or(DEFAULT_CNI_VERSION) + }) + }) +} + +fn version_response() -> Value { + serde_json::json!({ + "cniVersion": DEFAULT_CNI_VERSION, + "supportedVersions": SUPPORTED_CNI_VERSIONS + }) +} + +impl CniEnv { + fn from_process() -> Self { + Self { + command: std::env::var("CNI_COMMAND").unwrap_or_else(|_| "VERSION".to_string()), + netns: std::env::var_os("CNI_NETNS").map(PathBuf::from), + args: std::env::var("CNI_ARGS").ok(), + } + } + + fn pod_ref(&self) -> Option { + let args = self.args.as_deref()?; + let values = parse_cni_args(args); + let namespace = values.get("K8S_POD_NAMESPACE")?.to_string(); + let name = values.get("K8S_POD_NAME")?.to_string(); + Some(PodRef { namespace, name }) + } +} + +fn parse_cni_args(args: &str) -> BTreeMap<&str, &str> { + args.split(';') + .filter_map(|part| part.split_once('=')) + .collect() +} + +impl PodReader for Runtime { + fn pod_annotations(&self, kubeconfig: &Path, pod: &PodRef) -> Result> { + let client = KubeApiClient::from_kubeconfig(kubeconfig)?; + client.pod_annotations(pod) + } +} + +struct KubeApiClient { + server: String, + token: String, + client: reqwest::blocking::Client, +} + +/// Blocking HTTP client builder with bounded connect and request deadlines. The +/// CNI plugin runs synchronously on the CNI ADD path and in the installer's +/// reconcile/coverage loops; without deadlines an API stall would wedge pod +/// creation, invalidate the coverage wait's time bound, and stall reconciliation. +fn kube_client_builder() -> reqwest::blocking::ClientBuilder { + reqwest::blocking::Client::builder() + .connect_timeout(std::time::Duration::from_secs(5)) + .timeout(std::time::Duration::from_secs(15)) +} + +impl KubeApiClient { + fn from_kubeconfig(path: &Path) -> Result { + let contents = std::fs::read_to_string(path) + .into_diagnostic() + .wrap_err_with(|| format!("failed to read kubeconfig {}", path.display()))?; + let kubeconfig: KubeConfig = serde_yml::from_str(&contents) + .into_diagnostic() + .wrap_err("invalid kubeconfig")?; + let context = kubeconfig + .contexts + .iter() + .find(|context| context.name == kubeconfig.current_context) + .ok_or_else(|| miette::miette!("current kubeconfig context not found"))?; + let cluster = kubeconfig + .clusters + .iter() + .find(|cluster| cluster.name == context.context.cluster) + .ok_or_else(|| miette::miette!("current kubeconfig cluster not found"))?; + let user = kubeconfig + .users + .iter() + .find(|user| user.name == context.context.user) + .ok_or_else(|| miette::miette!("current kubeconfig user not found"))?; + let token = match (&user.user.token, &user.user.token_file) { + (Some(token), _) => token.clone(), + (None, Some(token_file)) => std::fs::read_to_string(token_file) + .into_diagnostic() + .wrap_err_with(|| format!("failed to read kubeconfig token file {token_file}"))? + .trim() + .to_string(), + (None, None) => { + return Err(miette::miette!( + "kubeconfig user must contain token or token-file" + )); + } + }; + let mut builder = kube_client_builder(); + if let Some(ca) = cluster_certificate_authority(path, &cluster.cluster)? { + builder = builder.add_root_certificate(ca); + } + let client = builder.build().into_diagnostic()?; + Ok(Self { + server: cluster.cluster.server.trim_end_matches('/').to_string(), + token, + client, + }) + } + + /// Builds a client from the pod's mounted in-cluster service account, + /// independent of the plugin kubeconfig (whose token-file path is a host + /// path that does not resolve inside the installer container). + fn from_in_cluster() -> Result { + const TOKEN_PATH: &str = "/var/run/secrets/kubernetes.io/serviceaccount/token"; + const CA_PATH: &str = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"; + let host = std::env::var("KUBERNETES_SERVICE_HOST").map_err(|_| { + miette::miette!("KUBERNETES_SERVICE_HOST is required for in-cluster access") + })?; + let port = std::env::var("KUBERNETES_SERVICE_PORT").map_err(|_| { + miette::miette!("KUBERNETES_SERVICE_PORT is required for in-cluster access") + })?; + let token = std::fs::read_to_string(TOKEN_PATH) + .into_diagnostic() + .wrap_err("failed to read in-cluster service account token")? + .trim() + .to_string(); + let ca_pem = std::fs::read(CA_PATH) + .into_diagnostic() + .wrap_err("failed to read in-cluster CA certificate")?; + let ca = reqwest::Certificate::from_pem(&ca_pem).into_diagnostic()?; + let client = kube_client_builder() + .add_root_certificate(ca) + .build() + .into_diagnostic()?; + Ok(Self { + server: format!("https://{host}:{port}"), + token, + client, + }) + } + + /// Sets (`value = Some`) or removes (`value = None`) a single node label via + /// a JSON merge patch. A null value in a merge patch deletes the key. + fn patch_node_label(&self, node: &str, key: &str, value: Option<&str>) -> Result<()> { + let url = format!("{}/api/v1/nodes/{}", self.server, node); + let label_value = value.map_or(Value::Null, |v| Value::String(v.to_string())); + let body = serde_json::json!({ "metadata": { "labels": { key: label_value } } }); + let response = self + .client + .patch(url) + .bearer_auth(&self.token) + .header( + reqwest::header::CONTENT_TYPE, + "application/merge-patch+json", + ) + .body(serde_json::to_vec(&body).into_diagnostic()?) + .send() + .into_diagnostic() + .wrap_err("failed to patch node label")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while patching node {}", + response.status(), + node + )); + } + Ok(()) + } + + /// Sets a single node annotation to `value` via a JSON merge patch. + fn patch_node_annotation(&self, node: &str, key: &str, value: &str) -> Result<()> { + let url = format!("{}/api/v1/nodes/{}", self.server, node); + let body = serde_json::json!({ "metadata": { "annotations": { key: Value::String(value.to_string()) } } }); + let response = self + .client + .patch(url) + .bearer_auth(&self.token) + .header( + reqwest::header::CONTENT_TYPE, + "application/merge-patch+json", + ) + .body(serde_json::to_vec(&body).into_diagnostic()?) + .send() + .into_diagnostic() + .wrap_err("failed to patch node annotation")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while annotating node {}", + response.status(), + node + )); + } + Ok(()) + } + + /// Returns the namespaces of all registration marker `ConfigMaps` (those + /// carrying `key=true`) across the cluster. + fn list_registration_namespaces(&self, key: &str) -> Result> { + let selector = format!("{key}=true"); + let url = format!("{}/api/v1/configmaps", self.server); + let response = self + .client + .get(url) + .query(&[("labelSelector", selector.as_str())]) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to list registration ConfigMaps")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while listing ConfigMaps", + response.status() + )); + } + let list = response.json::().into_diagnostic()?; + let mut namespaces: Vec = list["items"] + .as_array() + .map(|items| { + items + .iter() + .filter_map(|item| item["metadata"]["namespace"].as_str().map(str::to_string)) + .collect() + }) + .unwrap_or_default(); + namespaces.sort(); + namespaces.dedup(); + Ok(namespaces) + } + + /// Returns the distinct namespaces of all pods matching `selector` + /// cluster-wide (used to keep enforcement for namespaces with live sandboxes). + fn list_sandbox_pod_namespaces(&self, selector: &str) -> Result> { + let url = format!("{}/api/v1/pods", self.server); + let response = self + .client + .get(url) + .query(&[("labelSelector", selector)]) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to list sandbox pods")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while listing sandbox pods", + response.status() + )); + } + let list = response.json::().into_diagnostic()?; + let mut namespaces: Vec = list["items"] + .as_array() + .map(|items| { + items + .iter() + .filter_map(|item| item["metadata"]["namespace"].as_str().map(str::to_string)) + .collect() + }) + .unwrap_or_default(); + namespaces.sort(); + namespaces.dedup(); + Ok(namespaces) + } + + /// Returns true when at least one node carries `ready_label` and every such + /// node's `coverage_annotation` (a comma-separated list) contains `namespace`. + fn namespace_covered_on_all_ready_nodes( + &self, + namespace: &str, + ready_label: &str, + coverage_annotation: &str, + ) -> Result { + let url = format!("{}/api/v1/nodes", self.server); + let response = self + .client + .get(url) + .query(&[("labelSelector", ready_label)]) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to list cni-ready nodes")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while listing nodes", + response.status() + )); + } + let list = response.json::().into_diagnostic()?; + let Some(nodes) = list["items"].as_array() else { + return Ok(false); + }; + if nodes.is_empty() { + return Ok(false); + } + Ok(nodes.iter().all(|node| { + node["metadata"]["annotations"][coverage_annotation] + .as_str() + .unwrap_or_default() + .split(',') + .any(|ns| ns == namespace) + })) + } + + /// Removes the given taint (key + effect) from the node if present, using a + /// JSON Patch whose `test` on `metadata.resourceVersion` provides optimistic + /// concurrency so a concurrent taint change by another controller is not + /// clobbered (retried on conflict). A no-op when the taint is absent. + fn remove_node_taint(&self, node: &str, key: &str, effect: &str) -> Result<()> { + let url = format!("{}/api/v1/nodes/{}", self.server, node); + for _attempt in 0..5 { + let obj = self + .client + .get(&url) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to read node for taint removal")?; + if !obj.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while reading node {}", + obj.status(), + node + )); + } + let node_obj = obj.json::().into_diagnostic()?; + let resource_version = node_obj["metadata"]["resourceVersion"] + .as_str() + .unwrap_or_default() + .to_string(); + let taints = node_obj["spec"]["taints"].as_array(); + let filtered: Vec = taints + .map(|list| { + list.iter() + .filter(|t| { + !(t["key"].as_str() == Some(key) + && t["effect"].as_str() == Some(effect)) + }) + .cloned() + .collect() + }) + .unwrap_or_default(); + // Nothing to remove. + if taints.is_none_or(|list| list.len() == filtered.len()) { + return Ok(()); + } + let patch = serde_json::json!([ + { "op": "test", "path": "/metadata/resourceVersion", "value": resource_version }, + { "op": "replace", "path": "/spec/taints", "value": filtered }, + ]); + let response = self + .client + .patch(&url) + .bearer_auth(&self.token) + .header(reqwest::header::CONTENT_TYPE, "application/json-patch+json") + .body(serde_json::to_vec(&patch).into_diagnostic()?) + .send() + .into_diagnostic() + .wrap_err("failed to patch node taints")?; + let status = response.status().as_u16(); + // 409 (resourceVersion conflict) or 422 (test op failed) → retry. + if status == 409 || status == 422 { + continue; + } + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while removing taint from node {}", + response.status(), + node + )); + } + return Ok(()); + } + Err(miette::miette!( + "failed to remove taint from node {node} after retries (conflict)" + )) + } + + /// Returns true when the named `DaemonSet` is gone (404) or has a + /// `deletionTimestamp` set (being deleted). A non-404 HTTP error is + /// propagated so the caller can treat it as indeterminate. + fn daemonset_terminating(&self, namespace: &str, name: &str) -> Result { + let url = format!( + "{}/apis/apps/v1/namespaces/{}/daemonsets/{}", + self.server, namespace, name + ); + let response = self + .client + .get(url) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to query Kubernetes API for DaemonSet state")?; + if response.status().as_u16() == 404 { + return Ok(true); + } + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while reading DaemonSet {}/{}", + response.status(), + namespace, + name + )); + } + let ds = response.json::().into_diagnostic()?; + Ok(ds.metadata.deletion_timestamp.is_some()) + } + + fn pod_annotations(&self, pod: &PodRef) -> Result> { + let url = format!( + "{}/api/v1/namespaces/{}/pods/{}", + self.server, pod.namespace, pod.name + ); + let response = self + .client + .get(url) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to query Kubernetes API for pod annotations")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while reading pod {}/{}", + response.status(), + pod.namespace, + pod.name + )); + } + let pod = response.json::().into_diagnostic()?; + Ok(pod.metadata.annotations) + } +} + +fn cluster_certificate_authority( + kubeconfig_path: &Path, + cluster: &ClusterConfig, +) -> Result> { + if let Some(data) = cluster.certificate_authority_data.as_deref() { + let pem = base64::engine::general_purpose::STANDARD + .decode(data) + .into_diagnostic() + .wrap_err("invalid kubeconfig certificate-authority-data")?; + return Ok(Some( + reqwest::Certificate::from_pem(&pem).into_diagnostic()?, + )); + } + if let Some(path) = cluster.certificate_authority.as_deref() { + let ca_path = if Path::new(path).is_absolute() { + PathBuf::from(path) + } else { + kubeconfig_path + .parent() + .unwrap_or_else(|| Path::new(".")) + .join(path) + }; + let pem = std::fs::read(ca_path).into_diagnostic()?; + return Ok(Some( + reqwest::Certificate::from_pem(&pem).into_diagnostic()?, + )); + } + Ok(None) +} + +impl RuleInstaller for Runtime { + fn install(&self, netns: &Path, proxy_uid: u32) -> Result { + install_rules(netns, proxy_uid) + } + + fn verify(&self, netns: &Path) -> Result<()> { + verify_rules(netns) + } + + fn cleanup(&self, netns: &Path) -> Result<()> { + cleanup_rules(netns) + } +} + +#[allow(dead_code)] +fn generate_sidecar_bypass_ruleset(proxy_uid: u32, log_prefix: Option<&str>) -> String { + let log_tcp = log_prefix + .map(|p| { + format!( + "\n tcp flags syn limit rate 5/second burst 10 packets log prefix \"{p}\" flags skuid" + ) + }) + .unwrap_or_default(); + let log_udp = log_prefix + .map(|p| { + format!( + "\n meta l4proto udp limit rate 5/second burst 10 packets log prefix \"{p}\" flags skuid" + ) + }) + .unwrap_or_default(); + + format!( + r#"table inet {OPENSHELL_TABLE} {{ + chain output {{ + type filter hook output priority 0; policy accept; + + oifname "lo" accept + ct state established,related accept + meta skuid {proxy_uid} accept{log_tcp} + meta nfproto ipv4 meta l4proto tcp reject with icmp type port-unreachable + meta nfproto ipv6 meta l4proto tcp reject with icmpv6 type port-unreachable{log_udp} + meta nfproto ipv4 meta l4proto udp reject with icmp type port-unreachable + meta nfproto ipv6 meta l4proto udp reject with icmpv6 type port-unreachable + }} +}} +"# + ) +} + +#[cfg(target_os = "linux")] +fn install_rules(netns: &Path, proxy_uid: u32) -> Result { + // The preferred nft backend programs both families in one inet ruleset, so + // its IPv6 reject rules are harmless no-ops on IPv4-only pods and it needs no + // IPv6 probe. Only the iptables fallback must decide whether ip6tables is + // required, and it does so by inspecting the pod netns (fail-closed). + let nft_error = if let Some(nft) = find_nft() { + match install_nft_rules(netns, proxy_uid, &nft) { + Ok(()) => { + return Ok(InstallReport { backend: "nft" }); + } + Err(error) => Some(one_line_error(&error)), + } + } else { + None + }; + + if let Some(iptables) = find_iptables() { + let enforce_ipv6 = netns_requires_ipv6_enforcement(netns); + install_iptables_rules(netns, proxy_uid, &iptables, enforce_ipv6) + .wrap_err("iptables fallback failed")?; + return Ok(InstallReport { + backend: "iptables", + }); + } + + if let Some(nft_error) = nft_error { + return Err(miette::miette!( + "nft rule installation failed and iptables was not found on node: {nft_error}" + )); + } + + Err(miette::miette!( + "neither nft nor iptables was found on node; OpenShell CNI requires a pod-network firewall backend" + )) +} + +#[cfg(target_os = "linux")] +fn install_nft_rules(netns: &Path, proxy_uid: u32, nft: &str) -> Result<()> { + // Atomic replace: ensure-exists, delete, recreate in a SINGLE `nft -f` + // transaction. nft applies the whole file atomically, so a failure never + // leaves the pod with the table deleted-but-not-recreated (unenforced). The + // leading `table {}` makes the subsequent `delete table` safe on first apply. + let body = generate_sidecar_bypass_ruleset(proxy_uid, Some("openshell:cni-sidecar:")); + let ruleset = + format!("table inet {OPENSHELL_TABLE} {{}}\ndelete table inet {OPENSHELL_TABLE}\n{body}"); + run_nft_ruleset_in_netns(netns, &nft, &ruleset) +} + +/// Read-only check that the bypass-prevention rules are present in the netns. +/// Never modifies live rules. Prefers nft (the table), falls back to the iptables +/// chain; errors when neither is present so CNI CHECK surfaces the gap. +#[cfg(target_os = "linux")] +fn verify_rules(netns: &Path) -> Result<()> { + if let Some(nft) = find_nft() { + if run_nft_args_in_netns(netns, &nft, &["list", "table", "inet", OPENSHELL_TABLE]).is_ok() { + return Ok(()); + } + } + if let Some(iptables) = find_iptables() { + if run_command_in_netns( + netns, + &iptables.ipv4, + &["-w", "-t", "filter", "-n", "-L", OPENSHELL_IPTABLES_CHAIN], + ) + .is_ok() + { + return Ok(()); + } + } + Err(miette::miette!( + "OpenShell CNI bypass-prevention rules are not present in the pod network namespace" + )) +} + +#[cfg(not(target_os = "linux"))] +fn verify_rules(netns: &Path) -> Result<()> { + let _ = netns; + Err(miette::miette!( + "OpenShell CNI rule verification is supported only on Linux nodes" + )) +} + +#[cfg(not(target_os = "linux"))] +fn install_rules(netns: &Path, proxy_uid: u32) -> Result { + let _ = (netns, proxy_uid); + Err(miette::miette!( + "OpenShell CNI rule installation is supported only on Linux nodes" + )) +} + +/// Decides whether the iptables fallback must enforce IPv6, by probing the pod +/// netns for any non-loopback IPv6 address. Fails closed: any error (setns/exec +/// failure, or the probe reporting IPv6 present) returns true, so a pod with +/// IPv6 is never left unenforced just because detection was inconclusive. Only a +/// clean "no IPv6" result (probe exit 0) returns false, keeping IPv4-only nodes +/// working without ip6tables. +#[cfg(target_os = "linux")] +fn netns_requires_ipv6_enforcement(netns: &Path) -> bool { + // Re-exec this binary inside the target netns; its exit status encodes the + // result (exit 0 = no IPv6 → Ok, non-zero = IPv6/undetermined → Err). + run_command_in_netns(netns, "/proc/self/exe", &[NETNS_IPV6_PROBE_ARG]).is_err() +} + +/// Argument that runs the in-netns IPv6 probe (see `netns_probe_ipv6`). +#[cfg(target_os = "linux")] +const NETNS_IPV6_PROBE_ARG: &str = "__netns-probe-ipv6"; + +/// In-netns IPv6 probe entry point, re-exec'd inside the pod netns. +/// +/// Reports via exit status whether IPv6 enforcement is needed: success (exit 0) +/// means no non-loopback IPv6 is present; a non-success exit means IPv6 is +/// present or could not be determined, so the caller fails closed. A missing +/// `/proc/net/if_inet6` means IPv6 is disabled in the kernel (no enforcement +/// needed); any other read error is treated as indeterminate. +pub fn netns_probe_ipv6() -> Result<()> { + match std::fs::read_to_string("/proc/net/if_inet6") { + Ok(contents) => { + if if_inet6_has_non_loopback_ipv6(&contents) { + Err(miette::miette!("pod netns has non-loopback IPv6")) + } else { + Ok(()) + } + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(error) => Err(error) + .into_diagnostic() + .wrap_err("failed to read /proc/net/if_inet6 for IPv6 probe"), + } +} + +#[cfg(target_os = "linux")] +fn cleanup_rules(netns: &Path) -> Result<()> { + if let Some(nft) = find_nft() { + let _ = cleanup_nft_rules(netns, &nft); + } + if let Some(iptables) = find_iptables() { + cleanup_iptables_rules(netns, &iptables); + } + Ok(()) +} + +#[cfg(target_os = "linux")] +fn cleanup_nft_rules(netns: &Path, nft: &str) -> Result<()> { + run_nft_args_in_netns(netns, nft, &["delete", "table", "inet", OPENSHELL_TABLE]) +} + +#[cfg(not(target_os = "linux"))] +#[allow(clippy::unnecessary_wraps)] +fn cleanup_rules(netns: &Path) -> Result<()> { + let _ = netns; + Ok(()) +} + +#[cfg(target_os = "linux")] +fn run_nft_ruleset_in_netns(netns: &Path, nft: &str, ruleset: &str) -> Result<()> { + use std::io::Write; + + let mut tmp = tempfile::Builder::new() + .prefix("openshell-cni-") + .suffix(".nft") + .tempfile() + .into_diagnostic()?; + tmp.write_all(ruleset.as_bytes()).into_diagnostic()?; + let ruleset_path = tmp.path().to_string_lossy().to_string(); + run_nft_args_in_netns(netns, nft, &["-f", &ruleset_path]) +} + +#[cfg(target_os = "linux")] +fn run_nft_args_in_netns(netns: &Path, nft: &str, args: &[&str]) -> Result<()> { + run_command_in_netns(netns, nft, args) +} + +#[cfg(target_os = "linux")] +fn run_command_in_netns(netns: &Path, program: &str, args: &[&str]) -> Result<()> { + use std::os::fd::AsRawFd; + use std::os::unix::process::CommandExt; + + let netns = std::fs::File::open(netns).into_diagnostic()?; + let fd = netns.as_raw_fd(); + let output = { + let mut command = Command::new(program); + command.args(args); + // SAFETY: pre_exec runs in the child after fork and before exec. setns + // only affects that child process before it executes the firewall tool. + #[allow(unsafe_code)] + unsafe { + command.pre_exec(move || { + if libc::setns(fd, libc::CLONE_NEWNET) != 0 { + return Err(std::io::Error::last_os_error()); + } + Ok(()) + }); + } + command.output().into_diagnostic()? + }; + + if output.status.success() { + return Ok(()); + } + Err(miette::miette!( + "{} failed in CNI network namespace: {}", + program, + String::from_utf8_lossy(&output.stderr).trim() + )) +} + +#[cfg(target_os = "linux")] +fn find_nft() -> Option { + find_existing_binary(NFT_SEARCH_PATHS) +} + +#[cfg(target_os = "linux")] +fn find_iptables() -> Option { + find_existing_binary(IPTABLES_SEARCH_PATHS).map(|ipv4| IptablesBackend { + ipv4, + ipv6: find_existing_binary(IP6TABLES_SEARCH_PATHS), + }) +} + +#[cfg(target_os = "linux")] +fn find_existing_binary(paths: &[&str]) -> Option { + paths + .iter() + .find(|path| Path::new(path).is_file()) + .map(|path| (*path).to_string()) +} + +#[cfg(target_os = "linux")] +struct IptablesBackend { + ipv4: String, + ipv6: Option, +} + +#[cfg(target_os = "linux")] +fn install_iptables_rules( + netns: &Path, + proxy_uid: u32, + backend: &IptablesBackend, + enforce_ipv6: bool, +) -> Result<()> { + cleanup_iptables_family(netns, &backend.ipv4); + install_iptables_family(netns, &backend.ipv4, proxy_uid, "icmp-port-unreachable")?; + + if enforce_ipv6 { + // The pod has a routable IPv6 address, so unenforced IPv6 would be a + // policy bypass. Fail closed if ip6tables is missing rather than leaving + // IPv4-only enforcement in place. IPv4-only pods skip this entirely so + // nodes without ip6tables still work. + let ipv6 = backend.ipv6.as_deref().ok_or_else(|| { + miette::miette!( + "pod has IPv6 connectivity but ip6tables was not found on node; OpenShell CNI requires it to enforce IPv6 egress in the iptables fallback (install ip6tables or nft)" + ) + })?; + cleanup_iptables_family(netns, ipv6); + install_iptables_family(netns, ipv6, proxy_uid, "icmp6-port-unreachable")?; + } + + Ok(()) +} + +#[cfg(target_os = "linux")] +fn cleanup_iptables_rules(netns: &Path, backend: &IptablesBackend) { + cleanup_iptables_family(netns, &backend.ipv4); + if let Some(ipv6) = backend.ipv6.as_deref() { + cleanup_iptables_family(netns, ipv6); + } +} + +#[cfg(target_os = "linux")] +fn cleanup_iptables_family(netns: &Path, iptables: &str) { + for _ in 0..16 { + if run_command_in_netns( + netns, + iptables, + &[ + "-w", + "-t", + "filter", + "-D", + "OUTPUT", + "-j", + OPENSHELL_IPTABLES_CHAIN, + ], + ) + .is_err() + { + break; + } + } + let _ = run_command_in_netns( + netns, + iptables, + &["-w", "-t", "filter", "-F", OPENSHELL_IPTABLES_CHAIN], + ); + let _ = run_command_in_netns( + netns, + iptables, + &["-w", "-t", "filter", "-X", OPENSHELL_IPTABLES_CHAIN], + ); +} + +#[cfg(target_os = "linux")] +fn install_iptables_family( + netns: &Path, + iptables: &str, + proxy_uid: u32, + reject_with: &str, +) -> Result<()> { + for args in generate_iptables_install_commands(proxy_uid, reject_with) { + let args = args.iter().map(String::as_str).collect::>(); + run_command_in_netns(netns, iptables, &args)?; + } + Ok(()) +} + +#[cfg(target_os = "linux")] +fn generate_iptables_install_commands(proxy_uid: u32, reject_with: &str) -> Vec> { + let uid = proxy_uid.to_string(); + [ + vec!["-w", "-t", "filter", "-N", OPENSHELL_IPTABLES_CHAIN], + vec![ + "-w", + "-t", + "filter", + "-A", + OPENSHELL_IPTABLES_CHAIN, + "-o", + "lo", + "-j", + "RETURN", + ], + vec![ + "-w", + "-t", + "filter", + "-A", + OPENSHELL_IPTABLES_CHAIN, + "-m", + "conntrack", + "--ctstate", + "ESTABLISHED,RELATED", + "-j", + "RETURN", + ], + vec![ + "-w", + "-t", + "filter", + "-A", + OPENSHELL_IPTABLES_CHAIN, + "-m", + "owner", + "--uid-owner", + &uid, + "-j", + "RETURN", + ], + vec![ + "-w", + "-t", + "filter", + "-A", + OPENSHELL_IPTABLES_CHAIN, + "-p", + "tcp", + "-j", + "REJECT", + "--reject-with", + reject_with, + ], + vec![ + "-w", + "-t", + "filter", + "-A", + OPENSHELL_IPTABLES_CHAIN, + "-p", + "udp", + "-j", + "REJECT", + "--reject-with", + reject_with, + ], + vec![ + "-w", + "-t", + "filter", + "-I", + "OUTPUT", + "1", + "-j", + OPENSHELL_IPTABLES_CHAIN, + ], + ] + .into_iter() + .map(|args| args.into_iter().map(str::to_string).collect()) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + struct TestPods { + annotations: BTreeMap, + } + + impl PodReader for TestPods { + fn pod_annotations( + &self, + _kubeconfig: &Path, + _pod: &PodRef, + ) -> Result> { + Ok(self.annotations.clone()) + } + } + + #[derive(Default)] + struct TestInstaller { + installed: std::sync::Mutex>, + verified: std::sync::Mutex, + cleaned: std::sync::Mutex, + } + + impl RuleInstaller for TestInstaller { + fn install(&self, _netns: &Path, proxy_uid: u32) -> Result { + self.installed.lock().unwrap().push(proxy_uid); + Ok(InstallReport { backend: "test" }) + } + + fn verify(&self, _netns: &Path) -> Result<()> { + *self.verified.lock().unwrap() += 1; + Ok(()) + } + + fn cleanup(&self, _netns: &Path) -> Result<()> { + *self.cleaned.lock().unwrap() += 1; + Ok(()) + } + } + + fn cni_input() -> String { + serde_json::json!({ + "cniVersion": "1.0.0", + "name": "openshell", + "type": "openshell-cni", + "prevResult": { + "cniVersion": "1.0.0", + "interfaces": [] + }, + "openshell": { + "kubeconfig": "/tmp/openshell-kubeconfig", + "sandboxNamespaces": ["openshell"] + } + }) + .to_string() + } + + fn cni_input_with_log_file(log_file: &Path) -> String { + serde_json::json!({ + "cniVersion": "1.0.0", + "name": "openshell", + "type": "openshell-cni", + "openshell": { + "kubeconfig": "/tmp/openshell-kubeconfig", + "sandboxNamespaces": ["openshell"], + "logFile": log_file.to_string_lossy() + } + }) + .to_string() + } + + fn env(command: &str) -> CniEnv { + CniEnv { + command: command.to_string(), + netns: Some(PathBuf::from("/proc/1/ns/net")), + args: Some("K8S_POD_NAMESPACE=openshell;K8S_POD_NAME=sandbox-1".to_string()), + } + } + + fn openshell_annotations() -> BTreeMap { + BTreeMap::from([ + ( + OPENSHELL_CNI_ENABLED_ANNOTATION.to_string(), + "enabled".to_string(), + ), + ( + OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION.to_string(), + CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE.to_string(), + ), + ( + OPENSHELL_CNI_PROXY_UID_ANNOTATION.to_string(), + "1337".to_string(), + ), + ]) + } + + #[test] + fn parses_kubernetes_cni_args() { + let pod = env("ADD").pod_ref().unwrap(); + assert_eq!(pod.namespace, "openshell"); + assert_eq!(pod.name, "sandbox-1"); + } + + #[test] + fn version_returns_supported_versions() { + let pods = TestPods { + annotations: BTreeMap::new(), + }; + let installer = TestInstaller::default(); + let output = handle_command("", &env("VERSION"), &pods, &installer) + .unwrap() + .unwrap(); + assert_eq!(output["cniVersion"], DEFAULT_CNI_VERSION); + assert!( + output["supportedVersions"] + .as_array() + .unwrap() + .contains(&serde_json::json!("0.3.1")) + ); + assert!( + output["supportedVersions"] + .as_array() + .unwrap() + .contains(&serde_json::json!("1.0.0")) + ); + } + + #[test] + fn add_installs_for_annotated_openshell_pod() { + let pods = TestPods { + annotations: openshell_annotations(), + }; + let installer = TestInstaller::default(); + let output = handle_command(&cni_input(), &env("ADD"), &pods, &installer) + .unwrap() + .unwrap(); + assert_eq!(output["interfaces"], serde_json::json!([])); + assert_eq!(*installer.installed.lock().unwrap(), vec![1337]); + } + + #[test] + fn check_verifies_without_reinstalling() { + let pods = TestPods { + annotations: openshell_annotations(), + }; + let installer = TestInstaller::default(); + handle_command(&cni_input(), &env("CHECK"), &pods, &installer).unwrap(); + // CHECK must be read-only: verify, never install (a destructive reinstall + // that failed would leave the running pod unenforced). + assert_eq!(*installer.verified.lock().unwrap(), 1); + assert!(installer.installed.lock().unwrap().is_empty()); + } + + #[test] + fn add_passes_through_non_openshell_pod() { + let pods = TestPods { + annotations: BTreeMap::new(), + }; + let installer = TestInstaller::default(); + let output = handle_command(&cni_input(), &env("ADD"), &pods, &installer) + .unwrap() + .unwrap(); + assert_eq!(output["interfaces"], serde_json::json!([])); + assert!(installer.installed.lock().unwrap().is_empty()); + } + + #[test] + fn add_passes_through_unconfigured_namespace_without_api_lookup() { + struct FailingPods; + + impl PodReader for FailingPods { + fn pod_annotations( + &self, + _kubeconfig: &Path, + _pod: &PodRef, + ) -> Result> { + Err(miette::miette!("unexpected API lookup")) + } + } + + let installer = TestInstaller::default(); + let mut env = env("ADD"); + env.args = Some("K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=coredns".to_string()); + let output = handle_command(&cni_input(), &env, &FailingPods, &installer) + .unwrap() + .unwrap(); + assert_eq!(output["interfaces"], serde_json::json!([])); + assert!(installer.installed.lock().unwrap().is_empty()); + } + + #[test] + fn del_cleans_when_netns_available() { + let pods = TestPods { + annotations: openshell_annotations(), + }; + let installer = TestInstaller::default(); + handle_command("", &env("DEL"), &pods, &installer).unwrap(); + assert_eq!(*installer.cleaned.lock().unwrap(), 1); + } + + #[test] + fn sidecar_ruleset_allows_proxy_uid_before_rejects() { + let ruleset = generate_sidecar_bypass_ruleset(1337, Some("openshell:cni-sidecar:")); + let uid_pos = ruleset.find("meta skuid 1337 accept").unwrap(); + let reject_pos = ruleset + .find("meta nfproto ipv4 meta l4proto tcp reject") + .unwrap(); + assert!(uid_pos < reject_pos); + assert!(ruleset.contains("oifname \"lo\" accept")); + assert_eq!( + ruleset + .matches("log prefix \"openshell:cni-sidecar:\"") + .count(), + 2 + ); + } + + #[cfg(target_os = "linux")] + #[test] + fn iptables_fallback_commands_allow_proxy_uid_before_rejects() { + let commands = generate_iptables_install_commands(1337, "icmp-port-unreachable"); + let rendered = commands + .iter() + .map(|command| command.join(" ")) + .collect::>() + .join("\n"); + let uid_pos = rendered.find("--uid-owner 1337 -j RETURN").unwrap(); + let reject_pos = rendered.find("-p tcp -j REJECT").unwrap(); + assert!(uid_pos < reject_pos); + assert!(rendered.contains("-A OPENSHELL_OUTPUT -o lo -j RETURN")); + assert!(rendered.contains("-I OUTPUT 1 -j OPENSHELL_OUTPUT")); + assert!(rendered.contains("--reject-with icmp-port-unreachable")); + } + + #[test] + fn cni_errors_append_to_configured_log_file() { + let dir = tempfile::tempdir().unwrap(); + let log_file = dir.path().join("openshell-cni.log"); + let error = miette::miette!( + "neither nft nor iptables was found on node; OpenShell CNI requires a pod-network firewall backend" + ); + + log_cni_error(&cni_input_with_log_file(&log_file), &env("ADD"), &error); + + let log = std::fs::read_to_string(log_file).unwrap(); + assert!(log.contains("command=ADD")); + assert!(log.contains("pod=openshell/sandbox-1")); + assert!(log.contains("neither nft nor iptables was found")); + } + + #[test] + fn add_success_appends_to_configured_log_file() { + let dir = tempfile::tempdir().unwrap(); + let log_file = dir.path().join("openshell-cni.log"); + let pods = TestPods { + annotations: openshell_annotations(), + }; + let installer = TestInstaller::default(); + + handle_command( + &cni_input_with_log_file(&log_file), + &env("ADD"), + &pods, + &installer, + ) + .unwrap(); + + let log = std::fs::read_to_string(log_file).unwrap(); + assert!(log.contains("command=ADD")); + assert!(log.contains("pod=openshell/sandbox-1")); + assert!(log.contains("status=installed")); + assert!(log.contains("backend=test")); + assert!(log.contains("proxy_uid=1337")); + } + + #[cfg(target_os = "linux")] + #[test] + fn nft_search_path_includes_k3s_aux_path() { + assert!(NFT_SEARCH_PATHS.contains(&"/bin/aux/nft")); + } + + #[test] + fn cni_error_payload_is_spec_compliant() { + let error = miette::miette!("boom failure"); + let payload = cni_error_payload(&cni_input(), &error); + assert_eq!(payload["cniVersion"], "1.0.0"); + assert_eq!(payload["code"], serde_json::json!(CNI_PLUGIN_ERROR_CODE)); + assert_eq!(payload["msg"], "OpenShell CNI plugin error"); + assert!( + payload["details"] + .as_str() + .unwrap() + .contains("boom failure") + ); + } + + #[test] + fn cni_error_payload_defaults_cni_version_on_unparseable_input() { + let error = miette::miette!("boom"); + let payload = cni_error_payload("not valid json", &error); + assert_eq!(payload["cniVersion"], DEFAULT_CNI_VERSION); + } + + #[test] + fn if_inet6_detects_global_and_ula() { + // Global address on eth0. + let contents = "fd0010244000000000000000000000005 03 40 00 80 eth0\n"; + assert!(if_inet6_has_non_loopback_ipv6(contents)); + } + + #[test] + fn if_inet6_counts_link_local() { + // Link-local fe80::/10 can still reach on-link peers and node services. + let contents = "fe800000000000000042acfffe110002 02 40 20 80 eth0\n"; + assert!(if_inet6_has_non_loopback_ipv6(contents)); + } + + #[test] + fn if_inet6_ignores_loopback_only() { + // Only ::1 on lo → no enforcement needed. + let contents = "00000000000000000000000000000001 01 80 10 80 lo\n"; + assert!(!if_inet6_has_non_loopback_ipv6(contents)); + } + + #[test] + fn if_inet6_empty_means_no_ipv6() { + assert!(!if_inet6_has_non_loopback_ipv6("")); + } + + #[test] + fn parse_node_ready_args_set_and_clear() { + assert!(parse_node_ready_args(&["--set".to_string()]).unwrap()); + assert!(!parse_node_ready_args(&["--clear".to_string()]).unwrap()); + } + + #[test] + fn parse_node_ready_args_requires_a_flag() { + assert!(parse_node_ready_args(&[]).is_err()); + } + + #[test] + fn parse_node_ready_args_rejects_unknown_flag() { + assert!(parse_node_ready_args(&["--bogus".to_string()]).is_err()); + } +} diff --git a/crates/openshell-cni/src/main.rs b/crates/openshell-cni/src/main.rs new file mode 100644 index 0000000000..bf885a9eb2 --- /dev/null +++ b/crates/openshell-cni/src/main.rs @@ -0,0 +1,24 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +fn main() { + // The container runtime invokes the CNI plugin with no positional args + // (config on stdin, CNI_COMMAND in env). The `node-ready` subcommand is only + // reached when the installer DaemonSet calls the binary explicitly. + let args: Vec = std::env::args().collect(); + let result = match args.get(1).map(String::as_str) { + Some("node-ready") => openshell_cni::node_ready(&args[2..]), + Some("list-sandbox-namespaces") => openshell_cni::list_sandbox_namespaces(), + Some("set-node-coverage") => openshell_cni::set_node_coverage(&args[2..]), + Some("wait-coverage") => openshell_cni::wait_coverage(&args[2..]), + Some("daemonset-active") => openshell_cni::daemonset_active(), + // Internal: re-exec'd inside a pod netns to probe for IPv6 (see + // netns_requires_ipv6_enforcement). Not part of the public CLI surface. + Some("__netns-probe-ipv6") => openshell_cni::netns_probe_ipv6(), + _ => openshell_cni::run(), + }; + if let Err(error) = result { + eprintln!("{error:?}"); + std::process::exit(1); + } +} diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 1356e2d932..b9668e88ee 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -98,6 +98,28 @@ abstract socket whose peer PID must match that authenticated supervisor. Both supervisors exit if the control connection closes, coupling their container restart lifecycle before a new authoritative client can be established. +The `cni-sidecar` supervisor topology keeps the sidecar runtime model, but +removes the pod-local network init container and its `NET_ADMIN`. The driver +annotates sandbox pods for the OpenShell chained CNI plugin, and the privileged +OpenShell CNI DaemonSet installs the sidecar bypass-prevention rules during CNI +`ADD` before the workload starts. Because the node CNI programs the firewall, no +container in the sandbox pod holds `NET_ADMIN`. The network sidecar otherwise +keeps the same privilege profile as the other sidecar topologies: in the default +binary-aware mode it runs as UID 0 with `SYS_PTRACE` and `DAC_READ_SEARCH` to +resolve cross-UID `/proc`, and is non-root with no added capabilities only when +`process_binary_aware_network_policy` is disabled. The agent container is non-root +with no added Linux capabilities in either mode. + +The driver also sets a required `nodeAffinity` on the CNI installer's +`openshell.ai/cni-ready` node label for cni-sidecar pods, so a sandbox cannot +schedule onto a node before that node's chained plugin is active. + +Sidecar and cni-sidecar modes use the pod `fsGroup` to make the projected +service-account token and sandbox client TLS secret group-readable so the +non-root process supervisor can authenticate to the gateway. Treat the agent +container as trusted with respect to those in-pod gateway credentials until a +narrower credential handoff exists. + The driver can request a Kubernetes AppArmor profile through `app_armor_profile`. diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index fb471180a9..4fec186c58 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -65,6 +65,10 @@ pub enum SupervisorTopology { /// Run network supervision in a privileged sidecar and process supervision /// as a low-capability wrapper in the agent container. Sidecar, + /// Run network supervision in a sidecar, with pod-network rules installed + /// by the `OpenShell` chained CNI plugin instead of a privileged init + /// container. + CniSidecar, } impl std::fmt::Display for SupervisorTopology { @@ -72,6 +76,7 @@ impl std::fmt::Display for SupervisorTopology { match self { Self::Combined => f.write_str("combined"), Self::Sidecar => f.write_str("sidecar"), + Self::CniSidecar => f.write_str("cni-sidecar"), } } } @@ -83,6 +88,7 @@ impl FromStr for SupervisorTopology { match s { "combined" => Ok(Self::Combined), "sidecar" => Ok(Self::Sidecar), + "cni-sidecar" => Ok(Self::CniSidecar), other => Err(format!("unknown topology '{other}'")), } } @@ -564,6 +570,16 @@ mod tests { assert_eq!(cfg.topology, SupervisorTopology::Combined); } + #[test] + fn serde_override_topology_cni_sidecar() { + let json = serde_json::json!({ + "topology": "cni-sidecar" + }); + let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); + assert_eq!(cfg.topology, SupervisorTopology::CniSidecar); + assert_eq!(cfg.topology.to_string(), "cni-sidecar"); + } + #[test] fn serde_rejects_sidecar_binary_identity_field() { let json = serde_json::json!({ diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index 2d947b8a28..b689023f43 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -1445,6 +1445,22 @@ const SIDECAR_TLS_VOLUME_NAME: &str = "openshell-supervisor-tls"; const SIDECAR_TLS_MOUNT_PATH: &str = "/etc/openshell-tls/proxy"; const SIDECAR_CLIENT_TLS_MOUNT_PATH: &str = "/etc/openshell-tls/proxy/client"; +const SIDECAR_PROXY_PORT: u16 = 3128; + +const OPENSHELL_CNI_ENABLED_ANNOTATION: &str = "openshell.ai/cni"; +const OPENSHELL_CNI_SANDBOX_ID_ANNOTATION: &str = "openshell.ai/sandbox-id"; +const OPENSHELL_CNI_PROXY_UID_ANNOTATION: &str = "openshell.ai/proxy-uid"; +const OPENSHELL_CNI_PROXY_PORT_ANNOTATION: &str = "openshell.ai/proxy-port"; +const OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION: &str = + "openshell.ai/network-enforcement-mode"; +const CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE: &str = "cni-sidecar"; +/// Node label the CNI installer sets once per-node egress enforcement is in +/// place. Sandbox pods in the cni-sidecar topology require it via nodeAffinity +/// so they cannot schedule onto a node before the chained plugin is active +/// (closing the cold-start fail-open window). Must stay in sync with +/// `NODE_READY_LABEL` in `openshell-cni` and the CNI `DaemonSet`'s node-patch RBAC. +const OPENSHELL_CNI_READY_NODE_LABEL: &str = "openshell.ai/cni-ready"; + /// Build the emptyDir volume that holds the supervisor binary. /// /// The init container writes the binary here; the agent container reads it. @@ -1669,6 +1685,7 @@ fn supervisor_sidecar_env( template_environment: &std::collections::HashMap, spec_environment: &std::collections::HashMap, params: &SandboxPodParams<'_>, + topology: SupervisorTopology, ) -> Vec { let mut env = Vec::new(); apply_required_env( @@ -1701,7 +1718,7 @@ fn supervisor_sidecar_env( upsert_env( &mut env, openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY, - "sidecar", + &topology.to_string(), ); upsert_env( &mut env, @@ -1738,6 +1755,7 @@ fn supervisor_sidecar_container( template_environment: &std::collections::HashMap, spec_environment: &std::collections::HashMap, params: &SandboxPodParams<'_>, + topology: SupervisorTopology, ) -> serde_json::Value { let proxy_uid = effective_sidecar_proxy_uid(params); let capabilities = if params.process_binary_aware_network_policy { @@ -1757,7 +1775,7 @@ fn supervisor_sidecar_container( SUPERVISOR_IMAGE_BINARY_PATH, "--mode=network", ], - "env": supervisor_sidecar_env(template_environment, spec_environment, params), + "env": supervisor_sidecar_env(template_environment, spec_environment, params, topology), "securityContext": { "runAsUser": proxy_uid, "runAsGroup": params.sandbox_gid, @@ -1775,6 +1793,22 @@ fn supervisor_sidecar_container( } ] }); + // In the CNI-sidecar topology there is no network-init container to copy the + // gateway client mTLS bundle into the shared TLS emptyDir, because CNI-sidecar + // omits network-init (the node CNI plugin programs egress redirection instead). + // Mount the client TLS secret directly at the client subdir so the network + // supervisor can authenticate to the gateway (policy fetch, callbacks). The + // secret volume is provisioned by the base template with fsGroup-readable mode. + if topology == SupervisorTopology::CniSidecar && !params.client_tls_secret_name.is_empty() { + container["volumeMounts"] + .as_array_mut() + .expect("volumeMounts is an array") + .push(serde_json::json!({ + "name": CLIENT_TLS_VOLUME_NAME, + "mountPath": SIDECAR_CLIENT_TLS_MOUNT_PATH, + "readOnly": true + })); + } if !params.supervisor_image_pull_policy.is_empty() { container["imagePullPolicy"] = serde_json::json!(params.supervisor_image_pull_policy); } @@ -1856,6 +1890,7 @@ fn apply_supervisor_sidecar_topology( template_environment: &std::collections::HashMap, spec_environment: &std::collections::HashMap, params: &SandboxPodParams<'_>, + install_network_init: bool, ) { let Some(spec) = pod_template.get_mut("spec").and_then(|v| v.as_object_mut()) else { return; @@ -1892,12 +1927,14 @@ fn apply_supervisor_sidecar_topology( })); } - let init_containers = spec - .entry("initContainers") - .or_insert_with(|| serde_json::json!([])) - .as_array_mut(); - if let Some(init_containers) = init_containers { - init_containers.push(supervisor_network_init_container(params)); + if install_network_init { + let init_containers = spec + .entry("initContainers") + .or_insert_with(|| serde_json::json!([])) + .as_array_mut(); + if let Some(init_containers) = init_containers { + init_containers.push(supervisor_network_init_container(params)); + } } let Some(containers) = spec.get_mut("containers").and_then(|v| v.as_array_mut()) else { @@ -1979,7 +2016,7 @@ fn apply_supervisor_sidecar_topology( upsert_env( env, openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY, - "sidecar", + ¶ms.topology.to_string(), ); upsert_env( env, @@ -2009,6 +2046,7 @@ fn apply_supervisor_sidecar_topology( template_environment, spec_environment, params, + params.topology, )); } @@ -2033,6 +2071,8 @@ fn apply_workspace_persistence( image: &str, image_pull_policy: &str, sandbox_gid: u32, + workspace_init_uid: u32, + harden_init: bool, ) { let Some(spec) = pod_template.get_mut("spec").and_then(|v| v.as_object_mut()) else { return; @@ -2109,13 +2149,38 @@ fn apply_workspace_persistence( fi" ); + // The workspace-init container runs the user-selected (untrusted) agent + // image. Running it as UID 0 is a policy-bypass risk under binary-aware + // network policy, where the CNI firewall exempts UID 0 so the network + // sidecar can reach the gateway: an init running as root would inherit + // that exemption and egress freely before the sidecar enforces policy. + // Callers pass the sandbox UID in that mode (non-exempt, still able to + // seed the PVC via fsGroup) and 0 otherwise (where UID 0 is not exempt). + // + // Harden the init in every sidecar topology (`harden_init`), not just the + // binary-aware case. A sidecar pod always has a separate network + // supervisor at an exempt UID and shares the pod netns, so a root init + // with default capabilities could switch to the exempt UID (CAP_SETUID) + // or craft packets around the L4 firewall (CAP_NET_RAW) to bypass egress. + // Dropping all capabilities and disabling privilege escalation closes + // both; runAsNonRoot is asserted only when the init already runs non-root + // (binary-aware), since the non-binary-aware sidecar init still needs + // UID 0. Only the Combined topology (no separate exempt sidecar) keeps + // default root capabilities, preserving broad DAC read access to seed the + // PVC from image files owned by other UIDs with restrictive modes. + let mut security_context = serde_json::json!({ "runAsUser": workspace_init_uid }); + if harden_init { + security_context["allowPrivilegeEscalation"] = serde_json::json!(false); + security_context["capabilities"] = serde_json::json!({ "drop": ["ALL"] }); + if workspace_init_uid != 0 { + security_context["runAsNonRoot"] = serde_json::json!(true); + } + } let mut init_spec = serde_json::json!({ "name": WORKSPACE_INIT_CONTAINER_NAME, "image": image, "command": ["sh", "-c", copy_cmd], - "securityContext": { - "runAsUser": 0, - }, + "securityContext": security_context, "volumeMounts": [{ "name": WORKSPACE_VOLUME_NAME, "mountPath": WORKSPACE_INIT_MOUNT_PATH @@ -2235,7 +2300,11 @@ impl Default for SandboxPodParams<'_> { fn validate_sidecar_proxy_identity( params: &SandboxPodParams<'_>, ) -> Result<(), KubernetesDriverError> { - if params.topology == SupervisorTopology::Sidecar && params.proxy_uid == params.sandbox_uid { + if matches!( + params.topology, + SupervisorTopology::Sidecar | SupervisorTopology::CniSidecar + ) && params.proxy_uid == params.sandbox_uid + { return Err(KubernetesDriverError::Precondition(format!( "proxy_uid ({}) must not match sandbox_uid ({}) in sidecar topology", params.proxy_uid, params.sandbox_uid @@ -2402,6 +2471,7 @@ fn sandbox_template_to_k8s_with_validated_config( .iter() .map(|(key, value)| (key.clone(), serde_json::Value::String(value.clone()))) .collect::>(); + let cni_sidecar_topology = params.topology == SupervisorTopology::CniSidecar; if params.provider_spiffe_enabled { pod_labels.insert( LABEL_MANAGED_BY.to_string(), @@ -2435,6 +2505,36 @@ fn sandbox_template_to_k8s_with_validated_config( serde_json::Value::String(params.sandbox_id.to_string()), ); } + if cni_sidecar_topology { + pod_annotations.insert( + OPENSHELL_CNI_ENABLED_ANNOTATION.to_string(), + serde_json::Value::String("enabled".to_string()), + ); + if !params.sandbox_id.is_empty() { + pod_annotations.insert( + OPENSHELL_CNI_SANDBOX_ID_ANNOTATION.to_string(), + serde_json::Value::String(params.sandbox_id.to_string()), + ); + } + // Annotate the UID the sidecar proxy actually runs as so the node CNI + // plugin exempts the right identity from egress redirection. In + // binary-aware mode the sidecar runs as UID 0 (see + // effective_sidecar_proxy_uid); annotating the raw params.proxy_uid + // (e.g. 1337) would make the CNI redirect the sidecar's own egress and + // block it from reaching the gateway. + pod_annotations.insert( + OPENSHELL_CNI_PROXY_UID_ANNOTATION.to_string(), + serde_json::Value::String(effective_sidecar_proxy_uid(params).to_string()), + ); + pod_annotations.insert( + OPENSHELL_CNI_PROXY_PORT_ANNOTATION.to_string(), + serde_json::Value::String(SIDECAR_PROXY_PORT.to_string()), + ); + pod_annotations.insert( + OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION.to_string(), + serde_json::Value::String(CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE.to_string()), + ); + } if !pod_annotations.is_empty() { metadata.insert( "annotations".to_string(), @@ -2466,6 +2566,14 @@ fn sandbox_template_to_k8s_with_validated_config( } apply_pod_driver_config(&mut spec, &driver_config.pod); + // In the cni-sidecar topology, egress enforcement is programmed by the node + // CNI plugin, not an in-pod init container. Require the CNI installer's + // node-ready label so a sandbox pod cannot schedule onto a node before that + // node's chained plugin is active — closing the cold-start fail-open window. + if params.topology == SupervisorTopology::CniSidecar { + apply_cni_ready_node_affinity(&mut spec); + } + // Per-sandbox platform_config.host_users overrides the cluster-wide default. let use_user_namespaces = platform_config_bool(template, "host_users") .map_or(params.enable_user_namespaces, |host_users| !host_users); @@ -2606,7 +2714,7 @@ fn sandbox_template_to_k8s_with_validated_config( if !params.client_tls_secret_name.is_empty() { let client_tls_default_mode = match params.topology { SupervisorTopology::Combined => 0o400, - SupervisorTopology::Sidecar => 0o440, + SupervisorTopology::Sidecar | SupervisorTopology::CniSidecar => 0o440, }; volumes.push(serde_json::json!({ "name": CLIENT_TLS_VOLUME_NAME, @@ -2632,7 +2740,7 @@ fn sandbox_template_to_k8s_with_validated_config( // supervisor containers run with the sandbox GID and need group-read access. let sa_token_default_mode = match params.topology { SupervisorTopology::Combined => 0o400, - SupervisorTopology::Sidecar => 0o440, + SupervisorTopology::Sidecar | SupervisorTopology::CniSidecar => 0o440, }; volumes.push(serde_json::json!({ "name": SERVICE_ACCOUNT_TOKEN_VOLUME_NAME, @@ -2691,6 +2799,16 @@ fn sandbox_template_to_k8s_with_validated_config( &template.environment, spec_environment, params, + true, + ); + } + SupervisorTopology::CniSidecar => { + apply_supervisor_sidecar_topology( + &mut result, + &template.environment, + spec_environment, + params, + false, ); } } @@ -2699,17 +2817,82 @@ fn sandbox_template_to_k8s_with_validated_config( // that /sandbox data survives pod rescheduling. Skipped when the user // provides custom storage through driver_config. if inject_workspace { + // The sidecar topologies run a separate network supervisor at an exempt + // UID and share the pod netns, so their untrusted workspace-init must be + // hardened against egress bypass; the combined topology has no separate + // exempt sidecar and keeps default root capabilities for read access. + let harden_init = matches!( + params.topology, + SupervisorTopology::Sidecar | SupervisorTopology::CniSidecar + ); + // In binary-aware mode the CNI firewall exempts UID 0 itself, so the init + // must not run as root at all; run it as the non-exempt sandbox UID. + // Otherwise it runs as UID 0 but hardened (no CAP_SETUID/CAP_NET_RAW), so + // it cannot switch to the exempt proxy UID or craft raw packets. + let workspace_init_uid = if harden_init && params.process_binary_aware_network_policy { + params.sandbox_uid + } else { + 0 + }; apply_workspace_persistence( &mut result, image, params.image_pull_policy, params.sandbox_gid, + workspace_init_uid, + harden_init, ); } result } +/// Adds a required nodeAffinity term that gates scheduling on the CNI +/// installer's node-ready label. The requirement is added to every existing +/// nodeSelectorTerm (or a fresh one when none exist) — a match expression is +/// conjunctive within a term — so it composes with any operator-supplied +/// affinity without weakening it. `IgnoredDuringExecution` matches upstream +/// behavior: it gates placement, not eviction of running pods. +fn apply_cni_ready_node_affinity(spec: &mut serde_json::Map) { + let requirement = serde_json::json!({ + "key": OPENSHELL_CNI_READY_NODE_LABEL, + "operator": "In", + "values": ["true"], + }); + + let terms = spec + .entry("affinity") + .or_insert_with(|| serde_json::json!({})) + .as_object_mut() + .expect("affinity is an object") + .entry("nodeAffinity") + .or_insert_with(|| serde_json::json!({})) + .as_object_mut() + .expect("nodeAffinity is an object") + .entry("requiredDuringSchedulingIgnoredDuringExecution") + .or_insert_with(|| serde_json::json!({ "nodeSelectorTerms": [] })) + .as_object_mut() + .expect("nodeSelector is an object") + .entry("nodeSelectorTerms") + .or_insert_with(|| serde_json::json!([])) + .as_array_mut() + .expect("nodeSelectorTerms is an array"); + + if terms.is_empty() { + terms.push(serde_json::json!({ "matchExpressions": [requirement] })); + return; + } + for term in terms.iter_mut() { + term.as_object_mut() + .expect("nodeSelectorTerm is an object") + .entry("matchExpressions") + .or_insert_with(|| serde_json::json!([])) + .as_array_mut() + .expect("matchExpressions is an array") + .push(requirement.clone()); + } +} + fn apply_pod_driver_config( spec: &mut serde_json::Map, config: &KubernetesPodDriverConfig, @@ -4546,6 +4729,431 @@ mod tests { assert_eq!(network_init["command"][3], "0"); } + #[test] + fn cni_sidecar_topology_omits_network_init_and_adds_cni_annotations() { + let params = SandboxPodParams { + topology: SupervisorTopology::CniSidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-cni", + proxy_uid: 2200, + sandbox_uid: 1500, + sandbox_gid: 1500, + // Non-binary-aware: the sidecar runs as proxy_uid, so the CNI + // proxy-uid annotation should reflect that raw UID. + process_binary_aware_network_policy: false, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + false, + ¶ms, + ); + + let annotations = pod_template["metadata"]["annotations"].as_object().unwrap(); + assert_eq!( + annotations[OPENSHELL_CNI_ENABLED_ANNOTATION], + serde_json::json!("enabled") + ); + assert_eq!( + annotations[OPENSHELL_CNI_SANDBOX_ID_ANNOTATION], + serde_json::json!("sb-cni") + ); + assert_eq!( + annotations[OPENSHELL_CNI_PROXY_UID_ANNOTATION], + serde_json::json!("2200") + ); + assert_eq!( + annotations[OPENSHELL_CNI_PROXY_PORT_ANNOTATION], + serde_json::json!(SIDECAR_PROXY_PORT.to_string()) + ); + assert_eq!( + annotations[OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION], + serde_json::json!(CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE) + ); + + let init_containers = pod_template["spec"] + .get("initContainers") + .and_then(|containers| containers.as_array()) + .cloned() + .unwrap_or_default(); + assert!( + !init_containers + .iter() + .any(|container| container["name"] == SUPERVISOR_NETWORK_INIT_CONTAINER_NAME) + ); + + let containers = pod_template["spec"]["containers"].as_array().unwrap(); + assert_eq!(containers.len(), 2); + let agent = containers + .iter() + .find(|container| container["name"] == "agent") + .unwrap(); + assert_eq!( + rendered_env(agent, openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY), + Some("cni-sidecar") + ); + assert_eq!( + rendered_env(agent, openshell_core::sandbox_env::NETWORK_ENFORCEMENT_MODE), + Some("sidecar-nftables") + ); + let sidecar = containers + .iter() + .find(|container| container["name"] == SUPERVISOR_NETWORK_SIDECAR_NAME) + .unwrap(); + assert_eq!( + rendered_env(sidecar, openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY), + Some("cni-sidecar") + ); + } + + /// Regression test for the cni-sidecar proxy-uid annotation. + /// + /// In binary-aware mode the network sidecar runs as UID 0 (see + /// `effective_sidecar_proxy_uid`), not `params.proxy_uid`. The CNI proxy-uid + /// annotation tells the node plugin which UID to exempt from egress + /// redirection; annotating the raw `params.proxy_uid` would make the CNI + /// redirect the sidecar's own egress and block it from reaching the gateway. + /// The annotation must reflect the effective (runAsUser) UID. + #[test] + fn cni_sidecar_binary_aware_annotates_effective_proxy_uid() { + let params = SandboxPodParams { + topology: SupervisorTopology::CniSidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-cni-ba", + proxy_uid: 1337, + sandbox_uid: 1500, + sandbox_gid: 1500, + process_binary_aware_network_policy: true, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + false, + ¶ms, + ); + + // The sidecar runs as UID 0 in binary-aware mode; the annotation must + // match so the node CNI exempts the sidecar's own egress. + let annotations = pod_template["metadata"]["annotations"].as_object().unwrap(); + assert_eq!( + annotations[OPENSHELL_CNI_PROXY_UID_ANNOTATION], + serde_json::json!(BINARY_AWARE_SIDECAR_PROXY_UID.to_string()) + ); + + let sidecar = pod_template["spec"]["containers"] + .as_array() + .unwrap() + .iter() + .find(|container| container["name"] == SUPERVISOR_NETWORK_SIDECAR_NAME) + .unwrap(); + assert_eq!( + sidecar["securityContext"]["runAsUser"], + serde_json::json!(BINARY_AWARE_SIDECAR_PROXY_UID) + ); + } + + /// Regression test for the cni-sidecar gateway client mTLS bundle. + /// + /// The cni-sidecar topology omits the network-init container (the node CNI + /// plugin programs egress redirection instead). In the standard sidecar + /// topology, network-init also copies the gateway client mTLS bundle into + /// the shared TLS emptyDir at `/client`; dropping network-init would + /// otherwise leave that path empty, so the network supervisor cannot read + /// its client cert and crash-loops on policy fetch. cni-sidecar must instead + /// mount the client TLS secret directly onto the network sidecar so the + /// bundle is present at the path the sidecar env points at. + #[test] + fn cni_sidecar_topology_mounts_client_tls_on_network_sidecar() { + let params = SandboxPodParams { + topology: SupervisorTopology::CniSidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-cni-tls", + proxy_uid: 2200, + sandbox_uid: 1500, + sandbox_gid: 1500, + client_tls_secret_name: "openshell-client-tls", + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + false, + ¶ms, + ); + + // The client TLS secret volume must be present in the pod spec. + let volumes = pod_template["spec"]["volumes"].as_array().unwrap(); + let client_tls_volume = volumes + .iter() + .find(|volume| volume["name"] == CLIENT_TLS_VOLUME_NAME) + .expect("client TLS secret volume must be provisioned"); + assert_eq!( + client_tls_volume["secret"]["secretName"], + serde_json::json!("openshell-client-tls") + ); + + // The network sidecar must mount that secret at the client subdir the + // sidecar env points OPENSHELL_TLS_CA/CERT/KEY at, read-only. + let containers = pod_template["spec"]["containers"].as_array().unwrap(); + let sidecar = containers + .iter() + .find(|container| container["name"] == SUPERVISOR_NETWORK_SIDECAR_NAME) + .unwrap(); + let mounts = sidecar["volumeMounts"].as_array().unwrap(); + let client_tls_mount = mounts + .iter() + .find(|mount| mount["name"] == CLIENT_TLS_VOLUME_NAME) + .expect("network sidecar must mount the client TLS secret"); + assert_eq!( + client_tls_mount["mountPath"], + serde_json::json!(SIDECAR_CLIENT_TLS_MOUNT_PATH) + ); + assert_eq!(client_tls_mount["readOnly"], serde_json::json!(true)); + + // The sidecar env must reference that same mount path so the mount and + // the consuming paths stay in lockstep. + assert_eq!( + rendered_env(sidecar, "OPENSHELL_TLS_CA"), + Some(format!("{SIDECAR_CLIENT_TLS_MOUNT_PATH}/ca.crt").as_str()) + ); + } + + /// Regression test for the cold-start scheduling gate. + /// + /// cni-sidecar pods must carry a required nodeAffinity on the CNI installer's + /// node-ready label so they cannot land on a node before that node's chained + /// plugin is active. + #[test] + fn cni_sidecar_requires_cni_ready_node_affinity() { + let params = SandboxPodParams { + topology: SupervisorTopology::CniSidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-cni-gate", + proxy_uid: 1337, + sandbox_uid: 1500, + sandbox_gid: 1500, + process_binary_aware_network_policy: true, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + false, + ¶ms, + ); + + let terms = &pod_template["spec"]["affinity"]["nodeAffinity"]["requiredDuringSchedulingIgnoredDuringExecution"] + ["nodeSelectorTerms"]; + let expressions = terms[0]["matchExpressions"].as_array().unwrap(); + let requirement = expressions + .iter() + .find(|expr| expr["key"] == OPENSHELL_CNI_READY_NODE_LABEL) + .expect("cni-sidecar pod must require the cni-ready node label"); + assert_eq!(requirement["operator"], "In"); + assert_eq!(requirement["values"], serde_json::json!(["true"])); + } + + /// The scheduling gate is cni-sidecar-specific; other topologies enforce in + /// pod and must not carry the node-ready affinity. + #[test] + fn non_cni_sidecar_has_no_cni_ready_node_affinity() { + let params = SandboxPodParams { + topology: SupervisorTopology::Sidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-no-gate", + proxy_uid: 2200, + sandbox_uid: 1500, + sandbox_gid: 1500, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + false, + ¶ms, + ); + assert!( + pod_template["spec"]["affinity"]["nodeAffinity"].is_null(), + "non-cni-sidecar pods must not carry cni-ready nodeAffinity" + ); + } + + /// Regression test for the workspace-init policy-bypass hole. + /// + /// The workspace-init container runs the untrusted user image. Under + /// binary-aware network policy the CNI firewall exempts UID 0 so the network + /// sidecar can reach the gateway; an init running as root would inherit that + /// exemption and egress before policy is enforced. It must run as the + /// non-exempt sandbox UID instead. + #[test] + fn binary_aware_workspace_init_runs_as_sandbox_uid() { + let params = SandboxPodParams { + topology: SupervisorTopology::CniSidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-cni-init", + proxy_uid: 1337, + sandbox_uid: 1_000_790_000, + sandbox_gid: 1_000_790_000, + process_binary_aware_network_policy: true, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + true, // inject_workspace + ¶ms, + ); + + let init = pod_template["spec"]["initContainers"] + .as_array() + .unwrap() + .iter() + .find(|container| container["name"] == WORKSPACE_INIT_CONTAINER_NAME) + .expect("workspace-init container must be present"); + let sc = &init["securityContext"]; + assert_eq!( + sc["runAsUser"], + serde_json::json!(1_000_790_000), + "workspace-init must run as the non-exempt sandbox UID in binary-aware mode" + ); + // Hardening: a setuid binary in the untrusted image must not be able to + // regain the exempt UID 0. + assert_eq!(sc["allowPrivilegeEscalation"], serde_json::json!(false)); + assert_eq!(sc["runAsNonRoot"], serde_json::json!(true)); + assert_eq!(sc["capabilities"]["drop"], serde_json::json!(["ALL"])); + } + + /// The UID-0 exemption only exists in the sidecar topologies. A combined + /// deployment with binary-aware policy on must keep root for workspace-init + /// so it retains access to root-owned image and PVC contents. + #[test] + fn combined_topology_workspace_init_keeps_root() { + let params = SandboxPodParams { + topology: SupervisorTopology::Combined, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-combined-init", + sandbox_uid: 1500, + sandbox_gid: 1500, + process_binary_aware_network_policy: true, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + true, // inject_workspace + ¶ms, + ); + + let init = pod_template["spec"]["initContainers"] + .as_array() + .unwrap() + .iter() + .find(|container| container["name"] == WORKSPACE_INIT_CONTAINER_NAME) + .expect("workspace-init container must be present"); + assert_eq!( + init["securityContext"]["runAsUser"], + serde_json::json!(0), + "combined topology keeps root workspace-init even with binary-aware policy" + ); + // Root init keeps default capabilities (UID 0 is not CNI-exempt here) so + // it retains broad DAC read access; the setuid-hardening must NOT apply. + assert!(init["securityContext"]["allowPrivilegeEscalation"].is_null()); + assert!(init["securityContext"]["runAsNonRoot"].is_null()); + assert!(init["securityContext"]["capabilities"].is_null()); + } + + /// Without binary-aware policy the sidecar init still runs as UID 0 (it needs + /// root), but must be hardened: a sidecar pod has a separate exempt network + /// supervisor, so the untrusted init must not keep `CAP_SETUID`/`CAP_NET_RAW` + /// (it could switch to the exempt UID or craft raw packets to bypass egress). + #[test] + fn non_binary_aware_workspace_init_runs_as_root() { + let params = SandboxPodParams { + topology: SupervisorTopology::Sidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-init-root", + proxy_uid: 2200, + sandbox_uid: 1500, + sandbox_gid: 1500, + process_binary_aware_network_policy: false, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + true, // inject_workspace + ¶ms, + ); + + let init = pod_template["spec"]["initContainers"] + .as_array() + .unwrap() + .iter() + .find(|container| container["name"] == WORKSPACE_INIT_CONTAINER_NAME) + .expect("workspace-init container must be present"); + let sc = &init["securityContext"]; + assert_eq!( + sc["runAsUser"], + serde_json::json!(0), + "non-binary-aware sidecar init still needs UID 0" + ); + // Hardened even at UID 0: no CAP_SETUID/CAP_NET_RAW, no privilege + // escalation. runAsNonRoot is NOT set (it runs as root). + assert_eq!(sc["allowPrivilegeEscalation"], serde_json::json!(false)); + assert_eq!(sc["capabilities"]["drop"], serde_json::json!(["ALL"])); + assert!(sc["runAsNonRoot"].is_null()); + } + #[test] fn sidecar_topology_rejects_proxy_uid_matching_sandbox_uid() { let params = SandboxPodParams { @@ -5042,7 +5650,9 @@ mod tests { &mut pod_template, "openshell/sandbox:latest", "IfNotPresent", - 1000, // sandbox_gid + 1000, // sandbox_gid + 0, // workspace_init_uid (root) + false, // harden_init (combined) ); // Init container @@ -5102,6 +5712,8 @@ mod tests { "my-custom-image:v2", "IfNotPresent", 1000, + 0, + false, ); let init_image = pod_template["spec"]["initContainers"][0]["image"] @@ -5124,7 +5736,7 @@ mod tests { } }); - apply_workspace_persistence(&mut pod_template, "img:latest", "Always", 1000); + apply_workspace_persistence(&mut pod_template, "img:latest", "Always", 1000, 0, false); let cmd = pod_template["spec"]["initContainers"][0]["command"] .as_array() diff --git a/crates/openshell-sandbox/src/lib.rs b/crates/openshell-sandbox/src/lib.rs index f9555d9f24..6187c83bec 100644 --- a/crates/openshell-sandbox/src/lib.rs +++ b/crates/openshell-sandbox/src/lib.rs @@ -808,7 +808,7 @@ fn process_enforcement_mode() -> ProcessEnforcementMode { .ok() .as_deref() { - Some("sidecar") => ProcessEnforcementMode::NetworkOnly, + Some("sidecar" | "cni-sidecar") => ProcessEnforcementMode::NetworkOnly, _ => ProcessEnforcementMode::Full, } } diff --git a/deploy/docker/Dockerfile.supervisor b/deploy/docker/Dockerfile.supervisor index c760bbc890..3973993ead 100644 --- a/deploy/docker/Dockerfile.supervisor +++ b/deploy/docker/Dockerfile.supervisor @@ -10,8 +10,9 @@ # path. It also includes nftables so the Kubernetes supervisor sidecar can # install pod-namespace egress enforcement rules. # -# The Rust binary is built natively before this image build runs and staged at: +# The Rust binaries are built natively before this image build runs and staged at: # deploy/docker/.build/prebuilt-binaries//openshell-sandbox +# deploy/docker/.build/prebuilt-binaries//openshell-cni # # Use tasks/scripts/docker-build-image.sh supervisor (or `mise run build:docker:supervisor`) # to stage the binary and build the image in one step. CI builds the binary @@ -23,12 +24,13 @@ FROM alpine:3.22 AS supervisor ARG TARGETARCH -RUN apk add --no-cache nftables iptables iptables-legacy +RUN apk add --no-cache nftables iptables iptables-legacy jq # --chmod=0555 restores execute bits after the actions/upload-artifact + # download-artifact roundtrip strips them. Ownership stays root (0:0) for # Podman image-volume mounts, while world-execute lets the Kubernetes # network sidecar run this binary as the dedicated non-root proxy UID. COPY --chmod=0555 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-sandbox /openshell-sandbox +COPY --chmod=0555 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-cni /openshell-cni ENTRYPOINT ["/openshell-sandbox"] diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index d4310cb9a7..c8e621ec22 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -148,6 +148,25 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | certManager.enabled | bool | `false` | Create cert-manager Issuer and Certificate resources. When enabled, cert-manager owns TLS and the chart runs a JWT-only certgen hook to create the sandbox JWT signing Secret that cert-manager does not manage. | | certManager.serverDnsNames | list | `["openshell","openshell.openshell.svc","openshell.openshell.svc.cluster.local","localhost","openshell.localhost","*.openshell.localhost","host.docker.internal"]` | DNS SANs on the cert-manager-issued server certificate. | | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | +| cni.affinity | object | `{}` | | +| cni.binDir | string | `"/opt/cni/bin"` | Host CNI binary directory. | +| cni.chainDir | string | `""` | Host Multus vendor-cni-chain directory. Only used when mode is "multus-chain". The installer writes openshell-cni.conf here. | +| cni.confDir | string | `"/etc/cni/net.d"` | Host CNI config directory. | +| cni.configFile | string | `""` | Host CNI conflist filename patched by the installer. Empty selects the first non-OpenShell .conflist. | +| cni.enabled | bool | `false` | Install the OpenShell chained CNI plugin as a privileged node DaemonSet. The installer is a CLUSTER SINGLETON: one installation serves every OpenShell release's sandbox pods across all namespaces (enforcement is keyed on pod annotations the gateway sets). Enable it in exactly one release per cluster. Required for supervisor.topology="cni-sidecar" unless cni.external=true. | +| cni.external | bool | `false` | Use a CNI singleton installed by another release/chart instead of installing one here. Set this (with supervisor.topology="cni-sidecar" and cni.enabled=false) on additional gateway releases that share the cluster's existing OpenShell CNI installation. | +| cni.image.pullPolicy | string | `""` | CNI installer image pull policy. Empty uses supervisor.image.pullPolicy, then image.pullPolicy. | +| cni.image.repository | string | `""` | CNI installer image repository. Empty falls back to supervisor.image.repository, then the official supervisor repository. | +| cni.image.tag | string | `""` | CNI installer image tag. Empty falls back to supervisor.image.tag, then image.tag, then chart appVersion. | +| cni.logFile | string | `"/var/log/openshell-cni.log"` | Host log file written by the OpenShell CNI plugin and tailed by the installer DaemonSet. | +| cni.logLevel | string | `"info"` | Log level passed to the OpenShell CNI plugin. | +| cni.mode | string | `"conflist"` | CNI installer strategy. "conflist" appends the OpenShell plugin to an existing CNI .conflist (k3s / vanilla). "multus-chain" writes a standalone plugin .conf into a Multus vendor-cni-chain subdirectory (OpenShift), which never modifies a CNO-managed file. | +| cni.nodeSelector | object | `{}` | | +| cni.openshift.privilegedSCC | bool | `false` | Grant the privileged SecurityContextConstraints to the CNI ServiceAccount. Required on OpenShift for the privileged installer DaemonSet. Creates a ClusterRole + ClusterRoleBinding. | +| cni.resources | object | `{}` | | +| cni.sandboxNamespaces | list | `[]` | Optional STATIC additions to the plugin's enforcement allowlist. The allowlist is normally built automatically: each cni-sidecar release ships a Helm-owned marker ConfigMap (openshell.ai/cni-registration=true) in its sandbox namespace and the singleton aggregates all marker namespaces. This value is unioned with that discovery for namespaces you want enforced without a marker. Pods in namespaces neither registered nor listed are passed through without a Kubernetes API lookup (blast-radius guard). Empty additionally seeds the owner release's sandbox namespace. | +| cni.stateDir | string | `""` | Persistent host directory for plugin credentials (kubeconfig, token, ca.crt) in "multus-chain" mode. Empty falls back to confDir. Must not be a tmpfs path such as one under /run. | +| cni.tolerations | list | `[]` | | | fullnameOverride | string | `""` | Override the full generated resource name. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | | grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". | @@ -189,6 +208,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | sandboxServiceAccount.annotations | object | `{}` | Annotations to add to the generated sandbox service account. | | sandboxServiceAccount.create | bool | `true` | Create a service account for sandbox pods. | | sandboxServiceAccount.name | string | `""` | Existing service account name for sandbox pods when sandboxServiceAccount.create is false. | +| sandboxServiceAccount.openshift.binaryAwareSCC | bool | `false` | Create a minimal SecurityContextConstraints for sandbox pods and grant it to the sandbox ServiceAccount. Required on OpenShift for "sidecar" and "cni-sidecar" topologies when supervisor.sidecar.processBinaryAwareNetworkPolicy is true: the network sidecar must run as UID 0 with SYS_PTRACE and DAC_READ_SEARCH to inspect cross-UID /proc, which restricted-v2 forbids. The SCC is minimal — it adds only those two capabilities plus the image volume type over the restricted baseline; everything else stays locked down (no privileged, no host namespaces, drop ALL, seccomp runtime/default). Creates a SecurityContextConstraints + ClusterRole + ClusterRoleBinding. | | securityContext.allowPrivilegeEscalation | bool | `false` | Whether the gateway container can gain additional privileges. | | securityContext.capabilities.drop | list | `["ALL"]` | Linux capabilities dropped from the gateway container. | | securityContext.runAsNonRoot | bool | `true` | Require the gateway container to run as a non-root user. | @@ -242,9 +262,9 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | supervisor.image.repository | string | `"ghcr.io/nvidia/openshell/supervisor"` | Supervisor image repository. Changing it uses the effective gateway image tag unless tag is also set. | | supervisor.image.tag | string | `""` | Supervisor image tag override. Empty uses the version pinned into the gateway unless repository is changed. | | supervisor.sidecar.processBinaryAwareNetworkPolicy | bool | `true` | Keep process/binary-aware network policy enabled in sidecar topology. When false, the network sidecar runs as proxyUid, drops the extra /proc inspection capabilities, and enforces endpoint/L7 policy without matching policy.binaries. | -| supervisor.sidecar.proxyUid | int | `1337` | UID for relaxed long-running network sidecars in sidecar topology. Strict process/binary-aware sidecars run as UID 0 so Kubernetes grants the required /proc inspection capabilities into the effective set. The network init container installs nftables rules that exempt the effective sidecar UID. | +| supervisor.sidecar.proxyUid | int | `1337` | UID for relaxed long-running network sidecars in sidecar and cni-sidecar topologies. Strict process/binary-aware sidecars run as UID 0 so Kubernetes grants the required /proc inspection capabilities into the effective set. In sidecar topology the network init container installs nftables rules that exempt the effective sidecar UID; in cni-sidecar topology the CNI plugin installs equivalent nftables or iptables rules. | | supervisor.sideloadMethod | string | `""` | How the supervisor binary is delivered into sandbox pods. Empty (default) = auto-detect from cluster version: K8s >= v1.35 -> "image-volume" (ImageVolume enabled by default; GA in v1.36) K8s < v1.35 -> "init-container" (copies via init container + emptyDir) On K8s v1.33-v1.34 with the ImageVolume feature gate manually enabled, set this to "image-volume" explicitly. | -| supervisor.topology | string | `"combined"` | Supervisor pod topology for Kubernetes sandboxes. "combined" runs the current single supervisor container in the agent pod. "sidecar" runs network enforcement in a dedicated sidecar and the process supervisor as a low-capability wrapper in the agent container. | +| supervisor.topology | string | `"combined"` | Supervisor pod topology for Kubernetes sandboxes. "combined" runs the current single supervisor container in the agent pod. "sidecar" runs network enforcement in a dedicated sidecar and the process supervisor as a low-capability wrapper in the agent container. "cni-sidecar" keeps the sidecar runtime model but installs pod-network rules through the OpenShell CNI plugin. | | tolerations | list | `[]` | Tolerations for the gateway pod. | | workload.allowMultiReplicaStatefulSet | bool | `false` | Allow replicaCount > 1 while rendering a StatefulSet. Prefer workload.kind=deployment for external database-backed multi-replica gateways; this override exists for operators who explicitly require StatefulSet identity or storage semantics. | | workload.kind | string | `"statefulset"` | Gateway workload controller kind. Use `statefulset` for the default SQLite database, or `deployment` when server.externalDbSecret points at an external database. | diff --git a/deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml b/deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml new file mode 100644 index 0000000000..36669ba5bd --- /dev/null +++ b/deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Overlay for the Kubernetes CNI-sidecar topology on OpenShift (Multus / OVN-K). +# Injects the OpenShell chained plugin via the Multus vendor-cni-chain auxiliary +# chain instead of patching a CNI .conflist, and grants the privileged SCC. + +cni: + enabled: true + mode: multus-chain + # OpenShift cri-o plugin_dirs use /var/lib/cni/bin; /opt/cni/bin does not exist. + binDir: /var/lib/cni/bin + # Multus vendor-cni-chain subdirectory (derived from the clusterNetwork path in + # 00-multus.conf). Tmpfs, so it is re-asserted by the installer loop. + chainDir: /run/multus/cni/net.d/vendor-cni-chain + # Persistent directory for plugin credentials (chainDir is tmpfs). + stateDir: /etc/kubernetes/cni/openshell + openshift: + privilegedSCC: true + +supervisor: + topology: cni-sidecar + +sandboxServiceAccount: + openshift: + # Sandbox pods run the network sidecar as UID 0 with SYS_PTRACE and + # DAC_READ_SEARCH for binary-aware policy; restricted-v2 forbids this, so + # grant the sandbox SA the minimal purpose-built SCC. + binaryAwareSCC: true diff --git a/deploy/helm/openshell/ci/values-cni-sidecar.yaml b/deploy/helm/openshell/ci/values-cni-sidecar.yaml new file mode 100644 index 0000000000..fb649fa6d7 --- /dev/null +++ b/deploy/helm/openshell/ci/values-cni-sidecar.yaml @@ -0,0 +1,15 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# CI/dev overlay for exercising the Kubernetes CNI sidecar topology. +# This topology requires the OpenShell chained CNI plugin to be installed on +# every node before sandbox pods are created. + +cni: + enabled: true + # k3s configures containerd with k3s-specific CNI paths. + binDir: /bin + confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d + +supervisor: + topology: cni-sidecar diff --git a/deploy/helm/openshell/examples/cni-sidecar-k3s.yaml b/deploy/helm/openshell/examples/cni-sidecar-k3s.yaml new file mode 100644 index 0000000000..acbfbf8ed8 --- /dev/null +++ b/deploy/helm/openshell/examples/cni-sidecar-k3s.yaml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Example values for the cni-sidecar topology on vanilla Kubernetes / k3s. +# +# The OpenShell CNI installer is a CLUSTER SINGLETON: one installation enforces +# every OpenShell release's sandbox pods across all namespaces. Enable it in +# exactly one release per cluster. +# +# Copy and adjust the host CNI paths for your distro, then: +# helm install openshell deploy/helm/openshell -n openshell \ +# -f deploy/helm/openshell/examples/cni-sidecar-k3s.yaml + +cni: + enabled: true + # conflist (default) appends the OpenShell plugin to an existing CNI .conflist. + mode: conflist + # k3s configures containerd with k3s-specific CNI paths. On upstream Kubernetes + # the defaults (/opt/cni/bin, /etc/cni/net.d) usually apply — drop these two. + binDir: /bin + confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d + +supervisor: + topology: cni-sidecar diff --git a/deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml b/deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml new file mode 100644 index 0000000000..7335f9ae24 --- /dev/null +++ b/deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml @@ -0,0 +1,75 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# OPTIONAL, OpenShift-specific: reduce the node-reboot fail-open window for the +# cni-sidecar topology. +# +# Why: on OpenShift (multus-chain) the chained plugin config lives under /run +# (tmpfs) and is wiped on reboot, but the persistent `openshell.ai/cni-ready` +# Node label survives. Between a node coming back up and the OpenShell CNI +# DaemonSet re-installing the plugin, a sandbox pod could schedule onto a node +# with no egress enforcement. A NoSchedule taint repels workloads until +# enforcement is back; the OpenShell CNI DaemonSet tolerates it and REMOVES it +# once enforcement is ready (it never re-adds it, so a transient reconcile +# failure does not over-repel a running node). +# +# HONEST LIMITATION: there is no chart-shippable mechanism that is fully +# race-free. A taint must exist before the kubelet reports the node schedulable, +# which only node-level config can influence. This example uses two layers: +# 1. kubelet `--register-with-taints` — applied when the kubelet FIRST registers +# the node, so a freshly joined node is tainted with no window. (Configure it +# via a KubeletConfig / MachineConfig for your pool; see OpenShift docs.) +# 2. A boot systemd unit that re-asserts the taint on every boot and RETRIES +# until it succeeds — because kubelet does not re-apply register-with-taints +# when an existing Node object is re-registered after reboot. +# A small residual window remains on reboot (kubelet may mark Ready before the +# unit re-taints). `conflist` mode stores the plugin on persistent disk and is +# unaffected by reboot; prefer it, or cordon/drain before rebooting nodes, if the +# residual window is unacceptable. +# +# CAVEATS / prerequisites: +# * Tradeoff: a NoSchedule taint repels ALL non-tolerating pods on the node +# until the OpenShell CNI is ready. Scope this to a sandbox-dedicated +# MachineConfigPool if that is undesirable on shared nodes. +# * RBAC: the taint command must authenticate as an identity allowed to patch +# node taints. The kubelet's own identity is restricted by NodeRestriction, +# so provision a dedicated ServiceAccount kubeconfig (bound to a ClusterRole +# granting nodes get/patch) at the path referenced below. +# * `oc`/`kubectl` is assumed on the node image; adjust for your RHCOS tooling. +# +# The taint key/effect MUST match what the OpenShell CNI removes: +# openshell.ai/cni-not-ready=:NoSchedule +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: worker + name: 50-openshell-cni-boot-taint +spec: + config: + ignition: + version: 3.4.0 + systemd: + units: + - name: openshell-cni-boot-taint.service + enabled: true + contents: | + [Unit] + Description=Taint node not-ready for OpenShell CNI until enforcement is up + # Order as early as possible after the kubelet/API client is available. + # A residual race remains (see the file header); the retry loop below + # keeps trying so a transient failure never leaves the node untainted. + After=kubelet.service + Wants=kubelet.service + [Service] + Type=oneshot + RemainAfterExit=yes + # KUBECONFIG must authenticate as an identity allowed to patch node + # taints (see CAVEATS). Adjust the path/tooling for your nodes. + Environment=KUBECONFIG=/var/lib/openshell/cni-boot-taint.kubeconfig + # Retry until the taint is applied; never give up silently. + ExecStart=/bin/sh -c 'until /usr/bin/oc adm taint node "$(hostname)" openshell.ai/cni-not-ready=:NoSchedule --overwrite; do echo "retrying node taint..."; sleep 2; done' + Restart=on-failure + RestartSec=2 + [Install] + WantedBy=multi-user.target diff --git a/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml b/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml new file mode 100644 index 0000000000..00a2a3af91 --- /dev/null +++ b/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml @@ -0,0 +1,26 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Example values for an ADDITIONAL gateway release on a cluster that already has +# the OpenShell CNI singleton installed by another release. This release does not +# install its own CNI; it reuses the cluster's existing installation. +# +# Registration is AUTOMATIC: this release runs a helm hook that labels its +# sandbox namespace `openshell.ai/sandbox=true`, and the CNI singleton discovers +# every labeled namespace and adds it to the enforcement allowlist within one +# reconcile (~30s). No manual edit of the singleton owner is required. +# +# helm install openshell-team-b deploy/helm/openshell -n team-b \ +# -f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml + +cni: + # Do not install a second CNI installer; use the cluster singleton. + enabled: false + external: true + +supervisor: + topology: cni-sidecar + +sandboxServiceAccount: + openshift: + binaryAwareSCC: true diff --git a/deploy/helm/openshell/examples/cni-sidecar-openshift.yaml b/deploy/helm/openshell/examples/cni-sidecar-openshift.yaml new file mode 100644 index 0000000000..6be2a659ad --- /dev/null +++ b/deploy/helm/openshell/examples/cni-sidecar-openshift.yaml @@ -0,0 +1,39 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Example values for the cni-sidecar topology on OpenShift (Multus / OVN-K). +# +# The OpenShell CNI installer is a CLUSTER SINGLETON: one installation enforces +# every OpenShell release's sandbox pods across all namespaces. Enable it +# (cni.enabled=true) in exactly one release per cluster. Additional gateway +# releases that share it set cni.enabled=false and cni.external=true (see +# cni-sidecar-openshift-extra-release.yaml). +# +# Copy and adjust, then: +# helm install openshell deploy/helm/openshell -n openshell \ +# -f deploy/helm/openshell/examples/cni-sidecar-openshift.yaml + +cni: + enabled: true + # OpenShift's default network is managed by the Cluster Network Operator; there + # is no appendable CNI .conflist, so chain the plugin via Multus instead. + mode: multus-chain + # OpenShift cri-o resolves plugins from /var/lib/cni/bin (/opt/cni/bin is absent). + binDir: /var/lib/cni/bin + # Multus vendor-cni-chain directory (tmpfs; the installer re-asserts it). + chainDir: /run/multus/cni/net.d/vendor-cni-chain + # Persistent directory for plugin credentials (chainDir is tmpfs). + stateDir: /etc/kubernetes/cni/openshell + openshift: + # Grant the privileged SCC to the dedicated CNI ServiceAccount only. + privilegedSCC: true + +supervisor: + topology: cni-sidecar + +sandboxServiceAccount: + openshift: + # The network sidecar runs as UID 0 with SYS_PTRACE + DAC_READ_SEARCH for + # binary-aware policy; restricted-v2 forbids this, so grant the sandbox SA a + # minimal purpose-built SCC. + binaryAwareSCC: true diff --git a/deploy/helm/openshell/skaffold.yaml b/deploy/helm/openshell/skaffold.yaml index 119adf086b..e9125a27ae 100644 --- a/deploy/helm/openshell/skaffold.yaml +++ b/deploy/helm/openshell/skaffold.yaml @@ -121,6 +121,10 @@ deploy: #- ci/values-spire.yaml # To exercise the Kubernetes supervisor sidecar topology: #- ci/values-sidecar.yaml + # To exercise CNI sidecar topology, use the cni-sidecar Skaffold + # profile against a cluster where the OpenShell CNI DaemonSet can + # patch the node CNI conflist. + #- ci/values-cni-sidecar.yaml # To test multi-replica external PostgreSQL behavior: #- ci/values-high-availability.yaml setValueTemplates: @@ -143,3 +147,8 @@ profiles: path: /deploy/helm/releases/0/setValues value: server.disableTls: "false" + - name: cni-sidecar + patches: + - op: add + path: /deploy/helm/releases/0/valuesFiles/- + value: ci/values-cni-sidecar.yaml diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl index 5931047e5f..bb8d5b8d12 100644 --- a/deploy/helm/openshell/templates/_gateway-workload.tpl +++ b/deploy/helm/openshell/templates/_gateway-workload.tpl @@ -17,6 +17,7 @@ metadata: {{- end }} labels: {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: gateway {{- with .Values.podLabels }} {{- toYaml . | nindent 4 }} {{- end }} @@ -36,6 +37,20 @@ spec: {{- end }} securityContext: {{- toYaml .Values.podSecurityContext | nindent 4 }} + {{- if eq (.Values.supervisor.topology | default "combined") "cni-sidecar" }} + # Wait for cluster-wide CNI acknowledgement of this gateway's sandbox namespace + # before the gateway serves. Blocks until every cni-ready node reports the + # namespace in its coverage annotation, so a newly-registered (e.g. additional + # cni.external) release cannot create sandboxes during the discovery window. + initContainers: + - name: wait-cni-coverage + image: {{ include "openshell.cniImage" . | quote }} + imagePullPolicy: {{ include "openshell.cniImagePullPolicy" . | quote }} + command: + - /openshell-cni + - wait-coverage + - {{ .Values.server.sandboxNamespace | default .Release.Namespace | quote }} + {{- end }} containers: - name: openshell-gateway securityContext: diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 1b4598088f..ea67aef7f7 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -48,6 +48,33 @@ app.kubernetes.io/name: {{ include "openshell.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} {{- end }} +{{/* +CNI DaemonSet selector labels. The CNI installer runs a distinct +`app.kubernetes.io/name` (`-cni`) so it does NOT share the gateway's base +selector labels. This lets the gateway Service and workload selectors stay +base-only (name + instance) without also matching CNI pods — which in turn +avoids requiring an immutable-selector or component-label change on existing +gateway releases during upgrade. +*/}} +{{- define "openshell.cniSelectorLabels" -}} +app.kubernetes.io/name: {{ include "openshell.cniName" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Full label set for the CNI DaemonSet and its pods, mirroring openshell.labels +but keyed on the distinct CNI selector labels. +*/}} +{{- define "openshell.cniLabels" -}} +helm.sh/chart: {{ include "openshell.chart" . }} +{{ include "openshell.cniSelectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +app.kubernetes.io/component: cni +{{- end }} + {{/* Create the name of the service account to use */}} @@ -59,6 +86,30 @@ Create the name of the service account to use {{- end }} {{- end }} +{{/* +Cluster-singleton name for the CNI installer and its RBAC/SCC. The CNI installer +is a per-cluster singleton: one chained plugin serves every OpenShell release's +sandbox pods (keyed on pod annotations), and it writes fixed host state (the +`/run` chain file / conflist entry and the `openshell.ai/cni-ready` Node label). +The name is therefore a genuinely FIXED cluster identity — a constant, not +derived from nameOverride/fullname — so two releases with different name +overrides cannot install competing "singletons" over the same host state. A +second release that also sets cni.enabled=true collides on these fixed-named +cluster resources (Helm ownership) and must instead set cni.external=true. +*/}} +{{- define "openshell.cniName" -}} +{{- "openshell-cni" }} +{{- end }} + +{{/* +ServiceAccount used by the CNI installer DaemonSet. Same singleton name so the +privileged SCC and CNI RBAC stay scoped to the CNI installer rather than the +shared gateway service account. +*/}} +{{- define "openshell.cniServiceAccountName" -}} +{{- include "openshell.cniName" . }} +{{- end }} + {{/* Create the name of the service account assigned to sandbox pods */}} @@ -70,6 +121,14 @@ Create the name of the service account assigned to sandbox pods {{- end }} {{- end }} +{{/* +Name of the minimal SecurityContextConstraints (and its ClusterRole/binding) +granted to sandbox pods on OpenShift for binary-aware network policy. +*/}} +{{- define "openshell.sandboxSccName" -}} +{{- printf "%s-sandbox" (include "openshell.fullname" .) | trunc 63 | trimSuffix "-" }} +{{- end }} + {{/* Gateway image reference. Uses image.tag when set; falls back to .Chart.AppVersion so a released chart automatically pulls the matching image without extra overrides. @@ -103,6 +162,23 @@ a repository-only override uses the effective gateway image tag. {{- printf "%s:%s" $repository $tag }} {{- end }} +{{/* +CNI installer image reference. Defaults to the supervisor image because the +supervisor image carries both openshell-sandbox and openshell-cni. The +repository and tag fallbacks mirror openshell.supervisorImage so a +gateway-only tag override (image.tag) keeps the CNI image in lockstep with +the supervisor instead of silently pinning to the chart AppVersion. +*/}} +{{- define "openshell.cniImage" -}} +{{- $repository := .Values.cni.image.repository | default .Values.supervisor.image.repository | default (include "openshell.defaultSupervisorRepository" .) -}} +{{- $tag := .Values.cni.image.tag | default .Values.supervisor.image.tag | default .Values.image.tag | default .Chart.AppVersion -}} +{{- printf "%s:%s" $repository $tag }} +{{- end }} + +{{- define "openshell.cniImagePullPolicy" -}} +{{- .Values.cni.image.pullPolicy | default .Values.supervisor.image.pullPolicy | default .Values.image.pullPolicy -}} +{{- end }} + {{/* Namespaced Issuer (selfSigned) for cert-manager CA bootstrap. */}} diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml new file mode 100644 index 0000000000..c612a9d6c6 --- /dev/null +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -0,0 +1,426 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- $mode := .Values.cni.mode | default "conflist" -}} +{{- if and (eq (.Values.supervisor.topology | default "combined") "cni-sidecar") (not .Values.cni.enabled) (not .Values.cni.external) -}} +{{- fail "supervisor.topology=cni-sidecar requires the OpenShell CNI installer: set cni.enabled=true to install the cluster singleton here, or cni.external=true if it is installed by another release" -}} +{{- end }} +{{- if .Values.cni.enabled }} +{{- if not (has $mode (list "conflist" "multus-chain")) -}} +{{- fail "cni.mode must be \"conflist\" or \"multus-chain\"" -}} +{{- end }} +{{- if and (eq $mode "multus-chain") (not .Values.cni.chainDir) -}} +{{- fail "cni.mode=multus-chain requires cni.chainDir" -}} +{{- end }} +{{- $sandboxNamespace := .Values.server.sandboxNamespace | default .Release.Namespace }} +{{- $cniSandboxNamespaces := .Values.cni.sandboxNamespaces | default (list $sandboxNamespace) }} +{{- $cniLogFile := .Values.cni.logFile | default "/var/log/openshell-cni.log" }} +{{- $cniLogDir := dir $cniLogFile }} +{{- $stateDir := .Values.cni.confDir }} +{{- if eq $mode "multus-chain" }} +{{- $stateDir = .Values.cni.stateDir | default .Values.cni.confDir }} +{{- end }} +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: {{ include "openshell.cniName" . }} + labels: + {{- include "openshell.cniLabels" . | nindent 4 }} +spec: + selector: + matchLabels: + {{- include "openshell.cniSelectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "openshell.cniLabels" . | nindent 8 }} + spec: + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + serviceAccountName: {{ include "openshell.cniServiceAccountName" . }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: install-cni + image: {{ include "openshell.cniImage" . | quote }} + imagePullPolicy: {{ include "openshell.cniImagePullPolicy" . | quote }} + env: + # Used by `openshell-cni node-ready` to label this node ready/unready + # so the gateway can gate sandbox scheduling on per-node enforcement. + - name: NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + # Used by `openshell-cni daemonset-active` so the preStop hook can tell + # an ordinary pod restart from a real teardown (helm uninstall). + - name: DS_NAME + value: {{ include "openshell.cniName" . }} + - name: DS_NAMESPACE + value: {{ .Release.Namespace | quote }} + command: + - sh + - -ec + - | + bin_dir="/host{{ .Values.cni.binDir }}" + state_dir="/host{{ $stateDir }}" + host_state_dir="{{ $stateDir }}" + log_file="{{ $cniLogFile }}" + host_log_file="/host${log_file}" + kubeconfig="${state_dir}/openshell-cni-kubeconfig" + token_file="${state_dir}/openshell-cni-token" + ca_file="${state_dir}/openshell-cni-ca.crt" + plugin_config="${state_dir}/openshell-cni-plugin.json" + mkdir -p "${bin_dir}" "${state_dir}" + touch "${host_log_file}" + chmod 0644 "${host_log_file}" + install -m 0755 /openshell-cni "${bin_dir}/openshell-cni" + # Create the token atomically at mode 0600 (install, not cp+chmod) so + # this node-patch-capable credential is never briefly world-readable. + install -m 0644 /var/run/secrets/kubernetes.io/serviceaccount/ca.crt "${ca_file}" + install -m 0600 /var/run/secrets/kubernetes.io/serviceaccount/token "${token_file}" + cat > "${kubeconfig}" </dev/null)"; then + : + elif [ -f "${plugin_config}" ]; then + echo "OpenShell CNI: namespace discovery failed; keeping installed allowlist" >&2 + return 0 + else + discovered="" + fi + allowlist_json="$( + { printf '%s' "${sandbox_static_json}" | jq -r '.[]'; printf '%s\n' "${discovered}"; } \ + | grep -v '^$' | sort -u | jq -R . | jq -s . + )" + config_version="$(printf '%s|%s|%s|%s|v3' "${owner}" "${host_state_dir}/openshell-cni-kubeconfig" "{{ .Values.cni.logLevel }}" "${allowlist_json}" | sha256sum | cut -d' ' -f1 | cut -c1-16)" + _pc_tmp="$(mktemp "${state_dir}/openshell-cni-plugin.XXXXXX")" + cat > "${_pc_tmp}" <&2 + publish_coverage + } + # Publish, on this node, the CSV of namespaces we currently enforce + # (from the installed plugin config) so gateways can wait for + # cluster-wide acknowledgement of their namespace before serving. + publish_coverage() { + _csv="$(jq -r '(.openshell.sandboxNamespaces // []) | join(",")' "${plugin_config}" 2>/dev/null || echo "")" + "${openshell_cni}" set-node-coverage "${_csv}" \ + || echo "OpenShell CNI: failed to publish coverage" >&2 + } + # Fencing (clearing readiness) is security-critical: while enforcement + # is not in place the node must not advertise readiness. Retry a few + # times so a transient API error does not silently leave stale + # readiness; log loudly if it ultimately fails. + mark_unready() { + _i=0 + while [ "${_i}" -lt 5 ]; do + if "${openshell_cni}" node-ready --clear; then + return 0 + fi + _i=$((_i + 1)) + sleep 2 + done + echo "OpenShell CNI: FAILED to fence node after retries; readiness may be stale" >&2 + return 1 + } + {{- if eq $mode "conflist" }} + conf_dir="/host{{ .Values.cni.confDir }}" + resolve_conflist_target() { + {{- if .Values.cni.configFile }} + printf '%s' "${conf_dir}/{{ .Values.cni.configFile }}" + {{- else }} + find "${conf_dir}" -maxdepth 1 -type f -name '*.conflist' ! -name '*openshell*' | sort | head -n 1 + {{- end }} + } + # True when the target already carries our openshell-cni plugin at the + # desired config version. Readiness is bound to configVersion so a + # stale same-owner entry (e.g. after a config change) is treated as + # unhealthy and repaired before the node is re-marked ready. + conflist_ours() { + _t="$1" + [ -n "${_t}" ] && [ -f "${_t}" ] && jq -e --arg o "${owner}" --arg v "${config_version}" \ + 'any((.plugins // [])[]; .type == "openshell-cni" and .openshell.owner == $o and .openshell.configVersion == $v)' \ + "${_t}" >/dev/null 2>&1 + } + patch_conflist() { + target="$(resolve_conflist_target)" + if [ -z "${target}" ] || [ ! -f "${target}" ]; then + echo "OpenShell CNI requires an existing CNI .conflist in ${conf_dir}" >&2 + return 1 + fi + # The patch below preserves every non-openshell-cni plugin and + # replaces only the openshell-cni entry, so it never clobbers a + # foreign plugin. The plugin type is itself the ownership signal + # (the cluster singleton always writes owner "openshell"), so any + # existing openshell-cni entry — including one from an older owner + # format — is ours to upgrade in place. + if [ ! -f "${target}.openshell-backup" ]; then + cp "${target}" "${target}.openshell-backup" + fi + tmp="$(mktemp "${conf_dir}/openshell-cni.XXXXXX")" + # Only replace the live conflist when jq succeeded AND wrote a + # non-empty file. This function is invoked through `||`, which + # disables `set -e` inside it, so a bare `jq; mv` would clobber a + # working host conflist with an empty file on jq failure and break + # node networking. Fail closed instead: leave the target intact. + if jq --slurpfile openshell "${plugin_config}" ' + .plugins = ((.plugins // []) | map(select(.type != "openshell-cni")) + [$openshell[0]]) + ' "${target}" > "${tmp}" && [ -s "${tmp}" ]; then + mv "${tmp}" "${target}" + else + rm -f "${tmp}" + echo "OpenShell CNI conflist patch failed (jq error); left ${target} unchanged" >&2 + return 1 + fi + } + # Fence before the initial patch too: if a stale readiness label + # survives from a previous incarnation and this patch fails, the node + # must not keep advertising readiness. + if ! conflist_ours "$(resolve_conflist_target)"; then + mark_unready + fi + patch_conflist || exit 1 + mark_ready + echo "OpenShell CNI installed (conflist mode); plugin log ${log_file}; firewall backend selected during CNI ADD" + {{- else }} + chain_dir="/host{{ .Values.cni.chainDir }}" + chain_conf="${chain_dir}/openshell-cni.conf" + mkdir -p "${chain_dir}" + # True only when the chain conf is a well-formed openshell-cni config + # at the desired config version. Mere file existence is not enough — + # an empty, corrupt, stale-version, or foreign file must NOT mark the + # node ready. + chain_conf_healthy() { + [ -f "${chain_conf}" ] && jq -e --arg o "${owner}" --arg v "${config_version}" ' + .type == "openshell-cni" + and .openshell.owner == $o + and .openshell.configVersion == $v + and (.openshell.kubeconfig // "") != "" + ' "${chain_conf}" >/dev/null 2>&1 + } + write_chain_conf() { + # openshell-cni.conf is an OpenShell-owned file (fixed name, our + # plugin type); the cluster singleton always writes owner + # "openshell", so overwriting an existing one — including from an + # older owner format — is an in-place upgrade, not a clobber. + _tmp="$(mktemp "${chain_dir}/openshell-cni.XXXXXX")" + # Same fail-closed contract as patch_conflist: never install a + # truncated chain config if jq fails. + if jq -n --slurpfile openshell "${plugin_config}" \ + '{"cniVersion": "1.0.0", "name": "openshell-cni"} + $openshell[0]' > "${_tmp}" && [ -s "${_tmp}" ]; then + mv "${_tmp}" "${chain_conf}" + else + rm -f "${_tmp}" + echo "OpenShell CNI chain conf generation failed (jq error); left ${chain_conf} unchanged" >&2 + return 1 + fi + } + # Fence before the initial write too (see conflist path). + if ! chain_conf_healthy; then + mark_unready + fi + write_chain_conf || exit 1 + mark_ready + echo "OpenShell CNI installed (multus-chain mode) at ${chain_conf}; plugin log ${log_file}; firewall backend selected during CNI ADD" + {{- end }} + tail -n 0 -F "${host_log_file}" & + tail_pid="$!" + trap 'kill "${tail_pid}" 2>/dev/null || true; exit 0' INT TERM + trap 'kill "${tail_pid}" 2>/dev/null || true' EXIT + while true; do + cp /var/run/secrets/kubernetes.io/serviceaccount/token "${token_file}" + chmod 0600 "${token_file}" + # Refresh the aggregated allowlist (picks up newly labeled sandbox + # namespaces); config_version changes when the set does, so the + # health check below repairs the entry and re-enforces. + write_plugin_config + # Fence before repair: the instant enforcement is not verifiably in + # place, clear readiness so no new sandbox pod schedules here, then + # attempt repair and only re-mark ready once it is healthy again. + {{- if eq $mode "conflist" }} + if conflist_ours "$(resolve_conflist_target)"; then + mark_ready + else + mark_unready + if patch_conflist; then mark_ready; fi + fi + {{- else }} + if chain_conf_healthy; then + mark_ready + else + mark_unready + if write_chain_conf; then mark_ready; fi + fi + {{- end }} + sleep 30 + done + securityContext: + privileged: true + allowPrivilegeEscalation: true + lifecycle: + preStop: + exec: + command: + - sh + - -ec + - | + # Enforcement lives in the host CNI config and survives pod + # restarts, so an ordinary restart/rolling update must NOT strip + # it (that would fail open for running sandboxes). Only tear it + # down on a real teardown: daemonset-active exits 0 (preserve) + # unless the owning DaemonSet is confirmed gone/terminating. + if /openshell-cni daemonset-active; then + echo "OpenShell CNI: DaemonSet still active; preserving enforcement across restart" >&2 + exit 0 + fi + # Teardown: fence the node first and only remove enforcement if + # fencing succeeded, so we never strip enforcement while the node + # still advertises readiness. + if ! /openshell-cni node-ready --clear; then + echo "OpenShell CNI: fencing failed during teardown; leaving enforcement in place" >&2 + exit 0 + fi + {{- if eq $mode "conflist" }} + conf_dir="/host{{ .Values.cni.confDir }}" + for target in "${conf_dir}"/*.conflist; do + [ -f "${target}" ] || continue + tmp="$(mktemp "${conf_dir}/openshell-cni-cleanup.XXXXXX")" + jq ' + if .plugins then + .plugins = (.plugins | map(select(.type != "openshell-cni"))) + else + . + end + ' "${target}" > "${tmp}" + mv "${tmp}" "${target}" + done + {{- else }} + rm -f "/host{{ .Values.cni.chainDir }}/openshell-cni.conf" + {{- end }} + rm -f \ + "/host{{ $stateDir }}/openshell-cni-ca.crt" \ + "/host{{ $stateDir }}/openshell-cni-kubeconfig" \ + "/host{{ $stateDir }}/openshell-cni-plugin.json" \ + "/host{{ $stateDir }}/openshell-cni-token" + volumeMounts: + - name: cni-bin + mountPath: /host{{ .Values.cni.binDir }} + {{- if eq $mode "conflist" }} + - name: cni-conf + mountPath: /host{{ .Values.cni.confDir }} + {{- else }} + - name: cni-chain + mountPath: /host{{ .Values.cni.chainDir }} + - name: cni-state + mountPath: /host{{ $stateDir }} + {{- end }} + - name: cni-log + mountPath: /host{{ $cniLogDir }} + {{- with .Values.cni.resources }} + resources: + {{- toYaml . | nindent 12 }} + {{- end }} + volumes: + - name: cni-bin + hostPath: + path: {{ .Values.cni.binDir | quote }} + type: DirectoryOrCreate + {{- if eq $mode "conflist" }} + - name: cni-conf + hostPath: + path: {{ .Values.cni.confDir | quote }} + type: Directory + {{- else }} + - name: cni-chain + hostPath: + path: {{ .Values.cni.chainDir | quote }} + type: DirectoryOrCreate + - name: cni-state + hostPath: + path: {{ $stateDir | quote }} + type: DirectoryOrCreate + {{- end }} + - name: cni-log + hostPath: + path: {{ $cniLogDir | quote }} + type: DirectoryOrCreate + {{- with .Values.cni.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + tolerations: + # Tolerate the boot-time not-ready taint so the installer can run on a + # freshly rebooted node and remove the taint once enforcement is ready. + - key: openshell.ai/cni-not-ready + operator: Exists + effect: NoSchedule + {{- with .Values.cni.tolerations }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.cni.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} +{{- end }} diff --git a/deploy/helm/openshell/templates/cni-rbac.yaml b/deploy/helm/openshell/templates/cni-rbac.yaml new file mode 100644 index 0000000000..657cd0f270 --- /dev/null +++ b/deploy/helm/openshell/templates/cni-rbac.yaml @@ -0,0 +1,76 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if .Values.cni.enabled }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "openshell.cniServiceAccountName" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.cniLabels" . | nindent 4 }} +--- +# The CNI installer is a cluster singleton: one chained plugin enforces every +# OpenShell release's sandbox pods (keyed on pod annotations) across all +# namespaces. Its RBAC is therefore cluster-scoped. It is still minimal: +# - pods get: read a sandbox pod's annotations during CNI ADD (any namespace, +# because the singleton serves all sandbox namespaces). +# - nodes get/patch: set/clear the per-node readiness label for the gate. +# - daemonsets get (this DaemonSet only): let preStop tell an ordinary restart +# from a real teardown. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "openshell.cniName" . }} + labels: + {{- include "openshell.cniLabels" . | nindent 4 }} +rules: + - apiGroups: + - "" + resources: + - pods + verbs: + - get + # list sandbox pods (openshell.ai/managed-by=openshell) to keep a namespace + # enforced while its sandboxes are still running (drain-gated prune). + - list + # list registration marker ConfigMaps (openshell.ai/cni-registration=true) to + # aggregate their namespaces into the plugin allowlist (automatic multi-release + # discovery with Helm-owned lifecycle). + - apiGroups: + - "" + resources: + - configmaps + verbs: + - list + - apiGroups: + - "" + resources: + - nodes + verbs: + - get + - patch + - apiGroups: + - apps + resources: + - daemonsets + resourceNames: + - {{ include "openshell.cniName" . }} + verbs: + - get +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "openshell.cniName" . }} + labels: + {{- include "openshell.cniLabels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "openshell.cniName" . }} +subjects: + - kind: ServiceAccount + name: {{ include "openshell.cniServiceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/deploy/helm/openshell/templates/cni-registration.yaml b/deploy/helm/openshell/templates/cni-registration.yaml new file mode 100644 index 0000000000..e41e2a198a --- /dev/null +++ b/deploy/helm/openshell/templates/cni-registration.yaml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if eq (.Values.supervisor.topology | default "combined") "cni-sidecar" }} +{{- $sandboxNamespace := .Values.server.sandboxNamespace | default .Release.Namespace }} +# Helm-owned registration marker for this release's sandbox namespace. The CNI +# singleton lists ConfigMaps labeled openshell.ai/cni-registration=true and +# aggregates their namespaces into the enforcement allowlist, so every +# cni-sidecar release (owner or cni.external) is discovered automatically. As a +# normal Helm resource it is deleted on uninstall and moved on a sandboxNamespace +# change, so registrations have proper lifecycle — no orphaned state. +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "openshell.fullname" . }}-cni-registration + namespace: {{ $sandboxNamespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni + openshell.ai/cni-registration: "true" +data: + release: {{ .Release.Name | quote }} + releaseNamespace: {{ .Release.Namespace | quote }} +{{- end }} diff --git a/deploy/helm/openshell/templates/cni-scc.yaml b/deploy/helm/openshell/templates/cni-scc.yaml new file mode 100644 index 0000000000..ef9bd8643e --- /dev/null +++ b/deploy/helm/openshell/templates/cni-scc.yaml @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if and .Values.cni.enabled .Values.cni.openshift.privilegedSCC }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "openshell.cniName" . }}-scc + labels: + {{- include "openshell.cniLabels" . | nindent 4 }} +rules: + - apiGroups: + - security.openshift.io + resources: + - securitycontextconstraints + resourceNames: + - privileged + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "openshell.cniName" . }}-scc + labels: + {{- include "openshell.cniLabels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "openshell.cniName" . }}-scc +subjects: + - kind: ServiceAccount + name: {{ include "openshell.cniServiceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/deploy/helm/openshell/templates/sandbox-scc.yaml b/deploy/helm/openshell/templates/sandbox-scc.yaml new file mode 100644 index 0000000000..91b5287de6 --- /dev/null +++ b/deploy/helm/openshell/templates/sandbox-scc.yaml @@ -0,0 +1,85 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if .Values.sandboxServiceAccount.openshift.binaryAwareSCC }} +apiVersion: security.openshift.io/v1 +kind: SecurityContextConstraints +metadata: + name: {{ include "openshell.sandboxSccName" . }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: sandbox +# Minimal SCC for binary-aware network sidecars: the restricted-v2 baseline +# plus exactly what cross-UID /proc inspection requires — UID 0 and the two +# capabilities SYS_PTRACE and DAC_READ_SEARCH — and the image volume type used +# to sideload the supervisor binary. Everything else stays locked down. +allowPrivilegedContainer: false +allowPrivilegeEscalation: false +allowHostNetwork: false +allowHostPorts: false +allowHostPID: false +allowHostIPC: false +allowHostDirVolumePlugin: false +readOnlyRootFilesystem: false +defaultAddCapabilities: null +requiredDropCapabilities: + - ALL +allowedCapabilities: + - SYS_PTRACE + - DAC_READ_SEARCH +runAsUser: + type: RunAsAny +seLinuxContext: + type: MustRunAs +fsGroup: + type: RunAsAny +supplementalGroups: + type: RunAsAny +seccompProfiles: + - runtime/default +volumes: + - configMap + - csi + - downwardAPI + - emptyDir + - ephemeral + - image + - persistentVolumeClaim + - projected + - secret +users: [] +groups: [] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "openshell.sandboxSccName" . }}-scc + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: sandbox +rules: + - apiGroups: + - security.openshift.io + resources: + - securitycontextconstraints + resourceNames: + - {{ include "openshell.sandboxSccName" . }} + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "openshell.sandboxSccName" . }}-scc + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: sandbox +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "openshell.sandboxSccName" . }}-scc +subjects: + - kind: ServiceAccount + name: {{ include "openshell.sandboxServiceAccountName" . }} + namespace: {{ include "openshell.sandboxNamespace" . }} +{{- end }} diff --git a/deploy/helm/openshell/tests/cni_daemonset_test.yaml b/deploy/helm/openshell/tests/cni_daemonset_test.yaml new file mode 100644 index 0000000000..5a3d3f5ebc --- /dev/null +++ b/deploy/helm/openshell/tests/cni_daemonset_test.yaml @@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: cni daemonset +templates: + - templates/cni-daemonset.yaml +release: + name: openshell + namespace: openshell +tests: + - it: does not render by default + template: templates/cni-daemonset.yaml + asserts: + - hasDocuments: + count: 0 + + - it: renders privileged cni installer daemonset when enabled + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + asserts: + - isKind: + of: DaemonSet + - equal: + path: spec.template.spec.hostNetwork + value: true + - equal: + path: spec.template.spec.containers[0].securityContext.privileged + value: true + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-bin + mountPath: /host/opt/cni/bin + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-conf + mountPath: /host/etc/cni/net.d + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-log + mountPath: /host/var/log + # Tolerates the boot-time not-ready taint so it can run on a rebooted node + # and remove the taint once enforcement is ready. + - contains: + path: spec.template.spec.tolerations + content: + key: openshell.ai/cni-not-ready + operator: Exists + effect: NoSchedule + # Singleton owner + namespace allowlist defaulting to the release namespace. + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'sandbox_static_json=.\["openshell"\].' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'owner="openshell"' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '"logFile": "\$\{log_file\}"' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: "firewall backend selected during CNI ADD" + - matchRegex: + path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] + pattern: 'select\(\.type != "openshell-cni"\)' + - matchRegex: + path: spec.template.spec.serviceAccountName + pattern: -cni$ + # The CNI DaemonSet uses a distinct app name so the gateway Service and + # workload base selectors (name + instance) never match CNI pods. + - equal: + path: spec.selector.matchLabels["app.kubernetes.io/name"] + value: openshell-cni + - notExists: + path: spec.selector.matchLabels["app.kubernetes.io/component"] + - equal: + path: spec.template.metadata.labels["app.kubernetes.io/name"] + value: openshell-cni + - equal: + path: spec.template.metadata.labels["app.kubernetes.io/component"] + value: cni + # Readiness gate: the installer needs its node name and marks the node + # ready/unready so the gateway can gate sandbox scheduling. + - contains: + path: spec.template.spec.containers[0].env + content: + name: NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'node-ready --set' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'node-ready --clear' + - matchRegex: + path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] + pattern: 'node-ready --clear' + # preStop preserves enforcement across ordinary restarts (only tears down + # on real teardown, gated by daemonset-active). + - contains: + path: spec.template.spec.containers[0].env + content: + name: DS_NAME + value: openshell-cni + - matchRegex: + path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] + pattern: 'daemonset-active' + # Singleton owner stamp. + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'owner="openshell"' + # Auto-discovery: reconcile aggregates marker-ConfigMap namespaces, and + # publishes per-node coverage for the gateway wait-for-ack gate. + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'list-sandbox-namespaces' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'set-node-coverage' + + - it: aggregates multiple sandbox namespaces into the allowlist + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + cni.sandboxNamespaces: + - team-a + - team-b + asserts: + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'sandbox_static_json=.\["team-a","team-b"\].' + + - it: binds readiness to a config version + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + asserts: + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'config_version=' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '"configVersion": "' + + - it: creates the host token atomically at mode 0600 + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + asserts: + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'install -m 0600 /var/run/secrets/kubernetes.io/serviceaccount/token' + + - it: fails cni-sidecar topology when no CNI installer is available + template: templates/cni-daemonset.yaml + set: + supervisor.topology: cni-sidecar + asserts: + - failedTemplate: + errorMessage: "supervisor.topology=cni-sidecar requires the OpenShell CNI installer: set cni.enabled=true to install the cluster singleton here, or cni.external=true if it is installed by another release" + + - it: allows cni-sidecar with an external singleton and renders no installer + template: templates/cni-daemonset.yaml + set: + supervisor.topology: cni-sidecar + cni.enabled: false + cni.external: true + asserts: + - hasDocuments: + count: 0 + + - it: renders multus-chain conf and state volumes when mode is multus-chain + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + cni.mode: multus-chain + cni.chainDir: /run/multus/cni/net.d/vendor-cni-chain + cni.stateDir: /etc/kubernetes/cni/openshell + asserts: + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-chain + mountPath: /host/run/multus/cni/net.d/vendor-cni-chain + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-state + mountPath: /host/etc/kubernetes/cni/openshell + - notContains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-conf + mountPath: /host/etc/cni/net.d + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'multus-chain mode' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '"name": "openshell-cni"' + - matchRegex: + path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] + pattern: 'openshell-cni\.conf' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'host_state_dir="/etc/kubernetes/cni/openshell"' + - contains: + path: spec.template.spec.volumes + content: + name: cni-state + hostPath: + path: /etc/kubernetes/cni/openshell + type: DirectoryOrCreate + + - it: fails multus-chain mode without a chain dir + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + cni.mode: multus-chain + cni.chainDir: "" + asserts: + - failedTemplate: + errorMessage: cni.mode=multus-chain requires cni.chainDir + + - it: fails on an unknown cni mode + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + cni.mode: bogus + asserts: + - failedTemplate: + errorMessage: cni.mode must be "conflist" or "multus-chain" diff --git a/deploy/helm/openshell/tests/cni_gateway_wait_test.yaml b/deploy/helm/openshell/tests/cni_gateway_wait_test.yaml new file mode 100644 index 0000000000..efdc59dc61 --- /dev/null +++ b/deploy/helm/openshell/tests/cni_gateway_wait_test.yaml @@ -0,0 +1,34 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: gateway cni-coverage wait init +templates: + - templates/gateway-config.yaml + - templates/statefulset.yaml +release: + name: openshell + namespace: openshell +tests: + - it: has no wait init container by default + template: templates/statefulset.yaml + asserts: + - notExists: + path: spec.template.spec.initContainers + + # cni-sidecar gateways must not serve until their namespace is enforced + # cluster-wide, enforced by a wait-coverage init container. + - it: adds the wait-coverage init container in cni-sidecar mode + template: templates/statefulset.yaml + set: + supervisor.topology: cni-sidecar + cni.external: true + asserts: + - equal: + path: spec.template.spec.initContainers[0].name + value: wait-cni-coverage + - equal: + path: spec.template.spec.initContainers[0].command + value: + - /openshell-cni + - wait-coverage + - openshell diff --git a/deploy/helm/openshell/tests/cni_rbac_test.yaml b/deploy/helm/openshell/tests/cni_rbac_test.yaml new file mode 100644 index 0000000000..6f2f14a3fe --- /dev/null +++ b/deploy/helm/openshell/tests/cni_rbac_test.yaml @@ -0,0 +1,107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: cni rbac +templates: + - templates/cni-rbac.yaml +release: + name: openshell +tests: + - it: does not render by default + asserts: + - hasDocuments: + count: 0 + + # The CNI installer is a cluster singleton, so its RBAC is a ServiceAccount + # plus a cluster-scoped ClusterRole/ClusterRoleBinding (no namespaced Role). + - it: renders singleton service account and cluster-scoped rbac when cni enabled + set: + cni.enabled: true + asserts: + - hasDocuments: + count: 3 + - documentIndex: 0 + isKind: + of: ServiceAccount + - documentIndex: 0 + equal: + path: metadata.name + value: openshell-cni + - documentIndex: 1 + isKind: + of: ClusterRole + - documentIndex: 1 + equal: + path: metadata.name + value: openshell-cni + # pods get is cluster-wide because one installer serves all sandbox namespaces. + - documentIndex: 1 + contains: + path: rules + content: + apiGroups: + - "" + resources: + - pods + verbs: + - get + - list + # configmaps list: aggregate registration-marker namespaces into the allowlist. + - documentIndex: 1 + contains: + path: rules + content: + apiGroups: + - "" + resources: + - configmaps + verbs: + - list + - documentIndex: 1 + contains: + path: rules + content: + apiGroups: + - "" + resources: + - nodes + verbs: + - get + - patch + # daemonsets get, scoped by resourceNames to the singleton DaemonSet. + - documentIndex: 1 + contains: + path: rules + content: + apiGroups: + - apps + resources: + - daemonsets + resourceNames: + - openshell-cni + verbs: + - get + - documentIndex: 2 + isKind: + of: ClusterRoleBinding + - documentIndex: 2 + equal: + path: roleRef.name + value: openshell-cni + - documentIndex: 2 + equal: + path: subjects[0].name + value: openshell-cni + + # The singleton name is release-independent (chart name), so a different + # release name still yields the fixed cluster-singleton name. + - it: uses a release-independent singleton name + release: + name: my-gateway + set: + cni.enabled: true + asserts: + - documentIndex: 1 + equal: + path: metadata.name + value: openshell-cni diff --git a/deploy/helm/openshell/tests/cni_registration_test.yaml b/deploy/helm/openshell/tests/cni_registration_test.yaml new file mode 100644 index 0000000000..ea55a121d9 --- /dev/null +++ b/deploy/helm/openshell/tests/cni_registration_test.yaml @@ -0,0 +1,54 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: cni registration marker +templates: + - templates/cni-registration.yaml +release: + name: openshell + namespace: openshell +tests: + - it: does not render without cni-sidecar topology + asserts: + - hasDocuments: + count: 0 + + # Every cni-sidecar release (owner or cni.external) publishes a Helm-owned + # marker ConfigMap so the singleton auto-discovers its sandbox namespace, with + # lifecycle handled by Helm. + - it: renders a labeled marker ConfigMap for a cni-sidecar release + set: + supervisor.topology: cni-sidecar + asserts: + - hasDocuments: + count: 1 + - isKind: + of: ConfigMap + - equal: + path: metadata.labels["openshell.ai/cni-registration"] + value: "true" + - equal: + path: metadata.namespace + value: openshell + + - it: places the marker in the configured sandbox namespace + set: + supervisor.topology: cni-sidecar + server.sandboxNamespace: team-b + asserts: + - equal: + path: metadata.namespace + value: team-b + + # External releases render the marker too (topology drives it, not cni.enabled). + - it: renders the marker for an external release + set: + supervisor.topology: cni-sidecar + cni.enabled: false + cni.external: true + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.labels["openshell.ai/cni-registration"] + value: "true" diff --git a/deploy/helm/openshell/tests/cni_scc_test.yaml b/deploy/helm/openshell/tests/cni_scc_test.yaml new file mode 100644 index 0000000000..e973bca4dc --- /dev/null +++ b/deploy/helm/openshell/tests/cni_scc_test.yaml @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: cni scc +templates: + - templates/cni-scc.yaml +tests: + - it: does not render by default + asserts: + - hasDocuments: + count: 0 + + - it: does not render when cni enabled but scc disabled + set: + cni.enabled: true + asserts: + - hasDocuments: + count: 0 + + - it: renders clusterrole and binding when privilegedSCC enabled + set: + cni.enabled: true + cni.openshift.privilegedSCC: true + asserts: + - hasDocuments: + count: 2 + - documentIndex: 0 + isKind: + of: ClusterRole + - documentIndex: 0 + contains: + path: rules + content: + apiGroups: + - security.openshift.io + resources: + - securitycontextconstraints + resourceNames: + - privileged + verbs: + - use + - documentIndex: 1 + isKind: + of: ClusterRoleBinding + - documentIndex: 1 + matchRegex: + path: subjects[0].name + pattern: -cni$ diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index 90e4f9cef0..03009bb6e7 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -25,6 +25,13 @@ tests: - equal: path: kind value: StatefulSet + # The component label must stay out of the immutable selector so existing + # releases can upgrade; it lives on the pod template instead. + - notExists: + path: spec.selector.matchLabels["app.kubernetes.io/component"] + - equal: + path: spec.template.metadata.labels["app.kubernetes.io/component"] + value: gateway - it: treats a null workload map as the default StatefulSet template: templates/statefulset.yaml @@ -150,6 +157,16 @@ tests: path: data["gateway.toml"] pattern: '(?ms)\[openshell\.drivers\.kubernetes\.sidecar\].*?proxy_uid\s*=\s*2200' + - it: renders cni-sidecar supervisor topology under [openshell.drivers.kubernetes] + template: templates/gateway-config.yaml + set: + cni.enabled: true + supervisor.topology: cni-sidecar + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?topology\s*=\s*"cni-sidecar"' + - it: renders process binary aware network policy under [openshell.drivers.kubernetes.sidecar] template: templates/gateway-config.yaml set: @@ -373,6 +390,13 @@ tests: - equal: path: spec.replicas value: 2 + # The component label must stay out of the immutable selector so existing + # releases can upgrade; it lives on the pod template instead. + - notExists: + path: spec.selector.matchLabels["app.kubernetes.io/component"] + - equal: + path: spec.template.metadata.labels["app.kubernetes.io/component"] + value: gateway - equal: path: spec.template.spec.containers[0].name value: openshell-gateway diff --git a/deploy/helm/openshell/tests/sandbox_scc_test.yaml b/deploy/helm/openshell/tests/sandbox_scc_test.yaml new file mode 100644 index 0000000000..3c5fc6f93f --- /dev/null +++ b/deploy/helm/openshell/tests/sandbox_scc_test.yaml @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: sandbox scc +templates: + - templates/sandbox-scc.yaml +tests: + - it: does not render by default + asserts: + - hasDocuments: + count: 0 + + - it: renders scc, clusterrole and binding when binaryAwareSCC enabled + set: + sandboxServiceAccount.openshift.binaryAwareSCC: true + asserts: + - hasDocuments: + count: 3 + - documentIndex: 0 + isKind: + of: SecurityContextConstraints + - documentIndex: 0 + equal: + path: allowPrivilegedContainer + value: false + - documentIndex: 0 + equal: + path: allowPrivilegeEscalation + value: false + - documentIndex: 0 + equal: + path: requiredDropCapabilities + value: + - ALL + - documentIndex: 0 + equal: + path: allowedCapabilities + value: + - SYS_PTRACE + - DAC_READ_SEARCH + - documentIndex: 0 + equal: + path: runAsUser.type + value: RunAsAny + - documentIndex: 0 + contains: + path: volumes + content: image + - documentIndex: 1 + isKind: + of: ClusterRole + - documentIndex: 1 + contains: + path: rules + content: + apiGroups: + - security.openshift.io + resources: + - securitycontextconstraints + resourceNames: + - RELEASE-NAME-openshell-sandbox + verbs: + - use + - documentIndex: 2 + isKind: + of: ClusterRoleBinding + - documentIndex: 2 + matchRegex: + path: subjects[0].name + pattern: -sandbox$ + - documentIndex: 2 + matchRegex: + path: roleRef.name + pattern: -sandbox-scc$ + + - it: binds the scc to an existing sandbox service account name + set: + sandboxServiceAccount.openshift.binaryAwareSCC: true + sandboxServiceAccount.create: false + sandboxServiceAccount.name: custom-sandbox-sa + asserts: + - documentIndex: 2 + equal: + path: subjects[0].name + value: custom-sandbox-sa + + - it: scopes the scc binding subject to the explicit sandbox namespace + set: + sandboxServiceAccount.openshift.binaryAwareSCC: true + server.sandboxNamespace: agents + asserts: + - documentIndex: 2 + equal: + path: subjects[0].namespace + value: agents diff --git a/deploy/helm/openshell/tests/service_test.yaml b/deploy/helm/openshell/tests/service_test.yaml new file mode 100644 index 0000000000..2d346c00ec --- /dev/null +++ b/deploy/helm/openshell/tests/service_test.yaml @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: gateway service +templates: + - templates/service.yaml +tests: + # The Service selector stays base-only (name + instance) so a helm upgrade + # from a release whose gateway pods predate the component label does not blank + # the Service endpoints. The CNI DaemonSet uses a distinct app name + # (-cni), so base-only selection never matches CNI pods. + - it: selects gateway pods by base labels + template: templates/service.yaml + asserts: + - equal: + path: spec.selector["app.kubernetes.io/name"] + value: openshell + - notExists: + path: spec.selector["app.kubernetes.io/component"] diff --git a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml index a7b02310cf..eaa51affc1 100644 --- a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml +++ b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml @@ -10,6 +10,13 @@ release: namespace: my-namespace tests: + - it: labels gateway pods as gateway component + template: templates/statefulset.yaml + asserts: + - equal: + path: spec.template.metadata.labels["app.kubernetes.io/component"] + value: gateway + - it: mounts the server TLS secret ca.crt as client CA for built-in PKI template: templates/statefulset.yaml set: diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 0525ed475d..82f0ac7ab7 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -48,13 +48,16 @@ supervisor: # "combined" runs the current single supervisor container in the agent pod. # "sidecar" runs network enforcement in a dedicated sidecar and the process # supervisor as a low-capability wrapper in the agent container. + # "cni-sidecar" keeps the sidecar runtime model but installs pod-network rules + # through the OpenShell CNI plugin. topology: "combined" sidecar: - # -- UID for relaxed long-running network sidecars in sidecar topology. - # Strict process/binary-aware sidecars run as UID 0 so Kubernetes grants - # the required /proc inspection capabilities into the effective set. The - # network init container installs nftables rules that exempt the effective - # sidecar UID. + # -- UID for relaxed long-running network sidecars in sidecar and cni-sidecar + # topologies. Strict process/binary-aware sidecars run as UID 0 so Kubernetes + # grants the required /proc inspection capabilities into the effective set. In + # sidecar topology the network init container installs nftables rules that + # exempt the effective sidecar UID; in cni-sidecar topology the CNI plugin + # installs equivalent nftables or iptables rules. proxyUid: 1337 # -- Keep process/binary-aware network policy enabled in sidecar topology. # When false, the network sidecar runs as proxyUid, drops the extra /proc @@ -62,6 +65,66 @@ supervisor: # policy.binaries. processBinaryAwareNetworkPolicy: true +cni: + # -- Install the OpenShell chained CNI plugin as a privileged node DaemonSet. + # The installer is a CLUSTER SINGLETON: one installation serves every OpenShell + # release's sandbox pods across all namespaces (enforcement is keyed on pod + # annotations the gateway sets). Enable it in exactly one release per cluster. + # Required for supervisor.topology="cni-sidecar" unless cni.external=true. + enabled: false + # -- Use a CNI singleton installed by another release/chart instead of + # installing one here. Set this (with supervisor.topology="cni-sidecar" and + # cni.enabled=false) on additional gateway releases that share the cluster's + # existing OpenShell CNI installation. + external: false + # -- Optional STATIC additions to the plugin's enforcement allowlist. The + # allowlist is normally built automatically: each cni-sidecar release ships a + # Helm-owned marker ConfigMap (openshell.ai/cni-registration=true) in its sandbox + # namespace and the singleton aggregates all marker namespaces. This value is + # unioned with that discovery for namespaces you want enforced without a marker. + # Pods in namespaces neither registered nor listed are passed through without a + # Kubernetes API lookup (blast-radius guard). Empty additionally seeds the owner + # release's sandbox namespace. + sandboxNamespaces: [] + # -- CNI installer strategy. "conflist" appends the OpenShell plugin to an + # existing CNI .conflist (k3s / vanilla). "multus-chain" writes a standalone + # plugin .conf into a Multus vendor-cni-chain subdirectory (OpenShift), which + # never modifies a CNO-managed file. + mode: conflist + image: + # -- CNI installer image repository. Empty falls back to supervisor.image.repository, then the official supervisor repository. + repository: "" + # -- CNI installer image pull policy. Empty uses supervisor.image.pullPolicy, then image.pullPolicy. + pullPolicy: "" + # -- CNI installer image tag. Empty falls back to supervisor.image.tag, then image.tag, then chart appVersion. + tag: "" + # -- Host CNI binary directory. + binDir: /opt/cni/bin + # -- Host CNI config directory. + confDir: /etc/cni/net.d + # -- Log level passed to the OpenShell CNI plugin. + logLevel: info + # -- Host log file written by the OpenShell CNI plugin and tailed by the installer DaemonSet. + logFile: /var/log/openshell-cni.log + # -- Host CNI conflist filename patched by the installer. Empty selects the first non-OpenShell .conflist. + configFile: "" + # -- Host Multus vendor-cni-chain directory. Only used when mode is + # "multus-chain". The installer writes openshell-cni.conf here. + chainDir: "" + # -- Persistent host directory for plugin credentials (kubeconfig, token, + # ca.crt) in "multus-chain" mode. Empty falls back to confDir. Must not be a + # tmpfs path such as one under /run. + stateDir: "" + resources: {} + nodeSelector: {} + tolerations: [] + affinity: {} + openshift: + # -- Grant the privileged SecurityContextConstraints to the CNI + # ServiceAccount. Required on OpenShift for the privileged installer + # DaemonSet. Creates a ClusterRole + ClusterRoleBinding. + privilegedSCC: false + # -- Image pull secrets attached to gateway and helper pods. imagePullSecrets: [] # -- Override the chart name used in generated resource names. @@ -84,6 +147,17 @@ sandboxServiceAccount: annotations: {} # -- Existing service account name for sandbox pods when sandboxServiceAccount.create is false. name: "" + openshift: + # -- Create a minimal SecurityContextConstraints for sandbox pods and grant + # it to the sandbox ServiceAccount. Required on OpenShift for "sidecar" and + # "cni-sidecar" topologies when supervisor.sidecar.processBinaryAwareNetworkPolicy + # is true: the network sidecar must run as UID 0 with SYS_PTRACE and + # DAC_READ_SEARCH to inspect cross-UID /proc, which restricted-v2 forbids. + # The SCC is minimal — it adds only those two capabilities plus the image + # volume type over the restricted baseline; everything else stays locked + # down (no privileged, no host namespaces, drop ALL, seccomp runtime/default). + # Creates a SecurityContextConstraints + ClusterRole + ClusterRoleBinding. + binaryAwareSCC: false # -- Extra annotations to add to the gateway pod. podAnnotations: {} diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 7512eaa65e..6cefc6d7b1 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -87,6 +87,90 @@ openshell gateway add http://127.0.0.1:8080 --local --name openshift openshell status ``` +## Enable the cni-sidecar topology + +The `cni-sidecar` topology moves pod-network egress enforcement out of the +sandbox pod and onto a node-level CNI plugin, removing the per-pod privileged +network-init container. See [Supervisor Topology](/kubernetes/topology) for the +model. On OpenShift it uses Multus `multus-chain` mode plus purpose-built SCCs. + +The CNI installer is a **cluster singleton**: one installation enforces every +OpenShell release's sandbox pods across all namespaces. Enable it in exactly one +release per cluster. + + + +### Install with the cni-sidecar example values + +```shell +helm install openshell deploy/helm/openshell \ + --namespace openshell \ + -f deploy/helm/openshell/examples/cni-sidecar-openshift.yaml +``` + +The example sets `cni.mode: multus-chain`, the OpenShift CNI paths, and the two +SCC grants (`cni.openshift.privilegedSCC`, `sandboxServiceAccount.openshift.binaryAwareSCC`). +It requires a Multus thick daemon with `auxiliaryCNIChainName` configured +(default on OpenShift 4.x). + +### Verify the CNI installer is Ready on every node + +```shell +oc -n openshell rollout status daemonset/openshell-cni +oc get nodes -L openshell.ai/cni-ready +``` + +Every schedulable node should show `cni-ready=true`. Sandbox pods carry a +required `nodeAffinity` on this label, so they will not schedule onto a node +before its egress enforcement is active. + +### Create a sandbox and confirm enforcement + +```shell +openshell sandbox create --name probe -- sleep infinity +# The pod reaches 2/2 (agent + network sidecar) only on a cni-ready node: +oc get pod default--probe -n openshell +# Default-deny egress: a disallowed destination is blocked. +oc exec default--probe -n openshell -c agent -- \ + sh -c 'curl -sS -m 10 https://example.com; echo "exit=$?"' +``` + +A blocked request (connection/resolution failure) confirms the sandbox's egress +is routed through the OpenShell network sidecar and enforced by policy. + + + +To run **additional** gateway releases on the same cluster, do not install a +second CNI — reuse the singleton with `cni.enabled=false`, `cni.external=true` +(`-f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml`). +Registration is automatic and race-free: each `cni-sidecar` release ships a +Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true`) in its sandbox +namespace, and the singleton discovers every marker's namespace and adds it to the +enforcement allowlist — no manual edit of the singleton owner is required. Because +the marker is a normal Helm resource, uninstalling the release (or changing its +`sandboxNamespace`) removes it — but deregistration is drain-gated: the singleton +keeps a namespace enforced while it still has running sandbox pods +(`openshell.ai/managed-by=openshell`) and only prunes it once drained, so a +surviving sandbox is never left unenforced. The gateway does not +serve until enforcement is acknowledged: a `wait-cni-coverage` init container +blocks the gateway pod until every enforcement-ready node reports the namespace, +so an additional release never creates sandboxes during the discovery window. +(`cni.sandboxNamespaces` remains available as an optional static addition.) + + +On node reboot the Multus chain file (under `/run`, tmpfs) is wiped while the +`openshell.ai/cni-ready` Node label persists, so a sandbox could briefly schedule +onto a node before enforcement is re-established. To **narrow** this window, apply +the boot-time taint MachineConfig at +`deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml` (read its +caveats first — it is operator-provided node config and requires taint RBAC). The +CNI DaemonSet tolerates the taint and removes it once enforcement is ready. This +does not fully close the window: the taint is applied after the kubelet starts, so +a brief reboot race remains. Use `conflist` mode (plugin stored on persistent +disk, unaffected by reboot), or cordon/drain nodes before rebooting, to avoid it +entirely. + + ## Next Steps - For TLS-enabled deployments, refer to [Managing Certificates](/kubernetes/managing-certificates). diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index c2fca827f1..8242b81319 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -162,6 +162,10 @@ The most commonly changed values are: | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect based on cluster version: clusters running Kubernetes 1.35 or later use `image-volume` (ImageVolume GA in 1.36); older clusters use `init-container`. Set explicitly to `image-volume` on Kubernetes 1.33 or 1.34 with the ImageVolume feature gate enabled, or to `init-container` to force the legacy path on any version. | | `supervisor.topology` | Sandbox pod topology. Refer to [Topology](/kubernetes/topology). | | `supervisor.sidecar.proxyUid` | Non-root UID used when sidecar process/binary-aware network policy is disabled. The default binary-aware sidecar runs as UID 0 instead. The configured UID must not match the sandbox UID. | +| `cni.enabled` | Install the privileged OpenShell CNI DaemonSet (a per-cluster singleton). Enable in one release per cluster when `supervisor.topology=cni-sidecar`. | +| `cni.external` | Use a CNI singleton installed by another release instead of installing one here. Set on additional `cni-sidecar` gateway releases (with `cni.enabled=false`). | +| `cni.configFile` | Existing host CNI conflist to patch. Leave empty to patch the first non-OpenShell `.conflist` in `cni.confDir`. | +| `cni.logFile` | Host log file written by the OpenShell CNI plugin and tailed by the installer DaemonSet. Defaults to `/var/log/openshell-cni.log`. | Use a values file for repeatable deployments: @@ -245,7 +249,7 @@ The gateway exposes `/healthz` for process liveness and `/readyz` for dependency ## Next Steps -- To choose between combined and sidecar sandbox pods, refer to [Topology](/kubernetes/topology). +- To choose between combined, sidecar, and cni-sidecar sandbox topology, refer to [Topology](/kubernetes/topology). - To enable automatic certificate rotation with cert-manager, refer to [Managing Certificates](/kubernetes/managing-certificates). - To expose the gateway externally without port-forwarding, refer to [Ingress](/kubernetes/ingress). - To configure OIDC or reverse-proxy authentication, refer to [Access Control](/kubernetes/access-control). diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index 5bbb18e1ef..3f02d86457 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -3,14 +3,15 @@ # SPDX-License-Identifier: Apache-2.0 title: "Kubernetes Sandbox Topology" sidebar-title: "Topology" -description: "Choose between combined and sidecar supervisor topology for Kubernetes sandbox pods." +description: "Choose between combined, sidecar, and cni-sidecar supervisor topology for Kubernetes sandbox pods." keywords: "Generative AI, Cybersecurity, Kubernetes, Sandboxing, Sidecar, Network Policy, RuntimeClass" position: 2 --- -Kubernetes sandbox pods can run the OpenShell supervisor in `combined` or -`sidecar` topology. Choose the topology based on which controls you need inside -the pod and how much privilege your cluster allows on the agent container. +Kubernetes sandbox pods can run the OpenShell supervisor in `combined`, +`sidecar`, or `cni-sidecar` topology. Choose the topology based on which +controls you need inside the pod, how much privilege your cluster allows on the +agent container, and whether the cluster can install node-level CNI enforcement. ## Choose a Topology @@ -22,6 +23,7 @@ lower-privilege agent container. |---|---|---| | `combined` | You need OpenShell network, filesystem, and process controls in the sandbox workload. | The agent container carries the Linux capabilities the supervisor needs. | | `sidecar` | You need the agent container to run as non-root without added Linux capabilities, and network policy is the primary control. | Privilege-dropping and supervisor mount isolation do not run in the agent container. | +| `cni-sidecar` | You need sidecar-mode behavior without a privileged sandbox-pod init container, and a cluster admin can install the OpenShell CNI plugin on every node. | Requires privileged node-level CNI installation; the agent process supervisor is network-only. | ## Privilege Model @@ -33,6 +35,8 @@ The long-running container permissions differ by topology: | `sidecar` | Agent container, process-only supervisor (`network-only`) | `sandbox_uid:sandbox_gid` | `false` | Drops `ALL` | Agent and workload run without added Linux capabilities. | | `sidecar` | Network supervisor sidecar, binary-aware mode (default) | `0:sandbox_gid` | `false` | Drops `ALL`; adds `SYS_PTRACE` and `DAC_READ_SEARCH` | Root sidecar inspects cross-UID workload `/proc` entries. The nftables fence exempts UID 0, so do not inject other root containers into these pods. | | `sidecar` | Network supervisor sidecar, endpoint/L7-only mode | `proxyUid:sandbox_gid` | `false` | Drops `ALL` | Non-root sidecar enforces endpoint and L7 policy without matching `policy.binaries`. | +| `cni-sidecar` | Agent container, process-only supervisor (`network-only`) | `sandbox_uid:sandbox_gid` | `false` | Drops `ALL` | Agent and workload run without added Linux capabilities. | +| `cni-sidecar` | Network supervisor sidecar | Same as `sidecar` mode | `false` | Same as `sidecar` mode | Node-level CNI installs the pod network rules instead of a pod-local init container. | Short-lived setup containers still have the permissions needed to prepare the pod: @@ -41,6 +45,7 @@ pod: |---|---|---|---|---|---| | `combined` | Supervisor install init container | `0` | Not set | Not set | Copies the supervisor binary into the agent container volume. | | `sidecar` | Network init container | `0` | `false` | Drops `ALL`; adds `NET_ADMIN`, `NET_RAW`, `CHOWN`, and `FOWNER` | Installs pod-local nftables rules and prepares shared sidecar state. | +| `cni-sidecar` | None for network rules | N/A | N/A | N/A | The OpenShell CNI DaemonSet installs rules during pod network setup. | ## Combined Topology @@ -158,12 +163,102 @@ Sidecar pods use `shareProcessNamespace: true` so the network sidecar can resolve workload process and binary identity through `/proc/`. +## CNI Sidecar Topology + +CNI sidecar topology keeps the sidecar runtime model but moves network rule +installation out of the sandbox pod. A privileged OpenShell DaemonSet installs a +chained CNI plugin on each node. During CNI `ADD`, the plugin reads OpenShell +pod annotations and installs the sidecar bypass-prevention rules in the pod +network namespace before the workload starts. + +```mermaid +flowchart TB + Sandbox["agents.x-k8s.io Sandbox"] + + subgraph Node["Kubernetes node"] + DaemonSet["OpenShell CNI DaemonSet"] + CNIPlugin["OpenShell chained CNI plugin"] + Runtime["container runtime
CNI ADD / CHECK"] + + subgraph Pod["Sandbox pod"] + NetNS["pod network namespace"] + + subgraph Agent["agent container"] + ProcessSupervisor["process supervisor
network-only"] + Workload["Agent workload"] + end + + NetworkSidecar["network supervisor sidecar
proxyUid"] + end + end + + Gateway["OpenShell Gateway"] + External["External services"] + + Sandbox --> Pod + DaemonSet -->|"installs binary + patches conflist"| CNIPlugin + Runtime -->|"invokes"| CNIPlugin + CNIPlugin -->|"reads pod annotations"| Pod + CNIPlugin -->|"installs nftables or iptables rules"| NetNS + ProcessSupervisor --> Workload + Workload -->|"egress redirected on loopback"| NetworkSidecar + NetworkSidecar -->|"gateway forwarding"| Gateway + NetworkSidecar -->|"policy-enforced egress"| External +``` + +Use this topology when a cluster admin can install node-level CNI components and +you want sandbox pods to avoid the sidecar topology's privileged network init +container. + +Each node must provide a firewall backend usable from the host-executed plugin. +The plugin prefers `nft` and falls back to `iptables` when `nft` is unavailable. +The installer DaemonSet writes plugin failures to `cni.logFile` and tails that +file, so `kubectl logs daemonset/openshell-cni -c install-cni` includes CNI +`ADD` and `CHECK` errors that would otherwise only appear in pod events or node +logs. + +A per-node scheduling gate ensures enforcement is in place before a sandbox +runs. Once the chained plugin is installed, the installer labels its node +`openshell.ai/cni-ready=true`, and clears the label on shutdown or when a +reconcile tick cannot restore the plugin. The gateway sets a required +`nodeAffinity` on that label for every cni-sidecar sandbox pod, so a pod cannot +schedule onto a node whose egress enforcement is not active. An ungraceful CNI +DaemonSet pod deletion can briefly leave a stale `cni-ready=true` until the pod +is rescheduled and the next reconcile tick re-evaluates it. + + +`cni-sidecar` is experimental. It currently targets normal Kubernetes runtimes +first. Kata Containers and gVisor remain validation targets, because each +runtime must honor the CNI-installed pod-network rules for this topology to +provide the expected network enforcement. + + + +The CNI installer is a **cluster singleton**. Its chained plugin enforces pods +that carry the OpenShell annotations (set only by a gateway on its own sandbox +pods) **and** whose namespace is in the plugin's enforcement allowlist. Pods in +other namespaces are passed through without an API lookup, so the allowlist +bounds the blast radius. The allowlist is built automatically: each `cni-sidecar` +release ships a Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true`) +in its sandbox namespace and the singleton aggregates every marker's namespace +each reconcile — so enable the installer (`cni.enabled=true`) in exactly one +release per cluster, and additional releases (`cni.enabled=false`, +`cni.external=true`) are discovered automatically. Removing a marker (uninstall / +namespace change) deregisters drain-gated: the singleton keeps a namespace +enforced while it still has running sandbox pods and prunes it only once drained. +Each gateway waits (via a `wait-cni-coverage` init container) until every +enforcement-ready node acknowledges its namespace before serving, so there is no +discovery-window bypass. `cni.sandboxNamespaces` is an optional static addition. +The installer treats any `openshell-cni` chained entry as its own and upgrades it +in place. + + ## Credential Exposure -Sidecar topology keeps gateway credentials in the network sidecar. The agent -container does not mount the projected ServiceAccount token used for sandbox -token bootstrap, does not mount the sandbox client TLS secret, and does not get -gateway callback environment variables. +Sidecar and cni-sidecar topologies keep gateway credentials in the network +sidecar. The agent container does not mount the projected ServiceAccount token +used for sandbox token bootstrap, does not mount the sandbox client TLS secret, +and does not get gateway callback environment variables. The network sidecar serves the policy and workload-facing provider environment over a Unix control socket in the shared sidecar state volume. Before launching @@ -190,10 +285,13 @@ gVisor does not provide to the init container. A supported sandboxed runtime strengthens the container boundary while OpenShell focuses on network policy enforcement from the sidecar. +`cni-sidecar` is intended to test whether CNI-installed pod-network rules can +make sidecar-style enforcement work on stricter runtime classes. + Runtime classes do not re-enable the OpenShell privilege-drop or supervisor -mount-isolation controls that sidecar mode relaxes. Use them as an additional -workload boundary, not as a replacement for the combined topology's full -supervisor controls. +mount-isolation controls that sidecar and cni-sidecar modes relax. Use them as +an additional workload boundary, not as a replacement for the combined +topology's full supervisor controls. You can set a default runtime class in the Kubernetes driver configuration or override it per sandbox with driver config: @@ -206,7 +304,10 @@ openshell sandbox create \ ## Enable Sidecar Mode -For direct gateway TOML configuration, set the Kubernetes driver fields: +For direct gateway TOML configuration, set the Kubernetes driver fields for +the topology you want to test. + +For sidecar mode: ```toml [openshell.drivers.kubernetes] @@ -222,6 +323,19 @@ runs the sidecar as UID 0 instead. The network init container exempts the effective sidecar UID from proxy redirection so the sidecar can reach the gateway. +For CNI-sidecar mode: + +```toml +[openshell.drivers.kubernetes] +topology = "cni-sidecar" + +[openshell.drivers.kubernetes.sidecar] +proxy_uid = 1337 +``` + +In CNI-sidecar mode, the OpenShell CNI plugin installs the pod-network rules +that the network init container installs in `sidecar` mode. + When the Helm chart renders `gateway.toml`, set the equivalent chart values: ```yaml @@ -232,6 +346,64 @@ supervisor: processBinaryAwareNetworkPolicy: true ``` +Set `supervisor.topology=cni-sidecar` and enable the CNI installer to use CNI +sidecar mode: + +```yaml +cni: + enabled: true +supervisor: + topology: cni-sidecar + sidecar: + proxyUid: 1337 +``` + +The OpenShell CNI installer requires privileged node access so it can copy the +plugin into the host CNI binary directory and patch the node CNI conflist. + +### OpenShift (Multus / OVN-Kubernetes) + +On OpenShift the default network is managed by the Cluster Network Operator and +there is no CNI `.conflist` to append to. Use `cni.mode: multus-chain`, which +injects the OpenShell plugin through the Multus `vendor-cni-chain` auxiliary chain +without modifying any operator-managed file. The installer also needs the +`privileged` SecurityContextConstraints. + +Install with the example values (a step-by-step OpenShift walkthrough with +verification is in [OpenShift](/kubernetes/openshift#enable-the-cni-sidecar-topology)): + +```shell +helm install openshell deploy/helm/openshell \ + -f deploy/helm/openshell/examples/cni-sidecar-openshift.yaml +``` + +This sets `cni.binDir: /var/lib/cni/bin`, `cni.chainDir: +/run/multus/cni/net.d/vendor-cni-chain`, a persistent `cni.stateDir`, +`cni.openshift.privilegedSCC: true`, and +`sandboxServiceAccount.openshift.binaryAwareSCC: true`. It requires a Multus +thick daemon with `auxiliaryCNIChainName` configured (default on OpenShift 4.x). + +#### Sandbox pod SecurityContextConstraints + +With `supervisor.sidecar.processBinaryAwareNetworkPolicy` enabled (the default), +the network sidecar runs as UID 0 with the `SYS_PTRACE` and `DAC_READ_SEARCH` +capabilities so it can inspect the agent process's `/proc` entries across UID +boundaries. OpenShift's `restricted-v2` SCC forbids UID 0 and those +capabilities, so the sandbox ServiceAccount needs a purpose-built SCC. + +Setting `sandboxServiceAccount.openshift.binaryAwareSCC: true` creates a minimal +SCC — the `restricted-v2` baseline plus only UID 0, `SYS_PTRACE`, +`DAC_READ_SEARCH`, and the `image` volume type used to sideload the supervisor +binary — and a ClusterRole/ClusterRoleBinding granting it to the sandbox +ServiceAccount. Everything else stays locked down: no privileged container, no +host namespaces, `allowPrivilegeEscalation: false`, drop `ALL`, and the +`runtime/default` seccomp profile. + +To run with lower privileges at the cost of binary matching, set +`supervisor.sidecar.processBinaryAwareNetworkPolicy: false`; the sidecar then +runs as `proxyUid` without the extra capabilities, and the built-in +`restricted-v2` SCC suffices (leave `binaryAwareSCC` unset). + Leave `topology` unset, or set it to `combined`, to keep the original single-container supervisor path. For Helm installs, leave `supervisor.topology` unset or set it to `combined`. diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 2fd5717aec..5c0c33e8d5 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -329,6 +329,10 @@ supervisor_sideload_method = "image-volume" # "combined" runs the existing single supervisor container with full process, # filesystem, and network enforcement in the agent container. "sidecar" moves # pod-level network enforcement and gateway session handling into a network sidecar. +# "cni-sidecar" keeps the sidecar runtime model but expects an OpenShell CNI +# plugin to install the pod-network rules before the workload starts. In sidecar +# and cni-sidecar modes, the agent container runs non-root with no added Linux +# capabilities and process/filesystem enforcement is network-only. topology = "combined" grpc_endpoint = "https://openshell-gateway.agents.svc:8080" ssh_socket_path = "/run/openshell/ssh.sock" @@ -362,7 +366,8 @@ provider_spiffe_workload_api_socket_path = "/spiffe-workload-api/spire-agent.soc # UID used by relaxed long-running network sidecars. Strict process/binary-aware # sidecars run as UID 0 so Kubernetes grants the required /proc inspection # capabilities into the effective set. In sidecar topology the network init -# container installs nftables rules that exempt the effective sidecar UID. +# container installs nftables rules that exempt the effective sidecar UID; in +# cni-sidecar topology the OpenShell CNI plugin installs equivalent rules. proxy_uid = 1337 # Keep process/binary-aware network policy enabled in sidecar topology. Set # false to run the sidecar as proxy_uid, drop the sidecar's extra /proc diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 675ffaff6a..4542f7f2f5 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -320,8 +320,8 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA | `supervisor_image` | `supervisor.image.repository` / `supervisor.image.tag` | Override the supervisor image that provides the `openshell-sandbox` binary. The default repository with an empty tag uses the version-pinned image built into the gateway. Changing the repository uses the effective gateway image tag, while setting a tag pins that version explicitly. | | `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the Kubernetes image pull policy for the supervisor image. | | `supervisor_sideload_method` | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect from cluster version. Set to `image-volume` to mount the supervisor OCI image directly as a volume (requires Kubernetes 1.33+ with the ImageVolume feature gate; GA in 1.36), or `init-container` to copy it through an init container on older clusters. | -| `topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, or `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar. | -| `sidecar.proxy_uid` | `supervisor.sidecar.proxyUid` | Non-root UID used by the relaxed sidecar when process/binary-aware network policy is disabled. The default binary-aware sidecar runs as UID 0. The network init container exempts the effective sidecar UID from proxy redirection. | +| `topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar, or `cni-sidecar` to use that sidecar runtime model with CNI-installed pod-network rules. | +| `sidecar.proxy_uid` | `supervisor.sidecar.proxyUid` | Non-root UID used by the relaxed sidecar when process/binary-aware network policy is disabled. The default binary-aware sidecar runs as UID 0. In `sidecar` topology the network init container exempts the effective sidecar UID from proxy redirection; in `cni-sidecar` topology the OpenShell CNI plugin installs equivalent nftables or iptables exemptions. | | `sidecar.process_binary_aware_network_policy` | `supervisor.sidecar.processBinaryAwareNetworkPolicy` | Keep process/binary-aware network policy enabled in `sidecar` topology. The default runs the sidecar as UID 0 with `SYS_PTRACE` and `DAC_READ_SEARCH`. Set false to run as `proxy_uid`, drop both capabilities, and enforce endpoint/L7 policy without matching `policy.binaries`. | | `app_armor_profile` | `server.appArmorProfile` | Set the sandbox agent container's AppArmor profile. Helm defaults this to `Unconfined` so AppArmor-enabled nodes do not block supervisor network namespace setup. Set the Helm value to an empty string to omit the field, or use `RuntimeDefault` or `Localhost/` for operator-managed profiles. | | `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Set the default workspace PVC size for new sandboxes. | @@ -353,6 +353,11 @@ identity mount isolation. Network policy still runs in the sidecar, and sidecar pods set `shareProcessNamespace: true` so the network sidecar can resolve process/binary identity through `/proc/`. +CNI-sidecar mode keeps the sidecar runtime model and local snapshot boundary, +but removes the privileged pod-local network init container. Instead, the +privileged OpenShell CNI DaemonSet installs the nftables or iptables pod-network +rules during CNI `ADD` before the workload starts. + The Kubernetes driver creates namespaced `agents.x-k8s.io` `Sandbox` resources from the Kubernetes SIG Apps [agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) project. It detects the served Sandbox API at runtime, caches the selected API version for the gateway process, and uses `v1beta1` when available before falling back to `v1alpha1`, so supported Agent Sandbox installations work without version-specific operator configuration. The Agent Sandbox controller turns those resources into sandbox pods and related storage. If Agent Sandbox is upgraded in place, restart the OpenShell gateway after the controller and CRD rollout completes so the gateway can detect the served API versions again. diff --git a/docs/superpowers/specs/2026-07-30-cni-sidecar-openshift-multus-design.md b/docs/superpowers/specs/2026-07-30-cni-sidecar-openshift-multus-design.md new file mode 100644 index 0000000000..973e7ab58b --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-cni-sidecar-openshift-multus-design.md @@ -0,0 +1,174 @@ +# CNI-sidecar on OpenShift (Multus / OVN-Kubernetes) — Design + +Date: 2026-07-30 +Branch: `feat/kubernetes-cni-sidecar-topology` +Status: Approved design, pending implementation plan + +## Problem + +The `cni-sidecar` topology installs a chained CNI plugin (`openshell-cni`) on every +node via a privileged DaemonSet. During CNI `ADD`, the plugin reads pod annotations +and installs nftables/iptables bypass-prevention rules in the sandbox pod's network +namespace so the workload's egress is forced through the sidecar proxy +(`SIDECAR_PROXY_PORT = 3128`). + +The installer as written targets a **vanilla / k3s CNI layout**: it searches +`cni.confDir` for a `*.conflist`, appends `openshell-cni` to that file's `.plugins` +array, backs it up, and unpatches on `preStop`. + +This model does not work on OpenShift. Verified on a single-node OpenShift 4.22 +cluster (RHCOS, cri-o 1.35, **OVNKubernetes + Multus**): + +- CNI bin dir is `/var/lib/cni/bin` (cri-o `plugin_dirs`); `/opt/cni/bin` does not exist. +- CNI conf dir is `/etc/kubernetes/cni/net.d/` (cri-o `network_dir`); `/etc/cni/net.d` + does not exist. +- The only conf there is `00-multus.conf` (type `multus-shim`) — a single `.conf` + with **no `.plugins` array**. +- The delegated OVN config (`/run/multus/cni/net.d/10-ovn-kubernetes.conf`) is also a + single-plugin `.conf`, CNO-managed and regenerated. + +So there is no `.conflist` to append to. The installer's `find ... -name '*.conflist'` +step exits 1 (safe, but non-functional). Forcing it at a CNO-managed `.conf` would be +reverted and risks breaking cluster networking. + +## Chosen mechanism: Multus auxiliary CNI chain + +The Multus thick daemon on this cluster has `auxiliaryCNIChainName: "vendor-cni-chain"` +set (`multus-daemon-config` ConfigMap). Multus runs an **isolated auxiliary CNI chain +for every pod**, loading configs from a `vendor-cni-chain/` subdirectory of the +cluster-network config directory — **without touching any CNO-managed file**. The base +directory is derived from the `clusterNetwork` path in `00-multus.conf` +(`/host/run/multus/cni/net.d/10-ovn-kubernetes.conf`), so the on-host chain directory is: + +```text +/run/multus/cni/net.d/vendor-cni-chain/ +``` + +Dropping a standalone `openshell-cni` `.conf` there causes Multus to run our plugin as +an isolated auxiliary chain for all pods. Our plugin already no-ops for pods without the +`openshell.ai/cni: enabled` annotation (unit test `add_passes_through_non_openshell_pod`) +and passes `prevResult` through, so it does not disturb other pods. + +References: + +- Multus configuration reference: +- Multus thick plugin: + +## Approach + +Approach 1 (chosen): add a second **install mode** to the existing CNI DaemonSet, +selected by a new `cni.mode` value. Keep the conflist-append logic unchanged for +k3s/vanilla (preserves the existing `mise run e2e:kubernetes:cni-sidecar` CI path). Add +a `multus-chain` mode for OpenShift. Isolate the OpenShift difference to (a) a script +branch, (b) path defaults, and (c) an SCC template. + +Rejected: a separate OpenShift DaemonSet template (duplicates pod spec / volumes / +cert-refresh loop / RBAC) and a bespoke operator (YAGNI for a chained-plugin drop-in). + +## Design + +### Install modes + +`cni.mode` selects behavior: + +- `conflist` (default, unchanged): find first non-openshell `*.conflist` in + `cni.confDir`, append `openshell-cni` to `.plugins`, back up the original, unpatch on + `preStop`. +- `multus-chain` (new): write a standalone CNI `.conf` + (`{"cniVersion", "name":"openshell-cni", "type":"openshell-cni", "openshell":{...}}`) + into `cni.chainDir`. Never modify a CNO-managed file. `preStop` removes the `.conf` + and the credential files. + +The `openshell-cni` binary is installed into `cni.binDir` in both modes. + +### Paths & credentials + +New/updated values (OpenShift defaults): + +| Value | OpenShift default | Purpose | +|-------|-------------------|---------| +| `cni.mode` | `multus-chain` (overlay); `conflist` (chart default) | Install strategy | +| `cni.binDir` | `/var/lib/cni/bin` | Plugin binary location | +| `cni.chainDir` | `/run/multus/cni/net.d/vendor-cni-chain` | Aux-chain `.conf` (multus-chain only) | +| `cni.stateDir` | `/etc/kubernetes/cni/openshell` | **Persistent** kubeconfig/token/ca.crt/log | + +`cni.chainDir` is tmpfs (`/run`), so credentials cannot live there. The DaemonSet writes +kubeconfig/token/ca.crt to the persistent `cni.stateDir`, and the plugin `.conf` +references those `stateDir` paths. The DaemonSet's periodic reconcile loop (every 30s, +which also refreshes the SA token) additionally **re-asserts the chain `.conf` if +missing**, so it survives a Multus restart; on a full node reboot the DaemonSet +re-writes at startup. + +For back-compat, when `cni.mode == conflist` the `stateDir` default resolves to +`cni.confDir` (current behavior). + +### RBAC & SCC + +- Add `pods: get` for the CNI ServiceAccount, scoped to the sandbox namespace(s). Under + Multus, pod annotations are not passed via `CNI_ARGS`, so the plugin queries the API by + pod name/namespace and requires this verb. (Implicitly required on k3s too; verify the + existing e2e path.) +- New template `templates/cni-scc.yaml`, gated by `cni.openshift.privilegedSCC` + (default `false`): a ClusterRole granting `use` on + `securitycontextconstraints/privileged` (apiGroup `security.openshift.io`) plus a + ClusterRoleBinding to the CNI ServiceAccount. Gating keeps non-OpenShift installs from + referencing OpenShift-only APIs. + +### Chart guardrails + +- The existing `fail` guard (`topology=cni-sidecar` requires `cni.enabled=true`) stays. +- Add validation: `cni.mode` must be `conflist` or `multus-chain`; `multus-chain` + requires `cni.chainDir`. + +## Files + +- `deploy/helm/openshell/templates/cni-daemonset.yaml` — branch install/preStop on + `cni.mode`; add `chainDir`/`stateDir` volumes and mounts; re-assert conf in the loop. +- `deploy/helm/openshell/templates/cni-scc.yaml` — new, gated SCC ClusterRole+binding. +- `deploy/helm/openshell/templates/clusterrole.yaml` (or `role.yaml`) — add `pods: get`. +- `deploy/helm/openshell/values.yaml` — new `cni.mode`, `cni.chainDir`, `cni.stateDir`, + `cni.openshift.privilegedSCC`; updated comments. +- `deploy/helm/openshell/README.md` — regenerated via `mise run helm:docs`. +- `deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml` — new overlay. +- `deploy/helm/openshell/tests/cni_daemonset_test.yaml` — add multus-chain assertions. +- `deploy/helm/openshell/tests/cni_scc_test.yaml` — new SCC unit tests. +- `docs/kubernetes/topology.mdx`, `architecture/compute-runtimes.md` — document the + OpenShift/Multus mode. +- `crates/openshell-cni/*` — likely no functional change; confirm pod-annotation lookup + path and log-file writability under the aux chain. + +## Testing plan (on the OpenShift cluster) + +Prerequisites: + +1. Confirm the supervisor image ships `/openshell-cni` (DaemonSet copies it from the image). +2. Images onto the cluster: expose the internal registry (patch + `configs.imageregistry/cluster`: set `storage` and `defaultRoute: true`), + `oc registry login`, build via `mise run build:docker:gateway` / + `build:docker:supervisor`, tag/push; or push to ghcr. Set the image repo in the overlay. +3. Install the `agent-sandbox` CRDs + controller (`AGENT_SANDBOX_VERSION`) — not present + on the cluster. + +Steps: + +1. Helm install gateway + CNI with the OpenShift overlay. +2. Verify: DaemonSet Running; `openshell-cni` in `/var/lib/cni/bin`; `.conf` in + `/run/multus/cni/net.d/vendor-cni-chain/`; a throwaway pod confirms the aux chain runs + without breaking normal networking. +3. Functional: create a `cni-sidecar` sandbox; inspect the pod netns for the nft/iptables + redirect rules; confirm egress is forced through the sidecar proxy (allowed dest works, + direct bypass blocked). + +Validation gates before deploying: +`mise run pre-commit`, `mise run test`, `mise run helm:test`, `mise run helm:lint`, +`mise run helm:docs`. + +## Known limitations / follow-ups + +- **Fail-open**: the chained-plugin model does not block pod networking if the + `openshell-cni` conf is absent (e.g. brief window after a node reboot before the + DaemonSet re-asserts). This is inherent to `cni-sidecar` generally, not + OpenShift-specific. Fail-closed hardening is out of scope here and left as a follow-up. +- `vendor-cni-chain` availability depends on the Multus thick daemon having + `auxiliaryCNIChainName` set. Present on OpenShift 4.22; the chart should document the + requirement rather than attempt to configure Multus. diff --git a/rfc/cni-sidecar-topology-DRAFT.md b/rfc/cni-sidecar-topology-DRAFT.md new file mode 100644 index 0000000000..3aa6eb5444 --- /dev/null +++ b/rfc/cni-sidecar-topology-DRAFT.md @@ -0,0 +1,530 @@ +--- +authors: + - "@russellb" +state: draft +links: + - https://github.com/NVIDIA/OpenShell/pull/2074 - kubernetes combined topology + - https://github.com/NVIDIA/OpenShell/pull/2076 - kubernetes sidecar topology + - https://github.com/NVIDIA/OpenShell/pull/2078 - original cni-sidecar topology PR from TaylorMutch +--- + +# RFC NNNN - CNI-Sidecar Supervisor Topology (and OpenShift/Multus Enablement) + + + +## Summary + +This RFC proposes `cni-sidecar`, a third Kubernetes supervisor topology for +OpenShell sandbox pods. It keeps the split-supervisor runtime model of the +existing `sidecar` topology — a network-enforcement sidecar plus a +low-privilege process supervisor in the agent container — but moves pod-network +rule installation out of the sandbox pod entirely. A privileged, node-level +OpenShell CNI DaemonSet installs a chained CNI plugin on every node; during CNI +`ADD` the plugin reads OpenShell pod annotations and installs the +bypass-prevention rules in the pod's network namespace before the workload +starts. This removes the per-pod privileged network init container that +`sidecar` topology requires. + +The RFC also proposes the configuration surface needed to run this topology on +managed CNI platforms — specifically OpenShift with Multus and OVN-Kubernetes — +where there is no `.conflist` to append to and where pods run under restrictive +SecurityContextConstraints (SCC). This includes a second CNI install mode +(`multus-chain`), an accompanying set of host-path defaults, and two gated, +minimal SCC grants (one for the CNI DaemonSet, one for the sandbox pod). + +## Motivation + +OpenShell's `combined` topology runs the full supervisor — network, filesystem, +and process controls — inside the agent container, which requires that container +to carry elevated Linux capabilities (`SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, +and others). Many clusters will not admit a workload container with those +capabilities. The `sidecar` topology addresses part of this by moving network +enforcement into a dedicated sidecar and running the agent container as a +low-privilege, network-only process supervisor. But `sidecar` still needs a +**privileged network init container** in every sandbox pod to install the +pod-local nftables rules that fence egress through the sidecar. That init +container needs `NET_ADMIN`/`NET_RAW`, which is exactly the kind of per-pod +privilege stricter clusters — and stricter runtime classes like gVisor — want to +eliminate. + +The problem is worse on OpenShift. There, the sandbox pod init container's +capabilities collide with `restricted-v2`, and even if the sidecar model is +used, the network init container is a per-pod privileged surface that cluster +security teams object to. Operators who want OpenShell's network policy +enforcement on OpenShift currently have no clean path. + +If we leave the design unchanged, OpenShell's network enforcement remains +coupled to either a highly privileged agent container (`combined`) or a +per-pod privileged init container (`sidecar`). Neither fits clusters that push +network-privileged operations to the node/CNI layer, which is where cluster +admins already expect that privilege to live. Node-level CNI installation is a +one-time, admin-scoped grant; per-pod privileged init containers are a +recurring, workload-adjacent grant. Moving the rule installation to the CNI +layer aligns the privilege boundary with how clusters are actually governed. + +## Non-goals + +- **Fail-closed networking.** The chained-plugin model is fail-open: if the + OpenShell CNI conf is absent (e.g., briefly after a node reboot before the + DaemonSet re-asserts it), pod networking is not blocked. This is inherent to + the `cni-sidecar` approach and is left as a follow-up, not addressed here. +- **Configuring Multus.** This RFC consumes the Multus `vendor-cni-chain` + auxiliary chain when it is present; it does not propose configuring or + installing Multus, nor setting `auxiliaryCNIChainName`. +- **Replacing `combined` or `sidecar`.** Both remain; `combined` stays the + default and the only topology that provides the full supervisor contract + (filesystem policy, privilege drop, mount isolation). +- **Non-Kubernetes drivers.** Docker, Podman, and VM drivers are unaffected. + +## Proposal + +### Topology overview + +`cni-sidecar` reuses the `sidecar` runtime split and changes only where the +network rules come from: + +```mermaid +flowchart TB + Sandbox["agents.x-k8s.io Sandbox"] + + subgraph Node["Kubernetes node"] + DaemonSet["OpenShell CNI DaemonSet
(privileged, hostNetwork)"] + CNIPlugin["OpenShell chained CNI plugin"] + Runtime["container runtime
CNI ADD / CHECK"] + + subgraph Pod["Sandbox pod"] + NetNS["pod network namespace"] + subgraph Agent["agent container"] + ProcessSupervisor["process supervisor
network-only, no added caps"] + Workload["Agent workload"] + end + NetworkSidecar["network supervisor sidecar"] + end + end + + Gateway["OpenShell Gateway"] + External["External services"] + + Sandbox --> Pod + DaemonSet -->|"installs plugin binary + conf on node"| CNIPlugin + Runtime -->|"invokes on pod setup"| CNIPlugin + CNIPlugin -->|"reads pod annotations (API lookup)"| Pod + CNIPlugin -->|"installs nftables/iptables rules"| NetNS + ProcessSupervisor --> Workload + Workload -->|"egress redirected on loopback"| NetworkSidecar + NetworkSidecar -->|"gateway forwarding"| Gateway + NetworkSidecar -->|"policy-enforced egress"| External +``` + +Key difference from `sidecar`: there is **no per-pod network init container**. +The DaemonSet's chained plugin installs the loopback-redirect fence during pod +network setup, so the pod itself never needs `NET_ADMIN`/`NET_RAW`. + +### Component privilege model + +The privilege of each component is the central design point. The goal is that +the only privileged surface is the node-level installer (admin-scoped), and the +sandbox pod carries the minimum needed for the enforcement mode in use. + +| Component | Scope | UID | Privilege escalation | Capabilities | Notes | +|---|---|---|---|---|---| +| CNI installer DaemonSet | Node | 0 | true | `privileged: true` | `hostNetwork`, host-path mounts into CNI dirs. One privileged surface, installed once per node by an admin. | +| Agent container (process supervisor, `network-only`) | Pod | `sandbox_uid:sandbox_gid` | false | drops `ALL` | The workload runs here with no added Linux capabilities. | +| Network sidecar — binary-aware mode (default) | Pod | `0:sandbox_gid` | false | drops `ALL`, adds `SYS_PTRACE` + `DAC_READ_SEARCH` | UID 0 is required to inspect the agent's `/proc` across UID boundaries (see below). | +| Network sidecar — endpoint/L7-only mode | Pod | `proxyUid:sandbox_gid` | false | drops `ALL` | Non-root; enforces endpoint/L7 policy without `policy.binaries` matching. | +| Network rule setup | — | N/A | N/A | N/A | Performed by the node CNI plugin; **no pod-local init container**. | + +**Why the binary-aware sidecar must be UID 0.** Kubernetes `SecurityContext` +has no ambient-capability field. A process that starts as non-root loses added +capabilities across `execve`, because ambient caps are what carry capabilities +into a non-root program's effective set. To keep `SYS_PTRACE` and +`DAC_READ_SEARCH` effective — which the sidecar needs to read the agent +process's `/proc` entries across the UID boundary for binary-aware policy — the +sidecar must run as UID 0. This is not an OpenShell preference; it is the only +configuration in which Kubernetes keeps those capabilities effective for +cross-UID `/proc` inspection. The nftables fence exempts UID 0, so operators +must not inject other root containers into these pods. + +Operators who do not need binary matching can set +`processBinaryAwareNetworkPolicy: false`. The sidecar then runs as the non-root +`proxyUid` with no added capabilities and enforces endpoint/L7 policy only. On +OpenShift this variant admits under the built-in `restricted-v2` SCC with no +custom SCC required. + +### CNI install modes + +The CNI DaemonSet installs the `openshell-cni` binary into the host CNI bin +directory in all cases. How it wires the plugin into the node's CNI +configuration is selected by `cni.mode`: + +- **`conflist`** (default; k3s / vanilla): find the first non-OpenShell + `*.conflist` in `cni.confDir`, back it up, and append `openshell-cni` to its + `.plugins` array. `preStop` removes the plugin entry. This is the pre-existing + behavior and is unchanged. +- **`multus-chain`** (OpenShift): write a standalone CNI `.conf` into a Multus + `vendor-cni-chain` subdirectory. Multus runs an isolated auxiliary chain for + every pod from that directory **without modifying any operator-managed file**. + `preStop` removes the `.conf` and credential files. + +The mode split isolates the OpenShift-specific difference to a script branch, +path defaults, and RBAC/SCC — no separate DaemonSet, no operator. + +### Why `multus-chain` on OpenShift + +On OpenShift 4.x (RHCOS, cri-o, OVN-Kubernetes + Multus), the CNI layout defeats +the `conflist` approach: + +- CNI bin dir is `/var/lib/cni/bin` (cri-o `plugin_dirs`); `/opt/cni/bin` does + not exist. +- CNI conf dir is `/etc/kubernetes/cni/net.d/`; the only file is + `00-multus.conf` (type `multus-shim`), a single `.conf` with **no `.plugins` + array** to append to. +- The delegated OVN config is CNO-managed and regenerated, so patching it would + be reverted and risks breaking cluster networking. + +Multus's thick daemon, when configured with `auxiliaryCNIChainName` +(`vendor-cni-chain`, default on OpenShift 4.x), loads additional per-pod CNI +configs from a `vendor-cni-chain/` subdirectory of the cluster-network config +path. Dropping a standalone `openshell-cni.conf` there runs our plugin as an +isolated auxiliary chain for all pods without touching CNO-managed files. The +plugin already no-ops for pods lacking the OpenShell annotation and passes +`prevResult` through, so it does not disturb other pods. + +### Configuration surface + +New/changed chart values. Chart defaults remain non-OpenShift; OpenShift +settings live in the overlay `ci/values-cni-sidecar-openshift.yaml`. + +| Value | Default | OpenShift overlay | Purpose | +|---|---|---|---| +| `supervisor.topology` | `combined` | `cni-sidecar` | Select the topology. | +| `supervisor.sidecar.proxyUid` | `1337` | `1337` | UID for the relaxed endpoint/L7 sidecar; must be non-root and != sandbox UID. | +| `supervisor.sidecar.processBinaryAwareNetworkPolicy` | `true` | `true` | Keep binary-aware policy (sidecar as UID 0). | +| `cni.enabled` | `false` | `true` | Install the node CNI DaemonSet. Required for `cni-sidecar`. | +| `cni.mode` | `conflist` | `multus-chain` | Install strategy. | +| `cni.binDir` | `/opt/cni/bin` | `/var/lib/cni/bin` | Host CNI binary directory. | +| `cni.chainDir` | `""` | `/run/multus/cni/net.d/vendor-cni-chain` | Multus aux-chain dir (multus-chain only). | +| `cni.stateDir` | `""` (falls back to `confDir`) | `/etc/kubernetes/cni/openshell` | **Persistent** credential dir (chainDir is tmpfs). | +| `cni.openshift.privilegedSCC` | `false` | `true` | Grant `privileged` SCC to the CNI ServiceAccount. | +| `sandboxServiceAccount.openshift.binaryAwareSCC` | `false` | `true` | Create + grant the minimal sandbox SCC. | + +Guardrails enforced at template render time: + +- `supervisor.topology=cni-sidecar` requires the CNI singleton: `cni.enabled=true` + to install it here, or `cni.external=true` to use one installed by another + release. +- `cni.mode` must be `conflist` or `multus-chain`. +- `cni.mode=multus-chain` requires `cni.chainDir`. + +Because `cni.chainDir` is tmpfs (under `/run`), credentials cannot live there. +The DaemonSet writes kubeconfig/token/ca.crt to the persistent `cni.stateDir`, +references those paths from the plugin `.conf`, and its periodic reconcile loop +(every 30s, which also refreshes the SA token) re-asserts the chain `.conf` if it +goes missing, so the config survives a Multus restart. A node reboot wipes the +tmpfs chain file; the DaemonSet re-writes it on startup, and a boot-time taint +(see Risks) narrows the pre-reconcile scheduling window. + +### SCC model (OpenShift) + +Two independent, gated, minimal SCC grants — each off by default so non-OpenShift +installs never reference OpenShift-only APIs: + +1. **CNI DaemonSet** (`cni.openshift.privilegedSCC`): a ClusterRole granting + `use` on the built-in `privileged` SCC, bound to the CNI ServiceAccount. The + installer genuinely needs `privileged` for host-path writes into node CNI + directories. + +2. **Sandbox pod** (`sandboxServiceAccount.openshift.binaryAwareSCC`): a + purpose-built minimal SCC, plus a ClusterRole/ClusterRoleBinding granting it + to the sandbox ServiceAccount. It is the `restricted-v2` baseline plus exactly + what binary-aware `/proc` inspection needs — `runAsUser: RunAsAny` (permits + UID 0), `SYS_PTRACE`, `DAC_READ_SEARCH`, and the `image` volume type used to + sideload the supervisor binary. Everything else stays locked down: + `allowPrivilegedContainer: false`, `allowPrivilegeEscalation: false`, no host + namespaces, `requiredDropCapabilities: [ALL]`, `seccompProfiles: + [runtime/default]`. When `processBinaryAwareNetworkPolicy: false`, this SCC is + unnecessary and `restricted-v2` suffices. + +Neither uses `anyuid` (grants UID 0 but adds no capabilities) or the full +`privileged` SCC for the sandbox pod (wildly overbroad). The custom minimal SCC +is the smallest grant that retains functionality. + +### How it works end to end + +1. Operator installs the chart with the OpenShift overlay. Helm renders the + gateway, the CNI DaemonSet (`multus-chain`), the CNI `privileged` SCC grant, + and the minimal sandbox SCC + grant. +2. The DaemonSet, on each node, copies `openshell-cni` into `/var/lib/cni/bin`, + writes credentials into the persistent `stateDir`, and drops + `openshell-cni.conf` into the Multus `vendor-cni-chain` dir. +3. A `cni-sidecar` sandbox is created. The agent-sandbox controller reconciles + the `Sandbox` CR into a pod. OpenShift SCC admission evaluates the sandbox SA + and admits the pod under the minimal SCC (recorded in the + `openshift.io/scc` annotation). +4. During pod network setup, the runtime invokes the Multus aux chain; the + OpenShell plugin reads the pod's OpenShell annotations (via API lookup, since + Multus does not pass annotations through `CNI_ARGS`) and installs the + nftables/iptables loopback-redirect fence in the pod netns. +5. The sidecar starts (UID 0, `SYS_PTRACE` + `DAC_READ_SEARCH`), the process + supervisor starts network-only in the agent container, and workload egress is + fenced through the sidecar to the gateway and policy-enforced destinations. + +### Sample manifests + +Install (OpenShift overlay): + +```shell +helm install openshell deploy/helm/openshell \ + -f deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml +``` + +Overlay values (abridged): + +```yaml +cni: + enabled: true + mode: multus-chain + binDir: /var/lib/cni/bin + chainDir: /run/multus/cni/net.d/vendor-cni-chain + stateDir: /etc/kubernetes/cni/openshell + openshift: + privilegedSCC: true +supervisor: + topology: cni-sidecar +sandboxServiceAccount: + openshift: + binaryAwareSCC: true +``` + +Equivalent gateway TOML for the driver: + +```toml +[openshell.drivers.kubernetes] +topology = "cni-sidecar" + +[openshell.drivers.kubernetes.sidecar] +proxy_uid = 1337 +``` + +Rendered minimal sandbox SCC (abridged): + +```yaml +apiVersion: security.openshift.io/v1 +kind: SecurityContextConstraints +metadata: + name: openshell-sandbox +allowPrivilegedContainer: false +allowPrivilegeEscalation: false +allowHostNetwork: false +allowHostPID: false +allowHostIPC: false +requiredDropCapabilities: + - ALL +allowedCapabilities: + - SYS_PTRACE + - DAC_READ_SEARCH +runAsUser: + type: RunAsAny # permits UID 0 for the binary-aware sidecar +seLinuxContext: + type: MustRunAs +seccompProfiles: + - runtime/default +volumes: + - configMap + - csi + - downwardAPI + - emptyDir + - ephemeral + - image # sideloads the supervisor binary + - persistentVolumeClaim + - projected + - secret +``` + +Standalone Multus aux-chain `.conf` the DaemonSet writes into `chainDir`: + +```json +{ + "cniVersion": "1.0.0", + "name": "openshell-cni", + "type": "openshell-cni", + "openshell": { + "kubeconfig": "/etc/kubernetes/cni/openshell/openshell-cni-kubeconfig", + "sandboxNamespaces": ["openshell"], + "logLevel": "info", + "logFile": "/var/log/openshell-cni.log" + } +} +``` + +## Implementation plan + +The work is incremental and gated so no phase changes non-OpenShift behavior +until explicitly enabled. + +1. **CNI install mode.** Add `cni.mode` (+ `chainDir`, `stateDir`) to the chart; + branch the DaemonSet install/`preStop` script on mode; add `multus-chain` + volumes/mounts and the conf re-assert loop. Keep all existing `conflist` + render output byte-for-byte (existing `cni_daemonset_test.yaml` stays green). +2. **CNI SCC.** Add `templates/cni-scc.yaml` gated on + `cni.openshift.privilegedSCC`, with unit tests. +3. **Sandbox SCC.** Add `templates/sandbox-scc.yaml` + + `openshell.sandboxSccName` helper gated on + `sandboxServiceAccount.openshift.binaryAwareSCC`, with unit tests. +4. **RBAC.** Grant the CNI ServiceAccount `pods: get` scoped to the sandbox + namespace(s), needed because Multus does not pass annotations via `CNI_ARGS`. +5. **Overlay + docs.** Add `ci/values-cni-sidecar-openshift.yaml`; document the + topology, privilege model, and SCCs in `docs/kubernetes/topology.mdx` and + `architecture/compute-runtimes.md`; regenerate the chart README. +6. **Validation.** `mise run pre-commit`, `mise run test`, `mise run helm:test`, + `mise run helm:lint`; then a live cluster verify: DaemonSet Running, plugin + binary and conf present, a throwaway pod confirms the aux chain does not break + normal networking, and a `cni-sidecar` sandbox demonstrates the fence + (allowed dest works, direct bypass blocked) with the sidecar performing a + cross-UID `/proc` read under the minimal SCC. + +Existing users are unaffected: `combined` stays the default, and every OpenShift +behavior is behind a default-off value. + +## Risks + +- **Fail-open window.** If the OpenShell CNI conf is missing (post-reboot before + the DaemonSet re-asserts), pods could network without the fence. Mitigation: a + per-node readiness gate — the installer labels its node `openshell.ai/cni-ready` + and the gateway requires that label via `nodeAffinity`, so sandbox pods cannot + schedule before enforcement is verified. The reconcile loop fences before it + repairs (clears the label the instant enforcement is not verifiably present), + and enforcement lives in the host CNI config so ordinary DaemonSet restarts + never strip it — `preStop` removes it only on a real teardown, fencing first. +- **Node reboot (tmpfs enforcement).** In `multus-chain` mode the chain file + lives under `/run` (tmpfs) and is wiped on reboot, but the persistent + `cni-ready` label survives, so the label gate alone cannot cover a reboot. A + boot-time `NoSchedule` taint (`openshell.ai/cni-not-ready`) **narrows** this (it + does not fully close it): it is applied by operator-provided node config (only + node config runs before the scheduler; a MachineConfig example ships under + `deploy/helm/openshell/examples/`), the CNI DaemonSet tolerates it, and + `openshell-cni node-ready` removes it once enforcement is ready (never re-adding + it, so a transient unready never over-repels). A residual race remains because + the taint is applied after the kubelet starts. `conflist` mode stores the plugin + on persistent disk and is unaffected by reboot. Residual: an ungraceful installer-pod deletion (no + `preStop`) can leave a stale `cni-ready=true` until the pod is rescheduled and + the next reconcile re-checks. Operators diagnose via + `kubectl logs daemonset/openshell-cni` and the tailed plugin log. +- **Cluster-singleton CNI.** The chained plugin and readiness label are + cluster-global host state. Rather than key them to a per-release identity (which + two releases would fight over), the installer is a cluster singleton with a + fixed `openshell` owner and release-independent resource names. Its plugin + enforces pods that carry the OpenShell annotations (set only by a gateway on its + own sandbox pods) **and** whose namespace is in the plugin's `sandboxNamespaces` + allowlist; pods elsewhere pass through without an API lookup, bounding blast + radius. The allowlist is built automatically: each `cni-sidecar` release ships a + Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true`) in its sandbox + namespace, and the installer's reconcile aggregates every marker's namespace + (unioned with the optional static `cni.sandboxNamespaces`) into the config. As a + Helm resource the marker is removed on uninstall / sandboxNamespace change. + Deregistration is drain-gated and monotonic: the reconcile also unions in every + namespace that still has an OpenShell-managed sandbox pod + (`openshell.ai/managed-by=openshell`), so a removed marker never drops + enforcement while sandboxes run; the namespace is pruned only once drained. An + additional `cni.external` release is discovered within one reconcile with no + manual allowlist edit. To close the + discovery-window race, the installer publishes per-node coverage + (`openshell.ai/cni-sandbox-namespaces` annotation) and every gateway runs a + `wait-coverage` init container that blocks serving until every enforcement-ready + node acknowledges the gateway's namespace. Enable the installer in one release + per cluster. Readiness is bound to a `configVersion` (over the aggregated + allowlist and config) so a stale entry is repaired before the node is re-marked + ready, and the installer treats any `openshell-cni` chained entry as its own to + upgrade in place (the conflist patch preserves all other plugins). The + singleton's `pods get` and `configmaps list` RBAC are cluster-scoped so it can + discover marker ConfigMaps and read sandbox pods in any allowlisted namespace. +- **Dependency on Multus aux chain.** `multus-chain` requires the Multus thick + daemon to have `auxiliaryCNIChainName` set. It is default on OpenShift 4.x but + not guaranteed everywhere. Mitigation: document the requirement; the chart does + not attempt to configure Multus. +- **Privileged node DaemonSet.** The installer runs `privileged` with + host-path access to CNI directories — a real node-level surface. Mitigation: + it is admin-scoped, installed once per node, and its SCC grant is gated and + explicit; this is the standard privilege location for CNI components. +- **UID 0 sidecar.** The binary-aware sidecar runs as root in the sandbox pod. + Mitigation: it drops `ALL` and adds only two capabilities; the nftables fence + exempts UID 0 (so no other root container may be injected); operators can opt + down to the non-root endpoint/L7 mode. +- **Experimental runtime coverage.** `cni-sidecar` targets normal runtimes + first; Kata/gVisor must honor CNI-installed pod-network rules for enforcement + to hold. Mitigation: ship as experimental with those as validation targets. +- **SCC drift / cluster policy.** A cluster that further restricts SCC or blocks + custom SCCs could reject the sandbox pod. Mitigation: the SCC is minimal and + documented; the lower-privilege endpoint/L7 mode admits under `restricted-v2`. +- **Non-destructive rule maintenance.** CNI `CHECK` is read-only (verifies the + rules without reinstalling), and `ADD` applies the nft ruleset in a single + atomic transaction (ensure→delete→recreate), so a failed apply/check never + leaves a running pod with the table deleted-but-not-recreated. All Kubernetes + API calls from the plugin/installer use bounded connect/request timeouts so an + API stall cannot wedge CNI `ADD`, the coverage wait, or reconciliation. + +## Alternatives + +### Do nothing + +Operators keep choosing between a highly privileged agent container +(`combined`) or a per-pod privileged network init container (`sidecar`). On +OpenShift, neither is clean, and there is no supported path for CNI-layer +enforcement. Rejected: it leaves a real class of clusters unserved. + +### Separate OpenShift DaemonSet template + +Ship a dedicated OpenShift DaemonSet instead of a mode switch. Rejected: it +duplicates the pod spec, volumes, cert-refresh loop, and RBAC, doubling +maintenance for a difference that is really just install path + SCC. + +### Bespoke OpenShift operator + +Manage installation via a custom operator. Rejected as YAGNI for a +chained-plugin drop-in; it adds a large surface and a new support obligation for +no capability the DaemonSet lacks. + +### Patch the CNO-managed CNI config directly + +Append `openshell-cni` to the OVN/Multus config. Rejected: CNO regenerates and +reverts these files, and a bad edit risks cluster-wide networking outages. The +Multus aux chain exists precisely to avoid touching operator-managed files. + +### Broader SCC (`anyuid` or `privileged`) for the sandbox pod + +Use an existing SCC rather than a custom one. Rejected: `anyuid` permits UID 0 +but adds no capabilities (so binary-aware policy breaks), and `privileged` is +far broader than needed. The minimal custom SCC is the smallest grant that +retains functionality. + +## Prior art + +- **Multus auxiliary CNI chains** (`vendor-cni-chain`): the upstream mechanism + for injecting vendor plugins per-pod without modifying primary CNI config. + Lesson: use the platform's sanctioned extension point instead of patching + managed files. + +- **Chained CNI plugins** (CNI spec `plugins` arrays): the general pattern of + composing a plugin that reads prior results and adds behavior; the OpenShell + plugin no-ops for non-OpenShell pods and passes `prevResult` through. +- **OpenShell `sidecar` topology**: the split-supervisor and control-socket + bootstrap model this topology reuses; `cni-sidecar` changes only rule + installation. See `docs/kubernetes/topology.mdx`. +- **OpenShift SCC model**: `restricted-v2` as the secure baseline and minimal + custom SCCs as the sanctioned way to grant narrowly scoped extra privilege. + +## Open questions + +- Should fail-closed enforcement (deny pod networking when the OpenShell conf is + absent) be promoted from follow-up to a required part of graduating + `cni-sidecar` out of experimental? +- Is `pods: get` for the CNI ServiceAccount acceptable cluster-wide, or should + it be namespace-scoped via Role/RoleBinding per sandbox namespace by default? +- Should the chart detect Multus `auxiliaryCNIChainName` and fail fast (or warn) + when `multus-chain` is selected but the aux chain is not configured? +- What is the graduation criteria (which runtime classes, which CNIs) for + removing the experimental label? diff --git a/tasks/helm.toml b/tasks/helm.toml index 24b6667b1d..db321ace9e 100644 --- a/tasks/helm.toml +++ b/tasks/helm.toml @@ -65,6 +65,11 @@ description = "Run skaffold dev with the Kubernetes supervisor sidecar topology dir = "deploy/helm/openshell" run = "skaffold dev -p sidecar-mtls" +["helm:skaffold:dev:cni-sidecar"] +description = "Run skaffold dev with CNI sidecar topology; requires OpenShell CNI installer permissions on nodes" +dir = "deploy/helm/openshell" +run = "skaffold dev -p cni-sidecar" + ["helm:skaffold:run"] description = "Run skaffold run for deploy/helm/openshell (one-shot deploy)" dir = "deploy/helm/openshell" @@ -80,6 +85,11 @@ description = "Run skaffold run with the Kubernetes supervisor sidecar topology dir = "deploy/helm/openshell" run = "skaffold run -p sidecar-mtls" +["helm:skaffold:run:cni-sidecar"] +description = "Run skaffold run with CNI sidecar topology; requires OpenShell CNI installer permissions on nodes" +dir = "deploy/helm/openshell" +run = "skaffold run -p cni-sidecar" + ["helm:skaffold:delete"] description = "Run skaffold delete for deploy/helm/openshell" dir = "deploy/helm/openshell" @@ -95,6 +105,11 @@ description = "Run skaffold delete for the Kubernetes supervisor sidecar topolog dir = "deploy/helm/openshell" run = "skaffold delete -p sidecar-mtls" +["helm:skaffold:delete:cni-sidecar"] +description = "Run skaffold delete for the Kubernetes CNI sidecar topology" +dir = "deploy/helm/openshell" +run = "skaffold delete -p cni-sidecar" + ["helm:skaffold:diagnose"] description = "Run skaffold diagnose for deploy/helm/openshell" dir = "deploy/helm/openshell" diff --git a/tasks/scripts/docker-build-image.sh b/tasks/scripts/docker-build-image.sh index 8570180f12..b4d79389fa 100755 --- a/tasks/scripts/docker-build-image.sh +++ b/tasks/scripts/docker-build-image.sh @@ -44,7 +44,7 @@ required_prebuilt_binaries() { echo "openshell-gateway" ;; supervisor|supervisor-sideload|supervisor-output) - echo "openshell-sandbox" + echo "openshell-sandbox openshell-cni" ;; esac } diff --git a/tasks/scripts/stage-prebuilt-binaries.sh b/tasks/scripts/stage-prebuilt-binaries.sh index 331d45a5b4..28a860fb95 100755 --- a/tasks/scripts/stage-prebuilt-binaries.sh +++ b/tasks/scripts/stage-prebuilt-binaries.sh @@ -12,7 +12,7 @@ ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" source "${SCRIPT_DIR}/build-env.sh" usage() { - echo "Usage: stage-prebuilt-binaries.sh " >&2 + echo "Usage: stage-prebuilt-binaries.sh " >&2 } normalize_arch() { @@ -91,10 +91,15 @@ components_for_target() { echo "gateway" ;; sandbox|supervisor|supervisor-output) - echo "supervisor" + echo "supervisor cni" + ;; + cni) + echo "cni" ;; all) - echo "gateway supervisor" + # The supervisor image bundles openshell-cni, so the aggregate prebuilt + # target must stage it too (kept in sync with the supervisor target above). + echo "gateway supervisor cni" ;; *) usage @@ -115,6 +120,11 @@ resolve_component() { binary=openshell-sandbox target_libc=musl ;; + cni) + crate=openshell-cni + binary=openshell-cni + target_libc=musl + ;; *) echo "unsupported binary component: $1" >&2 exit 1 diff --git a/tasks/test.toml b/tasks/test.toml index ceb1c30086..3328cd82ad 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -146,6 +146,11 @@ description = "Run Kubernetes e2e with the supervisor sidecar topology overlay" env = { OPENSHELL_E2E_KUBE_EXTRA_VALUES = "deploy/helm/openshell/ci/values-sidecar.yaml" } run = "e2e/rust/e2e-kubernetes.sh" +["e2e:kubernetes:cni-sidecar"] +description = "Run Kubernetes e2e with the CNI sidecar topology overlay; requires OpenShell CNI installer permissions on nodes" +env = { OPENSHELL_E2E_KUBE_EXTRA_VALUES = "deploy/helm/openshell/ci/values-cni-sidecar.yaml" } +run = "e2e/rust/e2e-kubernetes.sh" + ["e2e:kubernetes:db"] description = "Run Kubernetes e2e with all database backend scenarios (SQLite and external PostgreSQL with existingSecret)" env = { OPENSHELL_E2E_KUBE_DB_SCENARIOS = "1" }