From 6936030c6c3b3b26aa297f1532da3c246964b334 Mon Sep 17 00:00:00 2001 From: Taylor Mutch Date: Tue, 30 Jun 2026 15:23:58 -0700 Subject: [PATCH 01/20] feat(kubernetes): add cni-sidecar supervisor topology Add a cni-sidecar supervisor topology that keeps the sidecar runtime model but installs pod-network bypass-prevention rules through a privileged OpenShell chained CNI plugin instead of a pod-local network init container. A new openshell-cni crate provides the chained plugin and node installer. The Kubernetes driver emits openshell.ai/* pod annotations consumed by the plugin during CNI ADD, omits the network init container in this topology, and reuses the sidecar network-only process supervision path. Helm gains a privileged CNI installer DaemonSet gated on cni.enabled and a supervisor.topology=cni-sidecar option. Reconstructed on current main: uses the renamed topology config field and drops the abandoned proxy-pod topology. Signed-off-by: Russell Bryant --- .../skills/debug-openshell-cluster/SKILL.md | 17 + .agents/skills/helm-dev-environment/SKILL.md | 42 +- .github/workflows/docker-build.yml | 3 + Cargo.lock | 14 + architecture/compute-runtimes.md | 8 + crates/openshell-cni/Cargo.toml | 29 + crates/openshell-cni/src/lib.rs | 1088 +++++++++++++++++ crates/openshell-cni/src/main.rs | 9 + crates/openshell-driver-kubernetes/README.md | 13 + .../openshell-driver-kubernetes/src/config.rs | 16 + .../openshell-driver-kubernetes/src/driver.rs | 160 ++- crates/openshell-sandbox/src/lib.rs | 2 +- deploy/docker/Dockerfile.supervisor | 6 +- deploy/helm/openshell/README.md | 17 +- .../helm/openshell/ci/values-cni-sidecar.yaml | 15 + deploy/helm/openshell/skaffold.yaml | 9 + .../openshell/templates/_gateway-workload.tpl | 1 + deploy/helm/openshell/templates/_helpers.tpl | 14 + .../openshell/templates/cni-daemonset.yaml | 181 +++ .../helm/openshell/templates/deployment.yaml | 1 + deploy/helm/openshell/templates/service.yaml | 1 + .../helm/openshell/templates/statefulset.yaml | 1 + .../openshell/tests/cni_daemonset_test.yaml | 71 ++ .../openshell/tests/gateway_config_test.yaml | 16 + deploy/helm/openshell/tests/service_test.yaml | 13 + .../tests/statefulset_client_ca_test.yaml | 7 + deploy/helm/openshell/values.yaml | 39 +- docs/kubernetes/setup.mdx | 5 +- docs/kubernetes/topology.mdx | 124 +- docs/reference/gateway-config.mdx | 7 +- docs/reference/sandbox-compute-drivers.mdx | 9 +- tasks/helm.toml | 15 + tasks/scripts/docker-build-image.sh | 2 +- tasks/scripts/stage-prebuilt-binaries.sh | 12 +- tasks/test.toml | 5 + 35 files changed, 1929 insertions(+), 43 deletions(-) create mode 100644 crates/openshell-cni/Cargo.toml create mode 100644 crates/openshell-cni/src/lib.rs create mode 100644 crates/openshell-cni/src/main.rs create mode 100644 deploy/helm/openshell/ci/values-cni-sidecar.yaml create mode 100644 deploy/helm/openshell/templates/cni-daemonset.yaml create mode 100644 deploy/helm/openshell/tests/cni_daemonset_test.yaml create mode 100644 deploy/helm/openshell/tests/service_test.yaml diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index d07bf1b6c8..bbef67f855 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -395,12 +395,28 @@ The shared state directory should preserve `sandbox_gid` inheritance `@openshell-sidecar-ssh`; the network sidecar verifies its peer PID before bridging gateway relay requests. No `ssh.sock` file should appear in the shared state directory. + +If `topology = "cni-sidecar"` is rendered, the gateway should render +the same process container and long-running network sidecar as sidecar mode, but +there should be no `openshell-network-init` init container in sandbox pods. +Instead, the chart must install the privileged `openshell-cni` DaemonSet and the +sandbox pod should carry `openshell.ai/cni=enabled`, +`openshell.ai/network-enforcement-mode=cni-sidecar`, and +`openshell.ai/proxy-uid=` annotations. The CNI DaemonSet copies +`/openshell-cni` into the host CNI binary directory and patches an existing CNI +`.conflist`; if sandbox pods bypass network enforcement or fail during pod +network setup, inspect the DaemonSet logs, the host CNI config, and whether the +cluster actually invokes chained CNI plugins for the sandbox runtime class. + Inspect all three when sandbox registration or egress enforcement fails: ```bash kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^\[openshell\.drivers\.kubernetes\]|^topology\s*=' kubectl -n get pod -o jsonpath='{range .spec.initContainers[*]}{.name}{" "}{.command}{"\n"}{end}' kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}' +kubectl -n get pod -o jsonpath='{.metadata.annotations}' +kubectl -n openshell get daemonset,pod -l app.kubernetes.io/component=cni +kubectl -n openshell logs daemonset/openshell-cni -c install-cni --tail=200 kubectl -n logs -c openshell-network-init --tail=200 kubectl -n logs -c openshell-supervisor-network --tail=200 kubectl -n logs -c agent --tail=200 @@ -446,6 +462,7 @@ openshell logs | HTTP request returns `middleware_failed` or `middleware_denied` | Selected stage failed or explicitly denied the admitted request | Sandbox OCSF logs; policy-local middleware config; service availability; `on_error` | | Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or configured with a reserved/mismatched name | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs | | Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials | +| CNI-sidecar sandbox pods fail network setup | OpenShell CNI DaemonSet did not patch the node CNI conflist, cannot read pods, or the runtime class does not invoke the chained plugin | `kubectl -n openshell logs daemonset/openshell-cni -c install-cni`, chart `cni.*` values, host CNI config | | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled | | HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://` | | HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https:// --oidc-issuer `, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster | diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index adb40a9575..bc5d795115 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -70,13 +70,20 @@ mise run helm:skaffold:run:sidecar mise run helm:skaffold:run:sidecar-mtls ``` -Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm +**Supervisor CNI-sidecar topology** (build once and leave running): +```bash +mise run helm:skaffold:run:cni-sidecar +``` + +These commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm chart. The sidecar profile renders an `openshell-network-init` init container for nftables setup and an `openshell-supervisor-network` runtime sidecar for proxying. Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and `DAC_READ_SEARCH`; relaxed mode can run it as the configured proxy UID. The sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores -`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install +`server.disableTls=false` inline for Skaffold. The cni-sidecar profile enables +the privileged OpenShell CNI DaemonSet and uses the sidecar runtime model +without the pod-local network init container. The `pkiInitJob` hook (a pre-install Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first install. Envoy Gateway opt-in; see the Optional Add-ons section below. @@ -87,6 +94,30 @@ The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or create the Secret named `openshell-ha-pg` with a `uri` key, then run `mise run helm:skaffold:run` or `mise run helm:skaffold:dev`. +### Kubernetes e2e profiles + +Run the default Kubernetes e2e environment: + +```bash +mise run e2e:kubernetes +``` + +Run the sidecar topology e2e environment: + +```bash +mise run e2e:kubernetes:sidecar +``` + +Run the CNI-sidecar topology e2e environment: + +```bash +mise run e2e:kubernetes:cni-sidecar +``` + +The cni-sidecar e2e task applies `ci/values-cni-sidecar.yaml` through +`OPENSHELL_E2E_KUBE_EXTRA_VALUES` and requires OpenShell CNI installer +permissions on the target cluster's nodes. + ### TLS behaviour `ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run @@ -149,6 +180,12 @@ For a sidecar-profile deployment: mise run helm:skaffold:delete:sidecar ``` +For a cni-sidecar-profile deployment: + +```bash +mise run helm:skaffold:delete:cni-sidecar +``` + ### Delete the cluster entirely ```bash @@ -274,6 +311,7 @@ for dependencies still declared in `Chart.yaml`. | `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) | | `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay | | `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev | +| `deploy/helm/openshell/ci/values-cni-sidecar.yaml` | Supervisor CNI-sidecar topology overlay for Kubernetes e2e/dev; enables the OpenShell CNI DaemonSet | | `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay | | `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev | | `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) | diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index d0d600e58b..1788d60492 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -246,6 +246,9 @@ jobs: fi mkdir -p "$stage" install -m 0755 "$found" "$stage/$binary" + if [[ "${{ inputs.component }}" == "supervisor" ]]; then + PREBUILT_ARCH="${{ matrix.arch }}" tasks/scripts/stage-prebuilt-binaries.sh cni + fi ls -lh "$stage/" - name: Build ${{ inputs.component }} image diff --git a/Cargo.lock b/Cargo.lock index 528309a970..6036c06680 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3862,6 +3862,20 @@ dependencies = [ "url", ] +[[package]] +name = "openshell-cni" +version = "0.0.0" +dependencies = [ + "base64 0.22.1", + "libc", + "miette", + "reqwest 0.12.28", + "serde", + "serde_json", + "serde_yml", + "tempfile", +] + [[package]] name = "openshell-core" version = "0.0.0" diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 55d7e7a29d..5c5a735749 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -241,6 +241,14 @@ already unprivileged. Sidecar pods use a shared process namespace so the network sidecar can resolve workload process and binary identity through `/proc/`. +The cni-sidecar topology keeps the sidecar runtime model and its shared-state +boundary, but removes the privileged `openshell-network-init` init container. +Instead, the privileged OpenShell CNI DaemonSet installs the pod-network +bypass-prevention rules during CNI `ADD` using nftables or iptables. The driver +annotates sandbox pods so the chained CNI plugin can read the proxy UID and +enforcement mode, and both the agent container and long-running network sidecar +stay non-root with no added Linux capabilities. + ## Images The gateway image and Helm chart are built from this repository. Sandbox images diff --git a/crates/openshell-cni/Cargo.toml b/crates/openshell-cni/Cargo.toml new file mode 100644 index 0000000000..835d1ef4c6 --- /dev/null +++ b/crates/openshell-cni/Cargo.toml @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-cni" +description = "OpenShell chained CNI plugin for Kubernetes sidecar network enforcement" +version.workspace = true +edition.workspace = true +license.workspace = true +repository.workspace = true +rust-version.workspace = true + +[dependencies] +base64 = { workspace = true } +miette = { workspace = true } +reqwest = { workspace = true, features = ["blocking"] } +serde = { workspace = true } +serde_json = { workspace = true } +serde_yml = { workspace = true } +tempfile = "3" + +[target.'cfg(target_os = "linux")'.dependencies] +libc = "0.2" + +[dev-dependencies] +tempfile = "3" + +[lints] +workspace = true diff --git a/crates/openshell-cni/src/lib.rs b/crates/openshell-cni/src/lib.rs new file mode 100644 index 0000000000..27c2f8e1b7 --- /dev/null +++ b/crates/openshell-cni/src/lib.rs @@ -0,0 +1,1088 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use base64::Engine; +use miette::{Context, IntoDiagnostic, Result}; +use serde::Deserialize; +use serde_json::Value; +use std::collections::BTreeMap; +use std::io::{Read, Write}; +use std::path::{Path, PathBuf}; +#[cfg(target_os = "linux")] +use std::process::Command; + +const DEFAULT_CNI_VERSION: &str = "1.0.0"; +const SUPPORTED_CNI_VERSIONS: &[&str] = &["0.3.0", "0.3.1", "0.4.0", "1.0.0"]; +const DEFAULT_KUBECONFIG_PATH: &str = "/etc/cni/net.d/openshell-cni-kubeconfig"; +const OPENSHELL_CNI_ENABLED_ANNOTATION: &str = "openshell.ai/cni"; +const OPENSHELL_CNI_PROXY_UID_ANNOTATION: &str = "openshell.ai/proxy-uid"; +const OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION: &str = + "openshell.ai/network-enforcement-mode"; +const CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE: &str = "cni-sidecar"; +#[allow(dead_code)] +const OPENSHELL_TABLE: &str = "openshell_sidecar_bypass"; +#[allow(dead_code)] +const OPENSHELL_IPTABLES_CHAIN: &str = "OPENSHELL_OUTPUT"; +#[cfg(target_os = "linux")] +const NFT_SEARCH_PATHS: &[&str] = &[ + "/usr/sbin/nft", + "/sbin/nft", + "/usr/bin/nft", + "/bin/nft", + "/opt/cni/bin/nft", + "/bin/aux/nft", +]; +#[cfg(target_os = "linux")] +const IPTABLES_SEARCH_PATHS: &[&str] = &[ + "/usr/sbin/iptables", + "/sbin/iptables", + "/usr/bin/iptables", + "/bin/iptables", + "/opt/cni/bin/iptables", + "/bin/aux/iptables", +]; +#[cfg(target_os = "linux")] +const IP6TABLES_SEARCH_PATHS: &[&str] = &[ + "/usr/sbin/ip6tables", + "/sbin/ip6tables", + "/usr/bin/ip6tables", + "/bin/ip6tables", + "/opt/cni/bin/ip6tables", + "/bin/aux/ip6tables", +]; + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +struct CniConfig { + cni_version: Option, + #[serde(default)] + prev_result: Option, + #[serde(default)] + openshell: OpenShellConfig, +} + +#[derive(Debug, Default, Deserialize)] +#[serde(rename_all = "camelCase")] +struct OpenShellConfig { + kubeconfig: Option, + log_file: Option, + #[serde(default)] + sandbox_namespaces: Vec, +} + +#[derive(Debug, Clone)] +struct CniEnv { + command: String, + netns: Option, + args: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct PodRef { + namespace: String, + name: String, +} + +#[derive(Debug, Deserialize)] +struct PodResponse { + metadata: PodMetadata, +} + +#[derive(Debug, Deserialize)] +struct PodMetadata { + #[serde(default)] + annotations: BTreeMap, +} + +#[derive(Debug, Deserialize)] +struct KubeConfig { + #[serde(rename = "current-context")] + current_context: String, + clusters: Vec, + contexts: Vec, + users: Vec, +} + +#[derive(Debug, Deserialize)] +struct NamedCluster { + name: String, + cluster: ClusterConfig, +} + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "kebab-case")] +struct ClusterConfig { + server: String, + certificate_authority_data: Option, + certificate_authority: Option, +} + +#[derive(Debug, Deserialize)] +struct NamedContext { + name: String, + context: ContextConfig, +} + +#[derive(Debug, Deserialize)] +struct ContextConfig { + cluster: String, + user: String, +} + +#[derive(Debug, Deserialize)] +struct NamedUser { + name: String, + user: UserConfig, +} + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "kebab-case")] +struct UserConfig { + token: Option, + token_file: Option, +} + +struct Runtime; + +trait PodReader { + fn pod_annotations(&self, kubeconfig: &Path, pod: &PodRef) -> Result>; +} + +trait RuleInstaller { + fn install(&self, netns: &Path, proxy_uid: u32) -> Result; + fn cleanup(&self, netns: &Path) -> Result<()>; +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct InstallReport { + backend: &'static str, +} + +pub fn run() -> Result<()> { + let mut input = String::new(); + std::io::stdin() + .read_to_string(&mut input) + .into_diagnostic()?; + let env = CniEnv::from_process(); + let runtime = Runtime; + let output = match handle_command(&input, &env, &runtime, &runtime) { + Ok(output) => output, + Err(error) => { + log_cni_error(&input, &env, &error); + return Err(error); + } + }; + if let Some(output) = output { + println!("{}", serde_json::to_string(&output).into_diagnostic()?); + } + Ok(()) +} + +fn log_cni_error(input: &str, env: &CniEnv, error: &miette::Report) { + let Ok(config) = serde_json::from_str::(input) else { + return; + }; + log_cni_info(&config, env, &format!("error={}", one_line_error(error))); +} + +fn log_cni_info(config: &CniConfig, env: &CniEnv, message: &str) { + let Some(log_file) = config.openshell.log_file.as_deref() else { + return; + }; + if log_file.is_empty() { + return; + } + + let pod = env.pod_ref().map_or_else( + || "-".to_string(), + |pod| format!("{}/{}", pod.namespace, pod.name), + ); + let Ok(mut file) = std::fs::OpenOptions::new() + .create(true) + .append(true) + .open(log_file) + else { + return; + }; + let _ = writeln!(file, "command={} pod={} {}", env.command, pod, message); +} + +fn one_line_error(error: &miette::Report) -> String { + format!("{error:?}") + .lines() + .map(str::trim) + .filter(|line| !line.is_empty()) + .collect::>() + .join(" | ") +} + +fn handle_command( + input: &str, + env: &CniEnv, + pod_reader: &impl PodReader, + installer: &impl RuleInstaller, +) -> Result> { + match env.command.as_str() { + "VERSION" => Ok(Some(version_response())), + "DEL" => { + if let Some(netns) = env.netns.as_deref() { + let _ = installer.cleanup(netns); + } + Ok(None) + } + "ADD" => { + let config: CniConfig = serde_json::from_str(input).into_diagnostic()?; + if let Some(workload) = workload_from_config(&config, env, pod_reader)? { + let netns = env.netns.as_deref().ok_or_else(|| { + miette::miette!("CNI_NETNS is required for OpenShell CNI ADD") + })?; + let report = installer.install(netns, workload.proxy_uid)?; + log_cni_info( + &config, + env, + &format!( + "status=installed backend={} proxy_uid={}", + report.backend, workload.proxy_uid + ), + ); + } + Ok(Some(pass_through_result(&config))) + } + "CHECK" => { + let config: CniConfig = serde_json::from_str(input).into_diagnostic()?; + if let Some(workload) = workload_from_config(&config, env, pod_reader)? { + let netns = env.netns.as_deref().ok_or_else(|| { + miette::miette!("CNI_NETNS is required for OpenShell CNI CHECK") + })?; + let report = installer.install(netns, workload.proxy_uid)?; + log_cni_info( + &config, + env, + &format!( + "status=installed backend={} proxy_uid={}", + report.backend, workload.proxy_uid + ), + ); + } + Ok(None) + } + other => Err(miette::miette!("unsupported CNI_COMMAND '{other}'")), + } +} + +#[derive(Debug, Clone, Copy)] +struct WorkloadConfig { + proxy_uid: u32, +} + +fn workload_from_config( + config: &CniConfig, + env: &CniEnv, + pod_reader: &impl PodReader, +) -> Result> { + let Some(pod) = env.pod_ref() else { + return Ok(None); + }; + if !config.openshell.sandbox_namespaces.is_empty() + && !config + .openshell + .sandbox_namespaces + .iter() + .any(|namespace| namespace == &pod.namespace) + { + return Ok(None); + } + let kubeconfig = config + .openshell + .kubeconfig + .as_deref() + .unwrap_or(DEFAULT_KUBECONFIG_PATH); + let annotations = pod_reader.pod_annotations(Path::new(kubeconfig), &pod)?; + if annotations + .get(OPENSHELL_CNI_ENABLED_ANNOTATION) + .map(String::as_str) + != Some("enabled") + { + return Ok(None); + } + if annotations + .get(OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION) + .map(String::as_str) + != Some(CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE) + { + return Ok(None); + } + let proxy_uid = annotations + .get(OPENSHELL_CNI_PROXY_UID_ANNOTATION) + .ok_or_else(|| miette::miette!("OpenShell CNI pod is missing proxy UID annotation"))? + .parse::() + .into_diagnostic() + .wrap_err("invalid OpenShell CNI proxy UID annotation")?; + Ok(Some(WorkloadConfig { proxy_uid })) +} + +fn pass_through_result(config: &CniConfig) -> Value { + config.prev_result.clone().unwrap_or_else(|| { + serde_json::json!({ + "cniVersion": config.cni_version.as_deref().unwrap_or(DEFAULT_CNI_VERSION) + }) + }) +} + +fn version_response() -> Value { + serde_json::json!({ + "cniVersion": DEFAULT_CNI_VERSION, + "supportedVersions": SUPPORTED_CNI_VERSIONS + }) +} + +impl CniEnv { + fn from_process() -> Self { + Self { + command: std::env::var("CNI_COMMAND").unwrap_or_else(|_| "VERSION".to_string()), + netns: std::env::var_os("CNI_NETNS").map(PathBuf::from), + args: std::env::var("CNI_ARGS").ok(), + } + } + + fn pod_ref(&self) -> Option { + let args = self.args.as_deref()?; + let values = parse_cni_args(args); + let namespace = values.get("K8S_POD_NAMESPACE")?.to_string(); + let name = values.get("K8S_POD_NAME")?.to_string(); + Some(PodRef { namespace, name }) + } +} + +fn parse_cni_args(args: &str) -> BTreeMap<&str, &str> { + args.split(';') + .filter_map(|part| part.split_once('=')) + .collect() +} + +impl PodReader for Runtime { + fn pod_annotations(&self, kubeconfig: &Path, pod: &PodRef) -> Result> { + let client = KubeApiClient::from_kubeconfig(kubeconfig)?; + client.pod_annotations(pod) + } +} + +struct KubeApiClient { + server: String, + token: String, + client: reqwest::blocking::Client, +} + +impl KubeApiClient { + fn from_kubeconfig(path: &Path) -> Result { + let contents = std::fs::read_to_string(path) + .into_diagnostic() + .wrap_err_with(|| format!("failed to read kubeconfig {}", path.display()))?; + let kubeconfig: KubeConfig = serde_yml::from_str(&contents) + .into_diagnostic() + .wrap_err("invalid kubeconfig")?; + let context = kubeconfig + .contexts + .iter() + .find(|context| context.name == kubeconfig.current_context) + .ok_or_else(|| miette::miette!("current kubeconfig context not found"))?; + let cluster = kubeconfig + .clusters + .iter() + .find(|cluster| cluster.name == context.context.cluster) + .ok_or_else(|| miette::miette!("current kubeconfig cluster not found"))?; + let user = kubeconfig + .users + .iter() + .find(|user| user.name == context.context.user) + .ok_or_else(|| miette::miette!("current kubeconfig user not found"))?; + let token = match (&user.user.token, &user.user.token_file) { + (Some(token), _) => token.clone(), + (None, Some(token_file)) => std::fs::read_to_string(token_file) + .into_diagnostic() + .wrap_err_with(|| format!("failed to read kubeconfig token file {token_file}"))? + .trim() + .to_string(), + (None, None) => { + return Err(miette::miette!( + "kubeconfig user must contain token or token-file" + )); + } + }; + let mut builder = reqwest::blocking::Client::builder(); + if let Some(ca) = cluster_certificate_authority(path, &cluster.cluster)? { + builder = builder.add_root_certificate(ca); + } + let client = builder.build().into_diagnostic()?; + Ok(Self { + server: cluster.cluster.server.trim_end_matches('/').to_string(), + token, + client, + }) + } + + fn pod_annotations(&self, pod: &PodRef) -> Result> { + let url = format!( + "{}/api/v1/namespaces/{}/pods/{}", + self.server, pod.namespace, pod.name + ); + let response = self + .client + .get(url) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to query Kubernetes API for pod annotations")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while reading pod {}/{}", + response.status(), + pod.namespace, + pod.name + )); + } + let pod = response.json::().into_diagnostic()?; + Ok(pod.metadata.annotations) + } +} + +fn cluster_certificate_authority( + kubeconfig_path: &Path, + cluster: &ClusterConfig, +) -> Result> { + if let Some(data) = cluster.certificate_authority_data.as_deref() { + let pem = base64::engine::general_purpose::STANDARD + .decode(data) + .into_diagnostic() + .wrap_err("invalid kubeconfig certificate-authority-data")?; + return Ok(Some( + reqwest::Certificate::from_pem(&pem).into_diagnostic()?, + )); + } + if let Some(path) = cluster.certificate_authority.as_deref() { + let ca_path = if Path::new(path).is_absolute() { + PathBuf::from(path) + } else { + kubeconfig_path + .parent() + .unwrap_or_else(|| Path::new(".")) + .join(path) + }; + let pem = std::fs::read(ca_path).into_diagnostic()?; + return Ok(Some( + reqwest::Certificate::from_pem(&pem).into_diagnostic()?, + )); + } + Ok(None) +} + +impl RuleInstaller for Runtime { + fn install(&self, netns: &Path, proxy_uid: u32) -> Result { + install_rules(netns, proxy_uid) + } + + fn cleanup(&self, netns: &Path) -> Result<()> { + cleanup_rules(netns) + } +} + +#[allow(dead_code)] +fn generate_sidecar_bypass_ruleset(proxy_uid: u32, log_prefix: Option<&str>) -> String { + let log_tcp = log_prefix + .map(|p| { + format!( + "\n tcp flags syn limit rate 5/second burst 10 packets log prefix \"{p}\" flags skuid" + ) + }) + .unwrap_or_default(); + let log_udp = log_prefix + .map(|p| { + format!( + "\n meta l4proto udp limit rate 5/second burst 10 packets log prefix \"{p}\" flags skuid" + ) + }) + .unwrap_or_default(); + + format!( + r#"table inet {OPENSHELL_TABLE} {{ + chain output {{ + type filter hook output priority 0; policy accept; + + oifname "lo" accept + ct state established,related accept + meta skuid {proxy_uid} accept{log_tcp} + meta nfproto ipv4 meta l4proto tcp reject with icmp type port-unreachable + meta nfproto ipv6 meta l4proto tcp reject with icmpv6 type port-unreachable{log_udp} + meta nfproto ipv4 meta l4proto udp reject with icmp type port-unreachable + meta nfproto ipv6 meta l4proto udp reject with icmpv6 type port-unreachable + }} +}} +"# + ) +} + +#[cfg(target_os = "linux")] +fn install_rules(netns: &Path, proxy_uid: u32) -> Result { + let nft_error = if let Some(nft) = find_nft() { + match install_nft_rules(netns, proxy_uid, &nft) { + Ok(()) => { + return Ok(InstallReport { backend: "nft" }); + } + Err(error) => Some(one_line_error(&error)), + } + } else { + None + }; + + if let Some(iptables) = find_iptables() { + install_iptables_rules(netns, proxy_uid, &iptables).wrap_err("iptables fallback failed")?; + return Ok(InstallReport { + backend: "iptables", + }); + } + + if let Some(nft_error) = nft_error { + return Err(miette::miette!( + "nft rule installation failed and iptables was not found on node: {nft_error}" + )); + } + + Err(miette::miette!( + "neither nft nor iptables was found on node; OpenShell CNI requires a pod-network firewall backend" + )) +} + +#[cfg(target_os = "linux")] +fn install_nft_rules(netns: &Path, proxy_uid: u32, nft: &str) -> Result<()> { + let _ = run_nft_args_in_netns(netns, &nft, &["delete", "table", "inet", OPENSHELL_TABLE]); + let ruleset = generate_sidecar_bypass_ruleset(proxy_uid, Some("openshell:cni-sidecar:")); + run_nft_ruleset_in_netns(netns, &nft, &ruleset) +} + +#[cfg(not(target_os = "linux"))] +fn install_rules(netns: &Path, proxy_uid: u32) -> Result { + let _ = (netns, proxy_uid); + Err(miette::miette!( + "OpenShell CNI rule installation is supported only on Linux nodes" + )) +} + +#[cfg(target_os = "linux")] +fn cleanup_rules(netns: &Path) -> Result<()> { + if let Some(nft) = find_nft() { + let _ = cleanup_nft_rules(netns, &nft); + } + if let Some(iptables) = find_iptables() { + cleanup_iptables_rules(netns, &iptables); + } + Ok(()) +} + +#[cfg(target_os = "linux")] +fn cleanup_nft_rules(netns: &Path, nft: &str) -> Result<()> { + run_nft_args_in_netns(netns, nft, &["delete", "table", "inet", OPENSHELL_TABLE]) +} + +#[cfg(not(target_os = "linux"))] +#[allow(clippy::unnecessary_wraps)] +fn cleanup_rules(netns: &Path) -> Result<()> { + let _ = netns; + Ok(()) +} + +#[cfg(target_os = "linux")] +fn run_nft_ruleset_in_netns(netns: &Path, nft: &str, ruleset: &str) -> Result<()> { + use std::io::Write; + + let mut tmp = tempfile::Builder::new() + .prefix("openshell-cni-") + .suffix(".nft") + .tempfile() + .into_diagnostic()?; + tmp.write_all(ruleset.as_bytes()).into_diagnostic()?; + let ruleset_path = tmp.path().to_string_lossy().to_string(); + run_nft_args_in_netns(netns, nft, &["-f", &ruleset_path]) +} + +#[cfg(target_os = "linux")] +fn run_nft_args_in_netns(netns: &Path, nft: &str, args: &[&str]) -> Result<()> { + run_command_in_netns(netns, nft, args) +} + +#[cfg(target_os = "linux")] +fn run_command_in_netns(netns: &Path, program: &str, args: &[&str]) -> Result<()> { + use std::os::fd::AsRawFd; + use std::os::unix::process::CommandExt; + + let netns = std::fs::File::open(netns).into_diagnostic()?; + let fd = netns.as_raw_fd(); + let output = { + let mut command = Command::new(program); + command.args(args); + // SAFETY: pre_exec runs in the child after fork and before exec. setns + // only affects that child process before it executes the firewall tool. + #[allow(unsafe_code)] + unsafe { + command.pre_exec(move || { + if libc::setns(fd, libc::CLONE_NEWNET) != 0 { + return Err(std::io::Error::last_os_error()); + } + Ok(()) + }); + } + command.output().into_diagnostic()? + }; + + if output.status.success() { + return Ok(()); + } + Err(miette::miette!( + "{} failed in CNI network namespace: {}", + program, + String::from_utf8_lossy(&output.stderr).trim() + )) +} + +#[cfg(target_os = "linux")] +fn find_nft() -> Option { + find_existing_binary(NFT_SEARCH_PATHS) +} + +#[cfg(target_os = "linux")] +fn find_iptables() -> Option { + find_existing_binary(IPTABLES_SEARCH_PATHS).map(|ipv4| IptablesBackend { + ipv4, + ipv6: find_existing_binary(IP6TABLES_SEARCH_PATHS), + }) +} + +#[cfg(target_os = "linux")] +fn find_existing_binary(paths: &[&str]) -> Option { + paths + .iter() + .find(|path| Path::new(path).is_file()) + .map(|path| (*path).to_string()) +} + +#[cfg(target_os = "linux")] +struct IptablesBackend { + ipv4: String, + ipv6: Option, +} + +#[cfg(target_os = "linux")] +fn install_iptables_rules(netns: &Path, proxy_uid: u32, backend: &IptablesBackend) -> Result<()> { + cleanup_iptables_family(netns, &backend.ipv4); + install_iptables_family(netns, &backend.ipv4, proxy_uid, "icmp-port-unreachable")?; + + if let Some(ipv6) = backend.ipv6.as_deref() { + cleanup_iptables_family(netns, ipv6); + install_iptables_family(netns, ipv6, proxy_uid, "icmp6-port-unreachable")?; + } + + Ok(()) +} + +#[cfg(target_os = "linux")] +fn cleanup_iptables_rules(netns: &Path, backend: &IptablesBackend) { + cleanup_iptables_family(netns, &backend.ipv4); + if let Some(ipv6) = backend.ipv6.as_deref() { + cleanup_iptables_family(netns, ipv6); + } +} + +#[cfg(target_os = "linux")] +fn cleanup_iptables_family(netns: &Path, iptables: &str) { + for _ in 0..16 { + if run_command_in_netns( + netns, + iptables, + &[ + "-w", + "-t", + "filter", + "-D", + "OUTPUT", + "-j", + OPENSHELL_IPTABLES_CHAIN, + ], + ) + .is_err() + { + break; + } + } + let _ = run_command_in_netns( + netns, + iptables, + &["-w", "-t", "filter", "-F", OPENSHELL_IPTABLES_CHAIN], + ); + let _ = run_command_in_netns( + netns, + iptables, + &["-w", "-t", "filter", "-X", OPENSHELL_IPTABLES_CHAIN], + ); +} + +#[cfg(target_os = "linux")] +fn install_iptables_family( + netns: &Path, + iptables: &str, + proxy_uid: u32, + reject_with: &str, +) -> Result<()> { + for args in generate_iptables_install_commands(proxy_uid, reject_with) { + let args = args.iter().map(String::as_str).collect::>(); + run_command_in_netns(netns, iptables, &args)?; + } + Ok(()) +} + +#[cfg(target_os = "linux")] +fn generate_iptables_install_commands(proxy_uid: u32, reject_with: &str) -> Vec> { + let uid = proxy_uid.to_string(); + [ + vec!["-w", "-t", "filter", "-N", OPENSHELL_IPTABLES_CHAIN], + vec![ + "-w", + "-t", + "filter", + "-A", + OPENSHELL_IPTABLES_CHAIN, + "-o", + "lo", + "-j", + "RETURN", + ], + vec![ + "-w", + "-t", + "filter", + "-A", + OPENSHELL_IPTABLES_CHAIN, + "-m", + "conntrack", + "--ctstate", + "ESTABLISHED,RELATED", + "-j", + "RETURN", + ], + vec![ + "-w", + "-t", + "filter", + "-A", + OPENSHELL_IPTABLES_CHAIN, + "-m", + "owner", + "--uid-owner", + &uid, + "-j", + "RETURN", + ], + vec![ + "-w", + "-t", + "filter", + "-A", + OPENSHELL_IPTABLES_CHAIN, + "-p", + "tcp", + "-j", + "REJECT", + "--reject-with", + reject_with, + ], + vec![ + "-w", + "-t", + "filter", + "-A", + OPENSHELL_IPTABLES_CHAIN, + "-p", + "udp", + "-j", + "REJECT", + "--reject-with", + reject_with, + ], + vec![ + "-w", + "-t", + "filter", + "-I", + "OUTPUT", + "1", + "-j", + OPENSHELL_IPTABLES_CHAIN, + ], + ] + .into_iter() + .map(|args| args.into_iter().map(str::to_string).collect()) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + struct TestPods { + annotations: BTreeMap, + } + + impl PodReader for TestPods { + fn pod_annotations( + &self, + _kubeconfig: &Path, + _pod: &PodRef, + ) -> Result> { + Ok(self.annotations.clone()) + } + } + + #[derive(Default)] + struct TestInstaller { + installed: std::sync::Mutex>, + cleaned: std::sync::Mutex, + } + + impl RuleInstaller for TestInstaller { + fn install(&self, _netns: &Path, proxy_uid: u32) -> Result { + self.installed.lock().unwrap().push(proxy_uid); + Ok(InstallReport { backend: "test" }) + } + + fn cleanup(&self, _netns: &Path) -> Result<()> { + *self.cleaned.lock().unwrap() += 1; + Ok(()) + } + } + + fn cni_input() -> String { + serde_json::json!({ + "cniVersion": "1.0.0", + "name": "openshell", + "type": "openshell-cni", + "prevResult": { + "cniVersion": "1.0.0", + "interfaces": [] + }, + "openshell": { + "kubeconfig": "/tmp/openshell-kubeconfig", + "sandboxNamespaces": ["openshell"] + } + }) + .to_string() + } + + fn cni_input_with_log_file(log_file: &Path) -> String { + serde_json::json!({ + "cniVersion": "1.0.0", + "name": "openshell", + "type": "openshell-cni", + "openshell": { + "kubeconfig": "/tmp/openshell-kubeconfig", + "sandboxNamespaces": ["openshell"], + "logFile": log_file.to_string_lossy() + } + }) + .to_string() + } + + fn env(command: &str) -> CniEnv { + CniEnv { + command: command.to_string(), + netns: Some(PathBuf::from("/proc/1/ns/net")), + args: Some("K8S_POD_NAMESPACE=openshell;K8S_POD_NAME=sandbox-1".to_string()), + } + } + + fn openshell_annotations() -> BTreeMap { + BTreeMap::from([ + ( + OPENSHELL_CNI_ENABLED_ANNOTATION.to_string(), + "enabled".to_string(), + ), + ( + OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION.to_string(), + CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE.to_string(), + ), + ( + OPENSHELL_CNI_PROXY_UID_ANNOTATION.to_string(), + "1337".to_string(), + ), + ]) + } + + #[test] + fn parses_kubernetes_cni_args() { + let pod = env("ADD").pod_ref().unwrap(); + assert_eq!(pod.namespace, "openshell"); + assert_eq!(pod.name, "sandbox-1"); + } + + #[test] + fn version_returns_supported_versions() { + let pods = TestPods { + annotations: BTreeMap::new(), + }; + let installer = TestInstaller::default(); + let output = handle_command("", &env("VERSION"), &pods, &installer) + .unwrap() + .unwrap(); + assert_eq!(output["cniVersion"], DEFAULT_CNI_VERSION); + assert!( + output["supportedVersions"] + .as_array() + .unwrap() + .contains(&serde_json::json!("0.3.1")) + ); + assert!( + output["supportedVersions"] + .as_array() + .unwrap() + .contains(&serde_json::json!("1.0.0")) + ); + } + + #[test] + fn add_installs_for_annotated_openshell_pod() { + let pods = TestPods { + annotations: openshell_annotations(), + }; + let installer = TestInstaller::default(); + let output = handle_command(&cni_input(), &env("ADD"), &pods, &installer) + .unwrap() + .unwrap(); + assert_eq!(output["interfaces"], serde_json::json!([])); + assert_eq!(*installer.installed.lock().unwrap(), vec![1337]); + } + + #[test] + fn add_passes_through_non_openshell_pod() { + let pods = TestPods { + annotations: BTreeMap::new(), + }; + let installer = TestInstaller::default(); + let output = handle_command(&cni_input(), &env("ADD"), &pods, &installer) + .unwrap() + .unwrap(); + assert_eq!(output["interfaces"], serde_json::json!([])); + assert!(installer.installed.lock().unwrap().is_empty()); + } + + #[test] + fn add_passes_through_unconfigured_namespace_without_api_lookup() { + struct FailingPods; + + impl PodReader for FailingPods { + fn pod_annotations( + &self, + _kubeconfig: &Path, + _pod: &PodRef, + ) -> Result> { + Err(miette::miette!("unexpected API lookup")) + } + } + + let installer = TestInstaller::default(); + let mut env = env("ADD"); + env.args = Some("K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=coredns".to_string()); + let output = handle_command(&cni_input(), &env, &FailingPods, &installer) + .unwrap() + .unwrap(); + assert_eq!(output["interfaces"], serde_json::json!([])); + assert!(installer.installed.lock().unwrap().is_empty()); + } + + #[test] + fn del_cleans_when_netns_available() { + let pods = TestPods { + annotations: openshell_annotations(), + }; + let installer = TestInstaller::default(); + handle_command("", &env("DEL"), &pods, &installer).unwrap(); + assert_eq!(*installer.cleaned.lock().unwrap(), 1); + } + + #[test] + fn sidecar_ruleset_allows_proxy_uid_before_rejects() { + let ruleset = generate_sidecar_bypass_ruleset(1337, Some("openshell:cni-sidecar:")); + let uid_pos = ruleset.find("meta skuid 1337 accept").unwrap(); + let reject_pos = ruleset + .find("meta nfproto ipv4 meta l4proto tcp reject") + .unwrap(); + assert!(uid_pos < reject_pos); + assert!(ruleset.contains("oifname \"lo\" accept")); + assert_eq!( + ruleset + .matches("log prefix \"openshell:cni-sidecar:\"") + .count(), + 2 + ); + } + + #[cfg(target_os = "linux")] + #[test] + fn iptables_fallback_commands_allow_proxy_uid_before_rejects() { + let commands = generate_iptables_install_commands(1337, "icmp-port-unreachable"); + let rendered = commands + .iter() + .map(|command| command.join(" ")) + .collect::>() + .join("\n"); + let uid_pos = rendered.find("--uid-owner 1337 -j RETURN").unwrap(); + let reject_pos = rendered.find("-p tcp -j REJECT").unwrap(); + assert!(uid_pos < reject_pos); + assert!(rendered.contains("-A OPENSHELL_OUTPUT -o lo -j RETURN")); + assert!(rendered.contains("-I OUTPUT 1 -j OPENSHELL_OUTPUT")); + assert!(rendered.contains("--reject-with icmp-port-unreachable")); + } + + #[test] + fn cni_errors_append_to_configured_log_file() { + let dir = tempfile::tempdir().unwrap(); + let log_file = dir.path().join("openshell-cni.log"); + let error = miette::miette!( + "neither nft nor iptables was found on node; OpenShell CNI requires a pod-network firewall backend" + ); + + log_cni_error(&cni_input_with_log_file(&log_file), &env("ADD"), &error); + + let log = std::fs::read_to_string(log_file).unwrap(); + assert!(log.contains("command=ADD")); + assert!(log.contains("pod=openshell/sandbox-1")); + assert!(log.contains("neither nft nor iptables was found")); + } + + #[test] + fn add_success_appends_to_configured_log_file() { + let dir = tempfile::tempdir().unwrap(); + let log_file = dir.path().join("openshell-cni.log"); + let pods = TestPods { + annotations: openshell_annotations(), + }; + let installer = TestInstaller::default(); + + handle_command( + &cni_input_with_log_file(&log_file), + &env("ADD"), + &pods, + &installer, + ) + .unwrap(); + + let log = std::fs::read_to_string(log_file).unwrap(); + assert!(log.contains("command=ADD")); + assert!(log.contains("pod=openshell/sandbox-1")); + assert!(log.contains("status=installed")); + assert!(log.contains("backend=test")); + assert!(log.contains("proxy_uid=1337")); + } + + #[cfg(target_os = "linux")] + #[test] + fn nft_search_path_includes_k3s_aux_path() { + assert!(NFT_SEARCH_PATHS.contains(&"/bin/aux/nft")); + } +} diff --git a/crates/openshell-cni/src/main.rs b/crates/openshell-cni/src/main.rs new file mode 100644 index 0000000000..ab22358268 --- /dev/null +++ b/crates/openshell-cni/src/main.rs @@ -0,0 +1,9 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +fn main() { + if let Err(error) = openshell_cni::run() { + eprintln!("{error:?}"); + std::process::exit(1); + } +} diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 1356e2d932..9b03707d4e 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -98,6 +98,19 @@ abstract socket whose peer PID must match that authenticated supervisor. Both supervisors exit if the control connection closes, coupling their container restart lifecycle before a new authoritative client can be established. +The `cni-sidecar` supervisor topology keeps the sidecar runtime model, but +removes the pod-local network init container. The driver annotates sandbox pods +for the OpenShell chained CNI plugin, and the privileged OpenShell CNI +DaemonSet installs the sidecar bypass-prevention rules during CNI `ADD` before +the workload starts. The agent container and long-running network sidecar remain +non-root with no added Linux capabilities. + +Sidecar and cni-sidecar modes use the pod `fsGroup` to make the projected +service-account token and sandbox client TLS secret group-readable so the +non-root process supervisor can authenticate to the gateway. Treat the agent +container as trusted with respect to those in-pod gateway credentials until a +narrower credential handoff exists. + The driver can request a Kubernetes AppArmor profile through `app_armor_profile`. diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index fb471180a9..4fec186c58 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -65,6 +65,10 @@ pub enum SupervisorTopology { /// Run network supervision in a privileged sidecar and process supervision /// as a low-capability wrapper in the agent container. Sidecar, + /// Run network supervision in a sidecar, with pod-network rules installed + /// by the `OpenShell` chained CNI plugin instead of a privileged init + /// container. + CniSidecar, } impl std::fmt::Display for SupervisorTopology { @@ -72,6 +76,7 @@ impl std::fmt::Display for SupervisorTopology { match self { Self::Combined => f.write_str("combined"), Self::Sidecar => f.write_str("sidecar"), + Self::CniSidecar => f.write_str("cni-sidecar"), } } } @@ -83,6 +88,7 @@ impl FromStr for SupervisorTopology { match s { "combined" => Ok(Self::Combined), "sidecar" => Ok(Self::Sidecar), + "cni-sidecar" => Ok(Self::CniSidecar), other => Err(format!("unknown topology '{other}'")), } } @@ -564,6 +570,16 @@ mod tests { assert_eq!(cfg.topology, SupervisorTopology::Combined); } + #[test] + fn serde_override_topology_cni_sidecar() { + let json = serde_json::json!({ + "topology": "cni-sidecar" + }); + let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); + assert_eq!(cfg.topology, SupervisorTopology::CniSidecar); + assert_eq!(cfg.topology.to_string(), "cni-sidecar"); + } + #[test] fn serde_rejects_sidecar_binary_identity_field() { let json = serde_json::json!({ diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index 2d947b8a28..7cc20fd65b 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -1445,6 +1445,16 @@ const SIDECAR_TLS_VOLUME_NAME: &str = "openshell-supervisor-tls"; const SIDECAR_TLS_MOUNT_PATH: &str = "/etc/openshell-tls/proxy"; const SIDECAR_CLIENT_TLS_MOUNT_PATH: &str = "/etc/openshell-tls/proxy/client"; +const SIDECAR_PROXY_PORT: u16 = 3128; + +const OPENSHELL_CNI_ENABLED_ANNOTATION: &str = "openshell.ai/cni"; +const OPENSHELL_CNI_SANDBOX_ID_ANNOTATION: &str = "openshell.ai/sandbox-id"; +const OPENSHELL_CNI_PROXY_UID_ANNOTATION: &str = "openshell.ai/proxy-uid"; +const OPENSHELL_CNI_PROXY_PORT_ANNOTATION: &str = "openshell.ai/proxy-port"; +const OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION: &str = + "openshell.ai/network-enforcement-mode"; +const CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE: &str = "cni-sidecar"; + /// Build the emptyDir volume that holds the supervisor binary. /// /// The init container writes the binary here; the agent container reads it. @@ -1669,6 +1679,7 @@ fn supervisor_sidecar_env( template_environment: &std::collections::HashMap, spec_environment: &std::collections::HashMap, params: &SandboxPodParams<'_>, + topology: SupervisorTopology, ) -> Vec { let mut env = Vec::new(); apply_required_env( @@ -1701,7 +1712,7 @@ fn supervisor_sidecar_env( upsert_env( &mut env, openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY, - "sidecar", + &topology.to_string(), ); upsert_env( &mut env, @@ -1738,6 +1749,7 @@ fn supervisor_sidecar_container( template_environment: &std::collections::HashMap, spec_environment: &std::collections::HashMap, params: &SandboxPodParams<'_>, + topology: SupervisorTopology, ) -> serde_json::Value { let proxy_uid = effective_sidecar_proxy_uid(params); let capabilities = if params.process_binary_aware_network_policy { @@ -1757,7 +1769,7 @@ fn supervisor_sidecar_container( SUPERVISOR_IMAGE_BINARY_PATH, "--mode=network", ], - "env": supervisor_sidecar_env(template_environment, spec_environment, params), + "env": supervisor_sidecar_env(template_environment, spec_environment, params, topology), "securityContext": { "runAsUser": proxy_uid, "runAsGroup": params.sandbox_gid, @@ -1856,6 +1868,7 @@ fn apply_supervisor_sidecar_topology( template_environment: &std::collections::HashMap, spec_environment: &std::collections::HashMap, params: &SandboxPodParams<'_>, + install_network_init: bool, ) { let Some(spec) = pod_template.get_mut("spec").and_then(|v| v.as_object_mut()) else { return; @@ -1892,12 +1905,14 @@ fn apply_supervisor_sidecar_topology( })); } - let init_containers = spec - .entry("initContainers") - .or_insert_with(|| serde_json::json!([])) - .as_array_mut(); - if let Some(init_containers) = init_containers { - init_containers.push(supervisor_network_init_container(params)); + if install_network_init { + let init_containers = spec + .entry("initContainers") + .or_insert_with(|| serde_json::json!([])) + .as_array_mut(); + if let Some(init_containers) = init_containers { + init_containers.push(supervisor_network_init_container(params)); + } } let Some(containers) = spec.get_mut("containers").and_then(|v| v.as_array_mut()) else { @@ -1979,7 +1994,7 @@ fn apply_supervisor_sidecar_topology( upsert_env( env, openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY, - "sidecar", + ¶ms.topology.to_string(), ); upsert_env( env, @@ -2009,6 +2024,7 @@ fn apply_supervisor_sidecar_topology( template_environment, spec_environment, params, + params.topology, )); } @@ -2235,7 +2251,11 @@ impl Default for SandboxPodParams<'_> { fn validate_sidecar_proxy_identity( params: &SandboxPodParams<'_>, ) -> Result<(), KubernetesDriverError> { - if params.topology == SupervisorTopology::Sidecar && params.proxy_uid == params.sandbox_uid { + if matches!( + params.topology, + SupervisorTopology::Sidecar | SupervisorTopology::CniSidecar + ) && params.proxy_uid == params.sandbox_uid + { return Err(KubernetesDriverError::Precondition(format!( "proxy_uid ({}) must not match sandbox_uid ({}) in sidecar topology", params.proxy_uid, params.sandbox_uid @@ -2402,6 +2422,7 @@ fn sandbox_template_to_k8s_with_validated_config( .iter() .map(|(key, value)| (key.clone(), serde_json::Value::String(value.clone()))) .collect::>(); + let cni_sidecar_topology = params.topology == SupervisorTopology::CniSidecar; if params.provider_spiffe_enabled { pod_labels.insert( LABEL_MANAGED_BY.to_string(), @@ -2435,6 +2456,30 @@ fn sandbox_template_to_k8s_with_validated_config( serde_json::Value::String(params.sandbox_id.to_string()), ); } + if cni_sidecar_topology { + pod_annotations.insert( + OPENSHELL_CNI_ENABLED_ANNOTATION.to_string(), + serde_json::Value::String("enabled".to_string()), + ); + if !params.sandbox_id.is_empty() { + pod_annotations.insert( + OPENSHELL_CNI_SANDBOX_ID_ANNOTATION.to_string(), + serde_json::Value::String(params.sandbox_id.to_string()), + ); + } + pod_annotations.insert( + OPENSHELL_CNI_PROXY_UID_ANNOTATION.to_string(), + serde_json::Value::String(params.proxy_uid.to_string()), + ); + pod_annotations.insert( + OPENSHELL_CNI_PROXY_PORT_ANNOTATION.to_string(), + serde_json::Value::String(SIDECAR_PROXY_PORT.to_string()), + ); + pod_annotations.insert( + OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION.to_string(), + serde_json::Value::String(CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE.to_string()), + ); + } if !pod_annotations.is_empty() { metadata.insert( "annotations".to_string(), @@ -2606,7 +2651,7 @@ fn sandbox_template_to_k8s_with_validated_config( if !params.client_tls_secret_name.is_empty() { let client_tls_default_mode = match params.topology { SupervisorTopology::Combined => 0o400, - SupervisorTopology::Sidecar => 0o440, + SupervisorTopology::Sidecar | SupervisorTopology::CniSidecar => 0o440, }; volumes.push(serde_json::json!({ "name": CLIENT_TLS_VOLUME_NAME, @@ -2632,7 +2677,7 @@ fn sandbox_template_to_k8s_with_validated_config( // supervisor containers run with the sandbox GID and need group-read access. let sa_token_default_mode = match params.topology { SupervisorTopology::Combined => 0o400, - SupervisorTopology::Sidecar => 0o440, + SupervisorTopology::Sidecar | SupervisorTopology::CniSidecar => 0o440, }; volumes.push(serde_json::json!({ "name": SERVICE_ACCOUNT_TOKEN_VOLUME_NAME, @@ -2691,6 +2736,16 @@ fn sandbox_template_to_k8s_with_validated_config( &template.environment, spec_environment, params, + true, + ); + } + SupervisorTopology::CniSidecar => { + apply_supervisor_sidecar_topology( + &mut result, + &template.environment, + spec_environment, + params, + false, ); } } @@ -4546,6 +4601,87 @@ mod tests { assert_eq!(network_init["command"][3], "0"); } + #[test] + fn cni_sidecar_topology_omits_network_init_and_adds_cni_annotations() { + let params = SandboxPodParams { + topology: SupervisorTopology::CniSidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-cni", + proxy_uid: 2200, + sandbox_uid: 1500, + sandbox_gid: 1500, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + false, + ¶ms, + ); + + let annotations = pod_template["metadata"]["annotations"].as_object().unwrap(); + assert_eq!( + annotations[OPENSHELL_CNI_ENABLED_ANNOTATION], + serde_json::json!("enabled") + ); + assert_eq!( + annotations[OPENSHELL_CNI_SANDBOX_ID_ANNOTATION], + serde_json::json!("sb-cni") + ); + assert_eq!( + annotations[OPENSHELL_CNI_PROXY_UID_ANNOTATION], + serde_json::json!("2200") + ); + assert_eq!( + annotations[OPENSHELL_CNI_PROXY_PORT_ANNOTATION], + serde_json::json!(SIDECAR_PROXY_PORT.to_string()) + ); + assert_eq!( + annotations[OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION], + serde_json::json!(CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE) + ); + + let init_containers = pod_template["spec"] + .get("initContainers") + .and_then(|containers| containers.as_array()) + .cloned() + .unwrap_or_default(); + assert!( + !init_containers + .iter() + .any(|container| container["name"] == SUPERVISOR_NETWORK_INIT_CONTAINER_NAME) + ); + + let containers = pod_template["spec"]["containers"].as_array().unwrap(); + assert_eq!(containers.len(), 2); + let agent = containers + .iter() + .find(|container| container["name"] == "agent") + .unwrap(); + assert_eq!( + rendered_env(agent, openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY), + Some("cni-sidecar") + ); + assert_eq!( + rendered_env(agent, openshell_core::sandbox_env::NETWORK_ENFORCEMENT_MODE), + Some("sidecar-nftables") + ); + let sidecar = containers + .iter() + .find(|container| container["name"] == SUPERVISOR_NETWORK_SIDECAR_NAME) + .unwrap(); + assert_eq!( + rendered_env(sidecar, openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY), + Some("cni-sidecar") + ); + } + #[test] fn sidecar_topology_rejects_proxy_uid_matching_sandbox_uid() { let params = SandboxPodParams { diff --git a/crates/openshell-sandbox/src/lib.rs b/crates/openshell-sandbox/src/lib.rs index f9555d9f24..6187c83bec 100644 --- a/crates/openshell-sandbox/src/lib.rs +++ b/crates/openshell-sandbox/src/lib.rs @@ -808,7 +808,7 @@ fn process_enforcement_mode() -> ProcessEnforcementMode { .ok() .as_deref() { - Some("sidecar") => ProcessEnforcementMode::NetworkOnly, + Some("sidecar" | "cni-sidecar") => ProcessEnforcementMode::NetworkOnly, _ => ProcessEnforcementMode::Full, } } diff --git a/deploy/docker/Dockerfile.supervisor b/deploy/docker/Dockerfile.supervisor index c760bbc890..3973993ead 100644 --- a/deploy/docker/Dockerfile.supervisor +++ b/deploy/docker/Dockerfile.supervisor @@ -10,8 +10,9 @@ # path. It also includes nftables so the Kubernetes supervisor sidecar can # install pod-namespace egress enforcement rules. # -# The Rust binary is built natively before this image build runs and staged at: +# The Rust binaries are built natively before this image build runs and staged at: # deploy/docker/.build/prebuilt-binaries//openshell-sandbox +# deploy/docker/.build/prebuilt-binaries//openshell-cni # # Use tasks/scripts/docker-build-image.sh supervisor (or `mise run build:docker:supervisor`) # to stage the binary and build the image in one step. CI builds the binary @@ -23,12 +24,13 @@ FROM alpine:3.22 AS supervisor ARG TARGETARCH -RUN apk add --no-cache nftables iptables iptables-legacy +RUN apk add --no-cache nftables iptables iptables-legacy jq # --chmod=0555 restores execute bits after the actions/upload-artifact + # download-artifact roundtrip strips them. Ownership stays root (0:0) for # Podman image-volume mounts, while world-execute lets the Kubernetes # network sidecar run this binary as the dedicated non-root proxy UID. COPY --chmod=0555 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-sandbox /openshell-sandbox +COPY --chmod=0555 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-cni /openshell-cni ENTRYPOINT ["/openshell-sandbox"] diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index d4310cb9a7..b03688eead 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -148,6 +148,19 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | certManager.enabled | bool | `false` | Create cert-manager Issuer and Certificate resources. When enabled, cert-manager owns TLS and the chart runs a JWT-only certgen hook to create the sandbox JWT signing Secret that cert-manager does not manage. | | certManager.serverDnsNames | list | `["openshell","openshell.openshell.svc","openshell.openshell.svc.cluster.local","localhost","openshell.localhost","*.openshell.localhost","host.docker.internal"]` | DNS SANs on the cert-manager-issued server certificate. | | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | +| cni.affinity | object | `{}` | | +| cni.binDir | string | `"/opt/cni/bin"` | Host CNI binary directory. | +| cni.confDir | string | `"/etc/cni/net.d"` | Host CNI config directory. | +| cni.configFile | string | `""` | Host CNI conflist filename patched by the installer. Empty selects the first non-OpenShell .conflist. | +| cni.enabled | bool | `false` | Install the OpenShell chained CNI plugin with a privileged node DaemonSet. Required when supervisor.topology is "cni-sidecar". | +| cni.image.pullPolicy | string | `""` | CNI installer image pull policy. Empty uses supervisor.image.pullPolicy, then image.pullPolicy. | +| cni.image.repository | string | `""` | CNI installer image repository. Empty uses supervisor.image.repository. | +| cni.image.tag | string | `""` | CNI installer image tag. Empty uses supervisor.image.tag, then chart appVersion. | +| cni.logFile | string | `"/var/log/openshell-cni.log"` | Host log file written by the OpenShell CNI plugin and tailed by the installer DaemonSet. | +| cni.logLevel | string | `"info"` | Log level passed to the OpenShell CNI plugin. | +| cni.nodeSelector | object | `{}` | | +| cni.resources | object | `{}` | | +| cni.tolerations | list | `[]` | | | fullnameOverride | string | `""` | Override the full generated resource name. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | | grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". | @@ -242,9 +255,9 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | supervisor.image.repository | string | `"ghcr.io/nvidia/openshell/supervisor"` | Supervisor image repository. Changing it uses the effective gateway image tag unless tag is also set. | | supervisor.image.tag | string | `""` | Supervisor image tag override. Empty uses the version pinned into the gateway unless repository is changed. | | supervisor.sidecar.processBinaryAwareNetworkPolicy | bool | `true` | Keep process/binary-aware network policy enabled in sidecar topology. When false, the network sidecar runs as proxyUid, drops the extra /proc inspection capabilities, and enforces endpoint/L7 policy without matching policy.binaries. | -| supervisor.sidecar.proxyUid | int | `1337` | UID for relaxed long-running network sidecars in sidecar topology. Strict process/binary-aware sidecars run as UID 0 so Kubernetes grants the required /proc inspection capabilities into the effective set. The network init container installs nftables rules that exempt the effective sidecar UID. | +| supervisor.sidecar.proxyUid | int | `1337` | UID for relaxed long-running network sidecars in sidecar and cni-sidecar topologies. Strict process/binary-aware sidecars run as UID 0 so Kubernetes grants the required /proc inspection capabilities into the effective set. In sidecar topology the network init container installs nftables rules that exempt the effective sidecar UID; in cni-sidecar topology the CNI plugin installs equivalent nftables or iptables rules. | | supervisor.sideloadMethod | string | `""` | How the supervisor binary is delivered into sandbox pods. Empty (default) = auto-detect from cluster version: K8s >= v1.35 -> "image-volume" (ImageVolume enabled by default; GA in v1.36) K8s < v1.35 -> "init-container" (copies via init container + emptyDir) On K8s v1.33-v1.34 with the ImageVolume feature gate manually enabled, set this to "image-volume" explicitly. | -| supervisor.topology | string | `"combined"` | Supervisor pod topology for Kubernetes sandboxes. "combined" runs the current single supervisor container in the agent pod. "sidecar" runs network enforcement in a dedicated sidecar and the process supervisor as a low-capability wrapper in the agent container. | +| supervisor.topology | string | `"combined"` | Supervisor pod topology for Kubernetes sandboxes. "combined" runs the current single supervisor container in the agent pod. "sidecar" runs network enforcement in a dedicated sidecar and the process supervisor as a low-capability wrapper in the agent container. "cni-sidecar" keeps the sidecar runtime model but installs pod-network rules through the OpenShell CNI plugin. | | tolerations | list | `[]` | Tolerations for the gateway pod. | | workload.allowMultiReplicaStatefulSet | bool | `false` | Allow replicaCount > 1 while rendering a StatefulSet. Prefer workload.kind=deployment for external database-backed multi-replica gateways; this override exists for operators who explicitly require StatefulSet identity or storage semantics. | | workload.kind | string | `"statefulset"` | Gateway workload controller kind. Use `statefulset` for the default SQLite database, or `deployment` when server.externalDbSecret points at an external database. | diff --git a/deploy/helm/openshell/ci/values-cni-sidecar.yaml b/deploy/helm/openshell/ci/values-cni-sidecar.yaml new file mode 100644 index 0000000000..fb649fa6d7 --- /dev/null +++ b/deploy/helm/openshell/ci/values-cni-sidecar.yaml @@ -0,0 +1,15 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# CI/dev overlay for exercising the Kubernetes CNI sidecar topology. +# This topology requires the OpenShell chained CNI plugin to be installed on +# every node before sandbox pods are created. + +cni: + enabled: true + # k3s configures containerd with k3s-specific CNI paths. + binDir: /bin + confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d + +supervisor: + topology: cni-sidecar diff --git a/deploy/helm/openshell/skaffold.yaml b/deploy/helm/openshell/skaffold.yaml index 119adf086b..e9125a27ae 100644 --- a/deploy/helm/openshell/skaffold.yaml +++ b/deploy/helm/openshell/skaffold.yaml @@ -121,6 +121,10 @@ deploy: #- ci/values-spire.yaml # To exercise the Kubernetes supervisor sidecar topology: #- ci/values-sidecar.yaml + # To exercise CNI sidecar topology, use the cni-sidecar Skaffold + # profile against a cluster where the OpenShell CNI DaemonSet can + # patch the node CNI conflist. + #- ci/values-cni-sidecar.yaml # To test multi-replica external PostgreSQL behavior: #- ci/values-high-availability.yaml setValueTemplates: @@ -143,3 +147,8 @@ profiles: path: /deploy/helm/releases/0/setValues value: server.disableTls: "false" + - name: cni-sidecar + patches: + - op: add + path: /deploy/helm/releases/0/valuesFiles/- + value: ci/values-cni-sidecar.yaml diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl index 5931047e5f..e9783a88ce 100644 --- a/deploy/helm/openshell/templates/_gateway-workload.tpl +++ b/deploy/helm/openshell/templates/_gateway-workload.tpl @@ -17,6 +17,7 @@ metadata: {{- end }} labels: {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: gateway {{- with .Values.podLabels }} {{- toYaml . | nindent 4 }} {{- end }} diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 1b4598088f..b5619585dd 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -103,6 +103,20 @@ a repository-only override uses the effective gateway image tag. {{- printf "%s:%s" $repository $tag }} {{- end }} +{{/* +CNI installer image reference. Defaults to the supervisor image because the +supervisor image carries both openshell-sandbox and openshell-cni. +*/}} +{{- define "openshell.cniImage" -}} +{{- $repository := .Values.cni.image.repository | default .Values.supervisor.image.repository -}} +{{- $tag := .Values.cni.image.tag | default .Values.supervisor.image.tag | default .Chart.AppVersion -}} +{{- printf "%s:%s" $repository $tag }} +{{- end }} + +{{- define "openshell.cniImagePullPolicy" -}} +{{- .Values.cni.image.pullPolicy | default .Values.supervisor.image.pullPolicy | default .Values.image.pullPolicy -}} +{{- end }} + {{/* Namespaced Issuer (selfSigned) for cert-manager CA bootstrap. */}} diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml new file mode 100644 index 0000000000..b19596db42 --- /dev/null +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -0,0 +1,181 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if and (eq (.Values.supervisor.topology | default "combined") "cni-sidecar") (not .Values.cni.enabled) -}} +{{- fail "supervisor.topology=cni-sidecar requires cni.enabled=true so the OpenShell chained CNI plugin is installed on every node" -}} +{{- end }} +{{- if .Values.cni.enabled }} +{{- $sandboxNamespace := .Values.server.sandboxNamespace | default .Release.Namespace }} +{{- $cniLogFile := .Values.cni.logFile | default "/var/log/openshell-cni.log" }} +{{- $cniLogDir := dir $cniLogFile }} +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: {{ include "openshell.fullname" . }}-cni + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni +spec: + selector: + matchLabels: + {{- include "openshell.selectorLabels" . | nindent 6 }} + app.kubernetes.io/component: cni + template: + metadata: + labels: + {{- include "openshell.selectorLabels" . | nindent 8 }} + app.kubernetes.io/component: cni + spec: + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + serviceAccountName: {{ include "openshell.serviceAccountName" . }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: install-cni + image: {{ include "openshell.cniImage" . | quote }} + imagePullPolicy: {{ include "openshell.cniImagePullPolicy" . | quote }} + command: + - sh + - -ec + - | + bin_dir="/host{{ .Values.cni.binDir }}" + conf_dir="/host{{ .Values.cni.confDir }}" + host_conf_dir="{{ .Values.cni.confDir }}" + log_file="{{ $cniLogFile }}" + host_log_file="/host${log_file}" + kubeconfig="${conf_dir}/openshell-cni-kubeconfig" + token_file="${conf_dir}/openshell-cni-token" + ca_file="${conf_dir}/openshell-cni-ca.crt" + plugin_config="${conf_dir}/openshell-cni-plugin.json" + mkdir -p "${bin_dir}" "${conf_dir}" + touch "${host_log_file}" + chmod 0644 "${host_log_file}" + install -m 0755 /openshell-cni "${bin_dir}/openshell-cni" + cp /var/run/secrets/kubernetes.io/serviceaccount/ca.crt "${ca_file}" + cp /var/run/secrets/kubernetes.io/serviceaccount/token "${token_file}" + chmod 0600 "${token_file}" + cat > "${kubeconfig}" < "${plugin_config}" <&2 + exit 1 + fi + if [ ! -f "${target}.openshell-backup" ]; then + cp "${target}" "${target}.openshell-backup" + fi + tmp="$(mktemp "${conf_dir}/openshell-cni.XXXXXX")" + jq --slurpfile openshell "${plugin_config}" ' + .plugins = ((.plugins // []) | map(select(.type != "openshell-cni")) + [$openshell[0]]) + ' "${target}" > "${tmp}" + mv "${tmp}" "${target}" + echo "OpenShell CNI installed; plugin log ${log_file}; firewall backend selected during CNI ADD" + tail -n 0 -F "${host_log_file}" & + tail_pid="$!" + trap 'kill "${tail_pid}" 2>/dev/null || true; exit 0' INT TERM + trap 'kill "${tail_pid}" 2>/dev/null || true' EXIT + while true; do + cp /var/run/secrets/kubernetes.io/serviceaccount/token "${token_file}" + chmod 0600 "${token_file}" + sleep 300 + done + securityContext: + privileged: true + allowPrivilegeEscalation: true + lifecycle: + preStop: + exec: + command: + - sh + - -ec + - | + conf_dir="/host{{ .Values.cni.confDir }}" + for target in "${conf_dir}"/*.conflist; do + [ -f "${target}" ] || continue + tmp="$(mktemp "${conf_dir}/openshell-cni-cleanup.XXXXXX")" + jq ' + if .plugins then + .plugins = (.plugins | map(select(.type != "openshell-cni"))) + else + . + end + ' "${target}" > "${tmp}" + mv "${tmp}" "${target}" + done + rm -f \ + "${conf_dir}/openshell-cni-ca.crt" \ + "${conf_dir}/openshell-cni-kubeconfig" \ + "${conf_dir}/openshell-cni-plugin.json" \ + "${conf_dir}/openshell-cni-token" + volumeMounts: + - name: cni-bin + mountPath: /host{{ .Values.cni.binDir }} + - name: cni-conf + mountPath: /host{{ .Values.cni.confDir }} + - name: cni-log + mountPath: /host{{ $cniLogDir }} + {{- with .Values.cni.resources }} + resources: + {{- toYaml . | nindent 12 }} + {{- end }} + volumes: + - name: cni-bin + hostPath: + path: {{ .Values.cni.binDir | quote }} + type: DirectoryOrCreate + - name: cni-conf + hostPath: + path: {{ .Values.cni.confDir | quote }} + type: Directory + - name: cni-log + hostPath: + path: {{ $cniLogDir | quote }} + type: DirectoryOrCreate + {{- with .Values.cni.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.cni.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.cni.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} +{{- end }} diff --git a/deploy/helm/openshell/templates/deployment.yaml b/deploy/helm/openshell/templates/deployment.yaml index e937979370..de9ae012df 100644 --- a/deploy/helm/openshell/templates/deployment.yaml +++ b/deploy/helm/openshell/templates/deployment.yaml @@ -13,6 +13,7 @@ spec: selector: matchLabels: {{- include "openshell.selectorLabels" . | nindent 6 }} + app.kubernetes.io/component: gateway template: {{- include "openshell.gatewayPodTemplate" . | nindent 4 }} {{- end }} diff --git a/deploy/helm/openshell/templates/service.yaml b/deploy/helm/openshell/templates/service.yaml index ebad42eab1..921eec4ee1 100644 --- a/deploy/helm/openshell/templates/service.yaml +++ b/deploy/helm/openshell/templates/service.yaml @@ -26,3 +26,4 @@ spec: {{- end }} selector: {{- include "openshell.selectorLabels" . | nindent 4 }} + app.kubernetes.io/component: gateway diff --git a/deploy/helm/openshell/templates/statefulset.yaml b/deploy/helm/openshell/templates/statefulset.yaml index 30571f80ba..205f9a8daa 100644 --- a/deploy/helm/openshell/templates/statefulset.yaml +++ b/deploy/helm/openshell/templates/statefulset.yaml @@ -14,6 +14,7 @@ spec: selector: matchLabels: {{- include "openshell.selectorLabels" . | nindent 6 }} + app.kubernetes.io/component: gateway template: {{- include "openshell.gatewayPodTemplate" . | nindent 4 }} volumeClaimTemplates: diff --git a/deploy/helm/openshell/tests/cni_daemonset_test.yaml b/deploy/helm/openshell/tests/cni_daemonset_test.yaml new file mode 100644 index 0000000000..d54f05ed39 --- /dev/null +++ b/deploy/helm/openshell/tests/cni_daemonset_test.yaml @@ -0,0 +1,71 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: cni daemonset +templates: + - templates/cni-daemonset.yaml +tests: + - it: does not render by default + template: templates/cni-daemonset.yaml + asserts: + - hasDocuments: + count: 0 + + - it: renders privileged cni installer daemonset when enabled + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + asserts: + - isKind: + of: DaemonSet + - equal: + path: spec.template.spec.hostNetwork + value: true + - equal: + path: spec.template.spec.containers[0].securityContext.privileged + value: true + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-bin + mountPath: /host/opt/cni/bin + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-conf + mountPath: /host/etc/cni/net.d + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-log + mountPath: /host/var/log + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '"sandboxNamespaces": \["NAMESPACE"\]' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '"logFile": "\$\{log_file\}"' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: "firewall backend selected during CNI ADD" + - matchRegex: + path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] + pattern: 'select\(\.type != "openshell-cni"\)' + + - it: scopes cni annotation lookup to explicit sandbox namespace + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + server.sandboxNamespace: agents + asserts: + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '"sandboxNamespaces": \["agents"\]' + + - it: fails cni-sidecar topology when cni installer is disabled + template: templates/cni-daemonset.yaml + set: + supervisor.topology: cni-sidecar + asserts: + - failedTemplate: + errorMessage: supervisor.topology=cni-sidecar requires cni.enabled=true so the OpenShell chained CNI plugin is installed on every node diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index 90e4f9cef0..563942fe16 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -25,6 +25,9 @@ tests: - equal: path: kind value: StatefulSet + - equal: + path: spec.selector.matchLabels["app.kubernetes.io/component"] + value: gateway - it: treats a null workload map as the default StatefulSet template: templates/statefulset.yaml @@ -150,6 +153,16 @@ tests: path: data["gateway.toml"] pattern: '(?ms)\[openshell\.drivers\.kubernetes\.sidecar\].*?proxy_uid\s*=\s*2200' + - it: renders cni-sidecar supervisor topology under [openshell.drivers.kubernetes] + template: templates/gateway-config.yaml + set: + cni.enabled: true + supervisor.topology: cni-sidecar + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?topology\s*=\s*"cni-sidecar"' + - it: renders process binary aware network policy under [openshell.drivers.kubernetes.sidecar] template: templates/gateway-config.yaml set: @@ -373,6 +386,9 @@ tests: - equal: path: spec.replicas value: 2 + - equal: + path: spec.selector.matchLabels["app.kubernetes.io/component"] + value: gateway - equal: path: spec.template.spec.containers[0].name value: openshell-gateway diff --git a/deploy/helm/openshell/tests/service_test.yaml b/deploy/helm/openshell/tests/service_test.yaml new file mode 100644 index 0000000000..16eca90cfd --- /dev/null +++ b/deploy/helm/openshell/tests/service_test.yaml @@ -0,0 +1,13 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: gateway service +templates: + - templates/service.yaml +tests: + - it: selects only gateway component pods + template: templates/service.yaml + asserts: + - equal: + path: spec.selector["app.kubernetes.io/component"] + value: gateway diff --git a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml index a7b02310cf..eaa51affc1 100644 --- a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml +++ b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml @@ -10,6 +10,13 @@ release: namespace: my-namespace tests: + - it: labels gateway pods as gateway component + template: templates/statefulset.yaml + asserts: + - equal: + path: spec.template.metadata.labels["app.kubernetes.io/component"] + value: gateway + - it: mounts the server TLS secret ca.crt as client CA for built-in PKI template: templates/statefulset.yaml set: diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 0525ed475d..c8535fb5ee 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -48,13 +48,16 @@ supervisor: # "combined" runs the current single supervisor container in the agent pod. # "sidecar" runs network enforcement in a dedicated sidecar and the process # supervisor as a low-capability wrapper in the agent container. + # "cni-sidecar" keeps the sidecar runtime model but installs pod-network rules + # through the OpenShell CNI plugin. topology: "combined" sidecar: - # -- UID for relaxed long-running network sidecars in sidecar topology. - # Strict process/binary-aware sidecars run as UID 0 so Kubernetes grants - # the required /proc inspection capabilities into the effective set. The - # network init container installs nftables rules that exempt the effective - # sidecar UID. + # -- UID for relaxed long-running network sidecars in sidecar and cni-sidecar + # topologies. Strict process/binary-aware sidecars run as UID 0 so Kubernetes + # grants the required /proc inspection capabilities into the effective set. In + # sidecar topology the network init container installs nftables rules that + # exempt the effective sidecar UID; in cni-sidecar topology the CNI plugin + # installs equivalent nftables or iptables rules. proxyUid: 1337 # -- Keep process/binary-aware network policy enabled in sidecar topology. # When false, the network sidecar runs as proxyUid, drops the extra /proc @@ -62,6 +65,32 @@ supervisor: # policy.binaries. processBinaryAwareNetworkPolicy: true +cni: + # -- Install the OpenShell chained CNI plugin with a privileged node DaemonSet. + # Required when supervisor.topology is "cni-sidecar". + enabled: false + image: + # -- CNI installer image repository. Empty uses supervisor.image.repository. + repository: "" + # -- CNI installer image pull policy. Empty uses supervisor.image.pullPolicy, then image.pullPolicy. + pullPolicy: "" + # -- CNI installer image tag. Empty uses supervisor.image.tag, then chart appVersion. + tag: "" + # -- Host CNI binary directory. + binDir: /opt/cni/bin + # -- Host CNI config directory. + confDir: /etc/cni/net.d + # -- Log level passed to the OpenShell CNI plugin. + logLevel: info + # -- Host log file written by the OpenShell CNI plugin and tailed by the installer DaemonSet. + logFile: /var/log/openshell-cni.log + # -- Host CNI conflist filename patched by the installer. Empty selects the first non-OpenShell .conflist. + configFile: "" + resources: {} + nodeSelector: {} + tolerations: [] + affinity: {} + # -- Image pull secrets attached to gateway and helper pods. imagePullSecrets: [] # -- Override the chart name used in generated resource names. diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index c2fca827f1..e09e759241 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -162,6 +162,9 @@ The most commonly changed values are: | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect based on cluster version: clusters running Kubernetes 1.35 or later use `image-volume` (ImageVolume GA in 1.36); older clusters use `init-container`. Set explicitly to `image-volume` on Kubernetes 1.33 or 1.34 with the ImageVolume feature gate enabled, or to `init-container` to force the legacy path on any version. | | `supervisor.topology` | Sandbox pod topology. Refer to [Topology](/kubernetes/topology). | | `supervisor.sidecar.proxyUid` | Non-root UID used when sidecar process/binary-aware network policy is disabled. The default binary-aware sidecar runs as UID 0 instead. The configured UID must not match the sandbox UID. | +| `cni.enabled` | Install the privileged OpenShell CNI DaemonSet. Required when `supervisor.topology=cni-sidecar`. | +| `cni.configFile` | Existing host CNI conflist to patch. Leave empty to patch the first non-OpenShell `.conflist` in `cni.confDir`. | +| `cni.logFile` | Host log file written by the OpenShell CNI plugin and tailed by the installer DaemonSet. Defaults to `/var/log/openshell-cni.log`. | Use a values file for repeatable deployments: @@ -245,7 +248,7 @@ The gateway exposes `/healthz` for process liveness and `/readyz` for dependency ## Next Steps -- To choose between combined and sidecar sandbox pods, refer to [Topology](/kubernetes/topology). +- To choose between combined, sidecar, and cni-sidecar sandbox topology, refer to [Topology](/kubernetes/topology). - To enable automatic certificate rotation with cert-manager, refer to [Managing Certificates](/kubernetes/managing-certificates). - To expose the gateway externally without port-forwarding, refer to [Ingress](/kubernetes/ingress). - To configure OIDC or reverse-proxy authentication, refer to [Access Control](/kubernetes/access-control). diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index 5bbb18e1ef..3d8024e736 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -3,14 +3,15 @@ # SPDX-License-Identifier: Apache-2.0 title: "Kubernetes Sandbox Topology" sidebar-title: "Topology" -description: "Choose between combined and sidecar supervisor topology for Kubernetes sandbox pods." +description: "Choose between combined, sidecar, and cni-sidecar supervisor topology for Kubernetes sandbox pods." keywords: "Generative AI, Cybersecurity, Kubernetes, Sandboxing, Sidecar, Network Policy, RuntimeClass" position: 2 --- -Kubernetes sandbox pods can run the OpenShell supervisor in `combined` or -`sidecar` topology. Choose the topology based on which controls you need inside -the pod and how much privilege your cluster allows on the agent container. +Kubernetes sandbox pods can run the OpenShell supervisor in `combined`, +`sidecar`, or `cni-sidecar` topology. Choose the topology based on which +controls you need inside the pod, how much privilege your cluster allows on the +agent container, and whether the cluster can install node-level CNI enforcement. ## Choose a Topology @@ -22,6 +23,7 @@ lower-privilege agent container. |---|---|---| | `combined` | You need OpenShell network, filesystem, and process controls in the sandbox workload. | The agent container carries the Linux capabilities the supervisor needs. | | `sidecar` | You need the agent container to run as non-root without added Linux capabilities, and network policy is the primary control. | Privilege-dropping and supervisor mount isolation do not run in the agent container. | +| `cni-sidecar` | You need sidecar-mode behavior without a privileged sandbox-pod init container, and a cluster admin can install the OpenShell CNI plugin on every node. | Requires privileged node-level CNI installation; the agent process supervisor is network-only. | ## Privilege Model @@ -33,6 +35,8 @@ The long-running container permissions differ by topology: | `sidecar` | Agent container, process-only supervisor (`network-only`) | `sandbox_uid:sandbox_gid` | `false` | Drops `ALL` | Agent and workload run without added Linux capabilities. | | `sidecar` | Network supervisor sidecar, binary-aware mode (default) | `0:sandbox_gid` | `false` | Drops `ALL`; adds `SYS_PTRACE` and `DAC_READ_SEARCH` | Root sidecar inspects cross-UID workload `/proc` entries. The nftables fence exempts UID 0, so do not inject other root containers into these pods. | | `sidecar` | Network supervisor sidecar, endpoint/L7-only mode | `proxyUid:sandbox_gid` | `false` | Drops `ALL` | Non-root sidecar enforces endpoint and L7 policy without matching `policy.binaries`. | +| `cni-sidecar` | Agent container, process-only supervisor (`network-only`) | `sandbox_uid:sandbox_gid` | `false` | Drops `ALL` | Agent and workload run without added Linux capabilities. | +| `cni-sidecar` | Network supervisor sidecar | Same as `sidecar` mode | `false` | Same as `sidecar` mode | Node-level CNI installs the pod network rules instead of a pod-local init container. | Short-lived setup containers still have the permissions needed to prepare the pod: @@ -41,6 +45,7 @@ pod: |---|---|---|---|---|---| | `combined` | Supervisor install init container | `0` | Not set | Not set | Copies the supervisor binary into the agent container volume. | | `sidecar` | Network init container | `0` | `false` | Drops `ALL`; adds `NET_ADMIN`, `NET_RAW`, `CHOWN`, and `FOWNER` | Installs pod-local nftables rules and prepares shared sidecar state. | +| `cni-sidecar` | None for network rules | N/A | N/A | N/A | The OpenShell CNI DaemonSet installs rules during pod network setup. | ## Combined Topology @@ -158,12 +163,73 @@ Sidecar pods use `shareProcessNamespace: true` so the network sidecar can resolve workload process and binary identity through `/proc/`. +## CNI Sidecar Topology + +CNI sidecar topology keeps the sidecar runtime model but moves network rule +installation out of the sandbox pod. A privileged OpenShell DaemonSet installs a +chained CNI plugin on each node. During CNI `ADD`, the plugin reads OpenShell +pod annotations and installs the sidecar bypass-prevention rules in the pod +network namespace before the workload starts. + +```mermaid +flowchart TB + Sandbox["agents.x-k8s.io Sandbox"] + + subgraph Node["Kubernetes node"] + DaemonSet["OpenShell CNI DaemonSet"] + CNIPlugin["OpenShell chained CNI plugin"] + Runtime["container runtime
CNI ADD / CHECK"] + + subgraph Pod["Sandbox pod"] + NetNS["pod network namespace"] + + subgraph Agent["agent container"] + ProcessSupervisor["process supervisor
network-only"] + Workload["Agent workload"] + end + + NetworkSidecar["network supervisor sidecar
proxyUid"] + end + end + + Gateway["OpenShell Gateway"] + External["External services"] + + Sandbox --> Pod + DaemonSet -->|"installs binary + patches conflist"| CNIPlugin + Runtime -->|"invokes"| CNIPlugin + CNIPlugin -->|"reads pod annotations"| Pod + CNIPlugin -->|"installs nftables or iptables rules"| NetNS + ProcessSupervisor --> Workload + Workload -->|"egress redirected on loopback"| NetworkSidecar + NetworkSidecar -->|"gateway forwarding"| Gateway + NetworkSidecar -->|"policy-enforced egress"| External +``` + +Use this topology when a cluster admin can install node-level CNI components and +you want sandbox pods to avoid the sidecar topology's privileged network init +container. + +Each node must provide a firewall backend usable from the host-executed plugin. +The plugin prefers `nft` and falls back to `iptables` when `nft` is unavailable. +The installer DaemonSet writes plugin failures to `cni.logFile` and tails that +file, so `kubectl logs daemonset/openshell-cni -c install-cni` includes CNI +`ADD` and `CHECK` errors that would otherwise only appear in pod events or node +logs. + + +`cni-sidecar` is experimental. It currently targets normal Kubernetes runtimes +first. Kata Containers and gVisor remain validation targets, because each +runtime must honor the CNI-installed pod-network rules for this topology to +provide the expected network enforcement. + + ## Credential Exposure -Sidecar topology keeps gateway credentials in the network sidecar. The agent -container does not mount the projected ServiceAccount token used for sandbox -token bootstrap, does not mount the sandbox client TLS secret, and does not get -gateway callback environment variables. +Sidecar and cni-sidecar topologies keep gateway credentials in the network +sidecar. The agent container does not mount the projected ServiceAccount token +used for sandbox token bootstrap, does not mount the sandbox client TLS secret, +and does not get gateway callback environment variables. The network sidecar serves the policy and workload-facing provider environment over a Unix control socket in the shared sidecar state volume. Before launching @@ -190,10 +256,13 @@ gVisor does not provide to the init container. A supported sandboxed runtime strengthens the container boundary while OpenShell focuses on network policy enforcement from the sidecar. +`cni-sidecar` is intended to test whether CNI-installed pod-network rules can +make sidecar-style enforcement work on stricter runtime classes. + Runtime classes do not re-enable the OpenShell privilege-drop or supervisor -mount-isolation controls that sidecar mode relaxes. Use them as an additional -workload boundary, not as a replacement for the combined topology's full -supervisor controls. +mount-isolation controls that sidecar and cni-sidecar modes relax. Use them as +an additional workload boundary, not as a replacement for the combined +topology's full supervisor controls. You can set a default runtime class in the Kubernetes driver configuration or override it per sandbox with driver config: @@ -206,7 +275,10 @@ openshell sandbox create \ ## Enable Sidecar Mode -For direct gateway TOML configuration, set the Kubernetes driver fields: +For direct gateway TOML configuration, set the Kubernetes driver fields for +the topology you want to test. + +For sidecar mode: ```toml [openshell.drivers.kubernetes] @@ -222,6 +294,19 @@ runs the sidecar as UID 0 instead. The network init container exempts the effective sidecar UID from proxy redirection so the sidecar can reach the gateway. +For CNI-sidecar mode: + +```toml +[openshell.drivers.kubernetes] +topology = "cni-sidecar" + +[openshell.drivers.kubernetes.sidecar] +proxy_uid = 1337 +``` + +In CNI-sidecar mode, the OpenShell CNI plugin installs the pod-network rules +that the network init container installs in `sidecar` mode. + When the Helm chart renders `gateway.toml`, set the equivalent chart values: ```yaml @@ -232,6 +317,21 @@ supervisor: processBinaryAwareNetworkPolicy: true ``` +Set `supervisor.topology=cni-sidecar` and enable the CNI installer to use CNI +sidecar mode: + +```yaml +cni: + enabled: true +supervisor: + topology: cni-sidecar + sidecar: + proxyUid: 1337 +``` + +The OpenShell CNI installer requires privileged node access so it can copy the +plugin into the host CNI binary directory and patch the node CNI conflist. + Leave `topology` unset, or set it to `combined`, to keep the original single-container supervisor path. For Helm installs, leave `supervisor.topology` unset or set it to `combined`. diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 2fd5717aec..5c0c33e8d5 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -329,6 +329,10 @@ supervisor_sideload_method = "image-volume" # "combined" runs the existing single supervisor container with full process, # filesystem, and network enforcement in the agent container. "sidecar" moves # pod-level network enforcement and gateway session handling into a network sidecar. +# "cni-sidecar" keeps the sidecar runtime model but expects an OpenShell CNI +# plugin to install the pod-network rules before the workload starts. In sidecar +# and cni-sidecar modes, the agent container runs non-root with no added Linux +# capabilities and process/filesystem enforcement is network-only. topology = "combined" grpc_endpoint = "https://openshell-gateway.agents.svc:8080" ssh_socket_path = "/run/openshell/ssh.sock" @@ -362,7 +366,8 @@ provider_spiffe_workload_api_socket_path = "/spiffe-workload-api/spire-agent.soc # UID used by relaxed long-running network sidecars. Strict process/binary-aware # sidecars run as UID 0 so Kubernetes grants the required /proc inspection # capabilities into the effective set. In sidecar topology the network init -# container installs nftables rules that exempt the effective sidecar UID. +# container installs nftables rules that exempt the effective sidecar UID; in +# cni-sidecar topology the OpenShell CNI plugin installs equivalent rules. proxy_uid = 1337 # Keep process/binary-aware network policy enabled in sidecar topology. Set # false to run the sidecar as proxy_uid, drop the sidecar's extra /proc diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 675ffaff6a..4542f7f2f5 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -320,8 +320,8 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA | `supervisor_image` | `supervisor.image.repository` / `supervisor.image.tag` | Override the supervisor image that provides the `openshell-sandbox` binary. The default repository with an empty tag uses the version-pinned image built into the gateway. Changing the repository uses the effective gateway image tag, while setting a tag pins that version explicitly. | | `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the Kubernetes image pull policy for the supervisor image. | | `supervisor_sideload_method` | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect from cluster version. Set to `image-volume` to mount the supervisor OCI image directly as a volume (requires Kubernetes 1.33+ with the ImageVolume feature gate; GA in 1.36), or `init-container` to copy it through an init container on older clusters. | -| `topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, or `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar. | -| `sidecar.proxy_uid` | `supervisor.sidecar.proxyUid` | Non-root UID used by the relaxed sidecar when process/binary-aware network policy is disabled. The default binary-aware sidecar runs as UID 0. The network init container exempts the effective sidecar UID from proxy redirection. | +| `topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar, or `cni-sidecar` to use that sidecar runtime model with CNI-installed pod-network rules. | +| `sidecar.proxy_uid` | `supervisor.sidecar.proxyUid` | Non-root UID used by the relaxed sidecar when process/binary-aware network policy is disabled. The default binary-aware sidecar runs as UID 0. In `sidecar` topology the network init container exempts the effective sidecar UID from proxy redirection; in `cni-sidecar` topology the OpenShell CNI plugin installs equivalent nftables or iptables exemptions. | | `sidecar.process_binary_aware_network_policy` | `supervisor.sidecar.processBinaryAwareNetworkPolicy` | Keep process/binary-aware network policy enabled in `sidecar` topology. The default runs the sidecar as UID 0 with `SYS_PTRACE` and `DAC_READ_SEARCH`. Set false to run as `proxy_uid`, drop both capabilities, and enforce endpoint/L7 policy without matching `policy.binaries`. | | `app_armor_profile` | `server.appArmorProfile` | Set the sandbox agent container's AppArmor profile. Helm defaults this to `Unconfined` so AppArmor-enabled nodes do not block supervisor network namespace setup. Set the Helm value to an empty string to omit the field, or use `RuntimeDefault` or `Localhost/` for operator-managed profiles. | | `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Set the default workspace PVC size for new sandboxes. | @@ -353,6 +353,11 @@ identity mount isolation. Network policy still runs in the sidecar, and sidecar pods set `shareProcessNamespace: true` so the network sidecar can resolve process/binary identity through `/proc/`. +CNI-sidecar mode keeps the sidecar runtime model and local snapshot boundary, +but removes the privileged pod-local network init container. Instead, the +privileged OpenShell CNI DaemonSet installs the nftables or iptables pod-network +rules during CNI `ADD` before the workload starts. + The Kubernetes driver creates namespaced `agents.x-k8s.io` `Sandbox` resources from the Kubernetes SIG Apps [agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) project. It detects the served Sandbox API at runtime, caches the selected API version for the gateway process, and uses `v1beta1` when available before falling back to `v1alpha1`, so supported Agent Sandbox installations work without version-specific operator configuration. The Agent Sandbox controller turns those resources into sandbox pods and related storage. If Agent Sandbox is upgraded in place, restart the OpenShell gateway after the controller and CRD rollout completes so the gateway can detect the served API versions again. diff --git a/tasks/helm.toml b/tasks/helm.toml index 24b6667b1d..db321ace9e 100644 --- a/tasks/helm.toml +++ b/tasks/helm.toml @@ -65,6 +65,11 @@ description = "Run skaffold dev with the Kubernetes supervisor sidecar topology dir = "deploy/helm/openshell" run = "skaffold dev -p sidecar-mtls" +["helm:skaffold:dev:cni-sidecar"] +description = "Run skaffold dev with CNI sidecar topology; requires OpenShell CNI installer permissions on nodes" +dir = "deploy/helm/openshell" +run = "skaffold dev -p cni-sidecar" + ["helm:skaffold:run"] description = "Run skaffold run for deploy/helm/openshell (one-shot deploy)" dir = "deploy/helm/openshell" @@ -80,6 +85,11 @@ description = "Run skaffold run with the Kubernetes supervisor sidecar topology dir = "deploy/helm/openshell" run = "skaffold run -p sidecar-mtls" +["helm:skaffold:run:cni-sidecar"] +description = "Run skaffold run with CNI sidecar topology; requires OpenShell CNI installer permissions on nodes" +dir = "deploy/helm/openshell" +run = "skaffold run -p cni-sidecar" + ["helm:skaffold:delete"] description = "Run skaffold delete for deploy/helm/openshell" dir = "deploy/helm/openshell" @@ -95,6 +105,11 @@ description = "Run skaffold delete for the Kubernetes supervisor sidecar topolog dir = "deploy/helm/openshell" run = "skaffold delete -p sidecar-mtls" +["helm:skaffold:delete:cni-sidecar"] +description = "Run skaffold delete for the Kubernetes CNI sidecar topology" +dir = "deploy/helm/openshell" +run = "skaffold delete -p cni-sidecar" + ["helm:skaffold:diagnose"] description = "Run skaffold diagnose for deploy/helm/openshell" dir = "deploy/helm/openshell" diff --git a/tasks/scripts/docker-build-image.sh b/tasks/scripts/docker-build-image.sh index 8570180f12..b4d79389fa 100755 --- a/tasks/scripts/docker-build-image.sh +++ b/tasks/scripts/docker-build-image.sh @@ -44,7 +44,7 @@ required_prebuilt_binaries() { echo "openshell-gateway" ;; supervisor|supervisor-sideload|supervisor-output) - echo "openshell-sandbox" + echo "openshell-sandbox openshell-cni" ;; esac } diff --git a/tasks/scripts/stage-prebuilt-binaries.sh b/tasks/scripts/stage-prebuilt-binaries.sh index 331d45a5b4..b153bdbbe8 100755 --- a/tasks/scripts/stage-prebuilt-binaries.sh +++ b/tasks/scripts/stage-prebuilt-binaries.sh @@ -12,7 +12,7 @@ ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" source "${SCRIPT_DIR}/build-env.sh" usage() { - echo "Usage: stage-prebuilt-binaries.sh " >&2 + echo "Usage: stage-prebuilt-binaries.sh " >&2 } normalize_arch() { @@ -91,7 +91,10 @@ components_for_target() { echo "gateway" ;; sandbox|supervisor|supervisor-output) - echo "supervisor" + echo "supervisor cni" + ;; + cni) + echo "cni" ;; all) echo "gateway supervisor" @@ -115,6 +118,11 @@ resolve_component() { binary=openshell-sandbox target_libc=musl ;; + cni) + crate=openshell-cni + binary=openshell-cni + target_libc=musl + ;; *) echo "unsupported binary component: $1" >&2 exit 1 diff --git a/tasks/test.toml b/tasks/test.toml index ceb1c30086..3328cd82ad 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -146,6 +146,11 @@ description = "Run Kubernetes e2e with the supervisor sidecar topology overlay" env = { OPENSHELL_E2E_KUBE_EXTRA_VALUES = "deploy/helm/openshell/ci/values-sidecar.yaml" } run = "e2e/rust/e2e-kubernetes.sh" +["e2e:kubernetes:cni-sidecar"] +description = "Run Kubernetes e2e with the CNI sidecar topology overlay; requires OpenShell CNI installer permissions on nodes" +env = { OPENSHELL_E2E_KUBE_EXTRA_VALUES = "deploy/helm/openshell/ci/values-cni-sidecar.yaml" } +run = "e2e/rust/e2e-kubernetes.sh" + ["e2e:kubernetes:db"] description = "Run Kubernetes e2e with all database backend scenarios (SQLite and external PostgreSQL with existingSecret)" env = { OPENSHELL_E2E_KUBE_DB_SCENARIOS = "1" } From d4596f6340b26f9f09f381d0253b5206120e9e77 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Thu, 30 Jul 2026 14:33:46 -0400 Subject: [PATCH 02/20] docs(cni): design for cni-sidecar on OpenShift Multus Signed-off-by: Russell Bryant --- ...-30-cni-sidecar-openshift-multus-design.md | 173 ++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-30-cni-sidecar-openshift-multus-design.md diff --git a/docs/superpowers/specs/2026-07-30-cni-sidecar-openshift-multus-design.md b/docs/superpowers/specs/2026-07-30-cni-sidecar-openshift-multus-design.md new file mode 100644 index 0000000000..f0d3fad328 --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-cni-sidecar-openshift-multus-design.md @@ -0,0 +1,173 @@ +# CNI-sidecar on OpenShift (Multus / OVN-Kubernetes) — Design + +Date: 2026-07-30 +Branch: `feat/kubernetes-cni-sidecar-topology` +Status: Approved design, pending implementation plan + +## Problem + +The `cni-sidecar` topology installs a chained CNI plugin (`openshell-cni`) on every +node via a privileged DaemonSet. During CNI `ADD`, the plugin reads pod annotations +and installs nftables/iptables bypass-prevention rules in the sandbox pod's network +namespace so the workload's egress is forced through the sidecar proxy +(`SIDECAR_PROXY_PORT = 3128`). + +The installer as written targets a **vanilla / k3s CNI layout**: it searches +`cni.confDir` for a `*.conflist`, appends `openshell-cni` to that file's `.plugins` +array, backs it up, and unpatches on `preStop`. + +This model does not work on OpenShift. Verified on a single-node OpenShift 4.22 +cluster (RHCOS, cri-o 1.35, **OVNKubernetes + Multus**): + +- CNI bin dir is `/var/lib/cni/bin` (cri-o `plugin_dirs`); `/opt/cni/bin` does not exist. +- CNI conf dir is `/etc/kubernetes/cni/net.d/` (cri-o `network_dir`); `/etc/cni/net.d` + does not exist. +- The only conf there is `00-multus.conf` (type `multus-shim`) — a single `.conf` + with **no `.plugins` array**. +- The delegated OVN config (`/run/multus/cni/net.d/10-ovn-kubernetes.conf`) is also a + single-plugin `.conf`, CNO-managed and regenerated. + +So there is no `.conflist` to append to. The installer's `find ... -name '*.conflist'` +step exits 1 (safe, but non-functional). Forcing it at a CNO-managed `.conf` would be +reverted and risks breaking cluster networking. + +## Chosen mechanism: Multus auxiliary CNI chain + +The Multus thick daemon on this cluster has `auxiliaryCNIChainName: "vendor-cni-chain"` +set (`multus-daemon-config` ConfigMap). Multus runs an **isolated auxiliary CNI chain +for every pod**, loading configs from a `vendor-cni-chain/` subdirectory of the +cluster-network config directory — **without touching any CNO-managed file**. The base +directory is derived from the `clusterNetwork` path in `00-multus.conf` +(`/host/run/multus/cni/net.d/10-ovn-kubernetes.conf`), so the on-host chain directory is: + +```text +/run/multus/cni/net.d/vendor-cni-chain/ +``` + +Dropping a standalone `openshell-cni` `.conf` there causes Multus to run our plugin as +an isolated auxiliary chain for all pods. Our plugin already no-ops for pods without the +`openshell.ai/cni: enabled` annotation (unit test `add_passes_through_non_openshell_pod`) +and passes `prevResult` through, so it does not disturb other pods. + +References: + +- Multus configuration reference: +- Multus thick plugin: + +## Approach + +Approach 1 (chosen): add a second **install mode** to the existing CNI DaemonSet, +selected by a new `cni.mode` value. Keep the conflist-append logic unchanged for +k3s/vanilla (preserves the existing `mise run e2e:kubernetes:cni-sidecar` CI path). Add +a `multus-chain` mode for OpenShift. Isolate the OpenShift difference to (a) a script +branch, (b) path defaults, and (c) an SCC template. + +Rejected: a separate OpenShift DaemonSet template (duplicates pod spec / volumes / +cert-refresh loop / RBAC) and a bespoke operator (YAGNI for a chained-plugin drop-in). + +## Design + +### Install modes + +`cni.mode` selects behavior: + +- `conflist` (default, unchanged): find first non-openshell `*.conflist` in + `cni.confDir`, append `openshell-cni` to `.plugins`, back up the original, unpatch on + `preStop`. +- `multus-chain` (new): write a standalone CNI `.conf` + (`{"cniVersion", "name":"openshell-cni", "type":"openshell-cni", "openshell":{...}}`) + into `cni.chainDir`. Never modify a CNO-managed file. `preStop` removes the `.conf` + and the credential files. + +The `openshell-cni` binary is installed into `cni.binDir` in both modes. + +### Paths & credentials + +New/updated values (OpenShift defaults): + +| Value | OpenShift default | Purpose | +|-------|-------------------|---------| +| `cni.mode` | `multus-chain` (overlay); `conflist` (chart default) | Install strategy | +| `cni.binDir` | `/var/lib/cni/bin` | Plugin binary location | +| `cni.chainDir` | `/run/multus/cni/net.d/vendor-cni-chain` | Aux-chain `.conf` (multus-chain only) | +| `cni.stateDir` | `/etc/kubernetes/cni/openshell` | **Persistent** kubeconfig/token/ca.crt/log | + +`cni.chainDir` is tmpfs (`/run`), so credentials cannot live there. The DaemonSet writes +kubeconfig/token/ca.crt to the persistent `cni.stateDir`, and the plugin `.conf` +references those `stateDir` paths. The DaemonSet's periodic loop (already refreshing the +SA token every 300s) additionally **re-asserts the chain `.conf` if missing**, so it +survives a Multus restart; on a full node reboot the DaemonSet re-writes at startup. + +For back-compat, when `cni.mode == conflist` the `stateDir` default resolves to +`cni.confDir` (current behavior). + +### RBAC & SCC + +- Add `pods: get` for the CNI ServiceAccount, scoped to the sandbox namespace(s). Under + Multus, pod annotations are not passed via `CNI_ARGS`, so the plugin queries the API by + pod name/namespace and requires this verb. (Implicitly required on k3s too; verify the + existing e2e path.) +- New template `templates/cni-scc.yaml`, gated by `cni.openshift.privilegedSCC` + (default `false`): a ClusterRole granting `use` on + `securitycontextconstraints/privileged` (apiGroup `security.openshift.io`) plus a + ClusterRoleBinding to the CNI ServiceAccount. Gating keeps non-OpenShift installs from + referencing OpenShift-only APIs. + +### Chart guardrails + +- The existing `fail` guard (`topology=cni-sidecar` requires `cni.enabled=true`) stays. +- Add validation: `cni.mode` must be `conflist` or `multus-chain`; `multus-chain` + requires `cni.chainDir`. + +## Files + +- `deploy/helm/openshell/templates/cni-daemonset.yaml` — branch install/preStop on + `cni.mode`; add `chainDir`/`stateDir` volumes and mounts; re-assert conf in the loop. +- `deploy/helm/openshell/templates/cni-scc.yaml` — new, gated SCC ClusterRole+binding. +- `deploy/helm/openshell/templates/clusterrole.yaml` (or `role.yaml`) — add `pods: get`. +- `deploy/helm/openshell/values.yaml` — new `cni.mode`, `cni.chainDir`, `cni.stateDir`, + `cni.openshift.privilegedSCC`; updated comments. +- `deploy/helm/openshell/README.md` — regenerated via `mise run helm:docs`. +- `deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml` — new overlay. +- `deploy/helm/openshell/tests/cni_daemonset_test.yaml` — add multus-chain assertions. +- `deploy/helm/openshell/tests/cni_scc_test.yaml` — new SCC unit tests. +- `docs/kubernetes/topology.mdx`, `architecture/compute-runtimes.md` — document the + OpenShift/Multus mode. +- `crates/openshell-cni/*` — likely no functional change; confirm pod-annotation lookup + path and log-file writability under the aux chain. + +## Testing plan (on the OpenShift cluster) + +Prerequisites: + +1. Confirm the supervisor image ships `/openshell-cni` (DaemonSet copies it from the image). +2. Images onto the cluster: expose the internal registry (patch + `configs.imageregistry/cluster`: set `storage` and `defaultRoute: true`), + `oc registry login`, build via `mise run build:docker:gateway` / + `build:docker:supervisor`, tag/push; or push to ghcr. Set the image repo in the overlay. +3. Install the `agent-sandbox` CRDs + controller (`AGENT_SANDBOX_VERSION`) — not present + on the cluster. + +Steps: + +1. Helm install gateway + CNI with the OpenShift overlay. +2. Verify: DaemonSet Running; `openshell-cni` in `/var/lib/cni/bin`; `.conf` in + `/run/multus/cni/net.d/vendor-cni-chain/`; a throwaway pod confirms the aux chain runs + without breaking normal networking. +3. Functional: create a `cni-sidecar` sandbox; inspect the pod netns for the nft/iptables + redirect rules; confirm egress is forced through the sidecar proxy (allowed dest works, + direct bypass blocked). + +Validation gates before deploying: +`mise run pre-commit`, `mise run test`, `mise run helm:test`, `mise run helm:lint`, +`mise run helm:docs`. + +## Known limitations / follow-ups + +- **Fail-open**: the chained-plugin model does not block pod networking if the + `openshell-cni` conf is absent (e.g. brief window after a node reboot before the + DaemonSet re-asserts). This is inherent to `cni-sidecar` generally, not + OpenShift-specific. Fail-closed hardening is out of scope here and left as a follow-up. +- `vendor-cni-chain` availability depends on the Multus thick daemon having + `auxiliaryCNIChainName` set. Present on OpenShift 4.22; the chart should document the + requirement rather than attempt to configure Multus. From 8002dd3d37ffbe864edb055fd5806e206b882ed3 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Thu, 30 Jul 2026 14:55:50 -0400 Subject: [PATCH 03/20] feat(cni): add multus-chain install mode for OpenShift Signed-off-by: Russell Bryant --- deploy/helm/openshell/README.md | 3 + .../openshell/templates/cni-daemonset.yaml | 81 +++++++++++++++---- .../openshell/tests/cni_daemonset_test.yaml | 52 ++++++++++++ deploy/helm/openshell/values.yaml | 12 +++ 4 files changed, 133 insertions(+), 15 deletions(-) diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index b03688eead..e8cfe2188d 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -150,6 +150,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | | cni.affinity | object | `{}` | | | cni.binDir | string | `"/opt/cni/bin"` | Host CNI binary directory. | +| cni.chainDir | string | `""` | Host Multus vendor-cni-chain directory. Only used when mode is "multus-chain". The installer writes openshell-cni.conf here. | | cni.confDir | string | `"/etc/cni/net.d"` | Host CNI config directory. | | cni.configFile | string | `""` | Host CNI conflist filename patched by the installer. Empty selects the first non-OpenShell .conflist. | | cni.enabled | bool | `false` | Install the OpenShell chained CNI plugin with a privileged node DaemonSet. Required when supervisor.topology is "cni-sidecar". | @@ -158,8 +159,10 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | cni.image.tag | string | `""` | CNI installer image tag. Empty uses supervisor.image.tag, then chart appVersion. | | cni.logFile | string | `"/var/log/openshell-cni.log"` | Host log file written by the OpenShell CNI plugin and tailed by the installer DaemonSet. | | cni.logLevel | string | `"info"` | Log level passed to the OpenShell CNI plugin. | +| cni.mode | string | `"conflist"` | CNI installer strategy. "conflist" appends the OpenShell plugin to an existing CNI .conflist (k3s / vanilla). "multus-chain" writes a standalone plugin .conf into a Multus vendor-cni-chain subdirectory (OpenShift), which never modifies a CNO-managed file. | | cni.nodeSelector | object | `{}` | | | cni.resources | object | `{}` | | +| cni.stateDir | string | `""` | Persistent host directory for plugin credentials (kubeconfig, token, ca.crt) in "multus-chain" mode. Empty falls back to confDir. Must not be a tmpfs path such as one under /run. | | cni.tolerations | list | `[]` | | | fullnameOverride | string | `""` | Override the full generated resource name. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml index b19596db42..8d2e6aff5b 100644 --- a/deploy/helm/openshell/templates/cni-daemonset.yaml +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -1,13 +1,24 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 +{{- $mode := .Values.cni.mode | default "conflist" -}} {{- if and (eq (.Values.supervisor.topology | default "combined") "cni-sidecar") (not .Values.cni.enabled) -}} {{- fail "supervisor.topology=cni-sidecar requires cni.enabled=true so the OpenShell chained CNI plugin is installed on every node" -}} {{- end }} {{- if .Values.cni.enabled }} +{{- if not (has $mode (list "conflist" "multus-chain")) -}} +{{- fail "cni.mode must be \"conflist\" or \"multus-chain\"" -}} +{{- end }} +{{- if and (eq $mode "multus-chain") (not .Values.cni.chainDir) -}} +{{- fail "cni.mode=multus-chain requires cni.chainDir" -}} +{{- end }} {{- $sandboxNamespace := .Values.server.sandboxNamespace | default .Release.Namespace }} {{- $cniLogFile := .Values.cni.logFile | default "/var/log/openshell-cni.log" }} {{- $cniLogDir := dir $cniLogFile }} +{{- $stateDir := .Values.cni.confDir }} +{{- if eq $mode "multus-chain" }} +{{- $stateDir = .Values.cni.stateDir | default .Values.cni.confDir }} +{{- end }} apiVersion: apps/v1 kind: DaemonSet metadata: @@ -42,15 +53,15 @@ spec: - -ec - | bin_dir="/host{{ .Values.cni.binDir }}" - conf_dir="/host{{ .Values.cni.confDir }}" - host_conf_dir="{{ .Values.cni.confDir }}" + state_dir="/host{{ $stateDir }}" + host_state_dir="{{ $stateDir }}" log_file="{{ $cniLogFile }}" host_log_file="/host${log_file}" - kubeconfig="${conf_dir}/openshell-cni-kubeconfig" - token_file="${conf_dir}/openshell-cni-token" - ca_file="${conf_dir}/openshell-cni-ca.crt" - plugin_config="${conf_dir}/openshell-cni-plugin.json" - mkdir -p "${bin_dir}" "${conf_dir}" + kubeconfig="${state_dir}/openshell-cni-kubeconfig" + token_file="${state_dir}/openshell-cni-token" + ca_file="${state_dir}/openshell-cni-ca.crt" + plugin_config="${state_dir}/openshell-cni-plugin.json" + mkdir -p "${bin_dir}" "${state_dir}" touch "${host_log_file}" chmod 0644 "${host_log_file}" install -m 0755 /openshell-cni "${bin_dir}/openshell-cni" @@ -65,11 +76,11 @@ spec: - name: cluster cluster: server: https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT} - certificate-authority: ${host_conf_dir}/openshell-cni-ca.crt + certificate-authority: ${host_state_dir}/openshell-cni-ca.crt users: - name: openshell-cni user: - token-file: ${host_conf_dir}/openshell-cni-token + token-file: ${host_state_dir}/openshell-cni-token contexts: - name: openshell-cni context: @@ -80,13 +91,15 @@ spec: { "type": "openshell-cni", "openshell": { - "kubeconfig": "${host_conf_dir}/openshell-cni-kubeconfig", + "kubeconfig": "${host_state_dir}/openshell-cni-kubeconfig", "sandboxNamespaces": [{{ $sandboxNamespace | quote }}], "logLevel": {{ .Values.cni.logLevel | quote }}, "logFile": "${log_file}" } } EOF + {{- if eq $mode "conflist" }} + conf_dir="/host{{ .Values.cni.confDir }}" {{- if .Values.cni.configFile }} target="${conf_dir}/{{ .Values.cni.configFile }}" {{- else }} @@ -104,7 +117,20 @@ spec: .plugins = ((.plugins // []) | map(select(.type != "openshell-cni")) + [$openshell[0]]) ' "${target}" > "${tmp}" mv "${tmp}" "${target}" - echo "OpenShell CNI installed; plugin log ${log_file}; firewall backend selected during CNI ADD" + echo "OpenShell CNI installed (conflist mode); plugin log ${log_file}; firewall backend selected during CNI ADD" + {{- else }} + chain_dir="/host{{ .Values.cni.chainDir }}" + chain_conf="${chain_dir}/openshell-cni.conf" + mkdir -p "${chain_dir}" + write_chain_conf() { + _tmp="$(mktemp "${chain_dir}/openshell-cni.XXXXXX")" + jq -n --slurpfile openshell "${plugin_config}" \ + '{"cniVersion": "1.0.0", "name": "openshell-cni"} + $openshell[0]' > "${_tmp}" + mv "${_tmp}" "${chain_conf}" + } + write_chain_conf + echo "OpenShell CNI installed (multus-chain mode) at ${chain_conf}; plugin log ${log_file}; firewall backend selected during CNI ADD" + {{- end }} tail -n 0 -F "${host_log_file}" & tail_pid="$!" trap 'kill "${tail_pid}" 2>/dev/null || true; exit 0' INT TERM @@ -112,6 +138,9 @@ spec: while true; do cp /var/run/secrets/kubernetes.io/serviceaccount/token "${token_file}" chmod 0600 "${token_file}" + {{- if eq $mode "multus-chain" }} + [ -f "${chain_conf}" ] || write_chain_conf + {{- end }} sleep 300 done securityContext: @@ -124,6 +153,7 @@ spec: - sh - -ec - | + {{- if eq $mode "conflist" }} conf_dir="/host{{ .Values.cni.confDir }}" for target in "${conf_dir}"/*.conflist; do [ -f "${target}" ] || continue @@ -137,16 +167,26 @@ spec: ' "${target}" > "${tmp}" mv "${tmp}" "${target}" done + {{- else }} + rm -f "/host{{ .Values.cni.chainDir }}/openshell-cni.conf" + {{- end }} rm -f \ - "${conf_dir}/openshell-cni-ca.crt" \ - "${conf_dir}/openshell-cni-kubeconfig" \ - "${conf_dir}/openshell-cni-plugin.json" \ - "${conf_dir}/openshell-cni-token" + "/host{{ $stateDir }}/openshell-cni-ca.crt" \ + "/host{{ $stateDir }}/openshell-cni-kubeconfig" \ + "/host{{ $stateDir }}/openshell-cni-plugin.json" \ + "/host{{ $stateDir }}/openshell-cni-token" volumeMounts: - name: cni-bin mountPath: /host{{ .Values.cni.binDir }} + {{- if eq $mode "conflist" }} - name: cni-conf mountPath: /host{{ .Values.cni.confDir }} + {{- else }} + - name: cni-chain + mountPath: /host{{ .Values.cni.chainDir }} + - name: cni-state + mountPath: /host{{ $stateDir }} + {{- end }} - name: cni-log mountPath: /host{{ $cniLogDir }} {{- with .Values.cni.resources }} @@ -158,10 +198,21 @@ spec: hostPath: path: {{ .Values.cni.binDir | quote }} type: DirectoryOrCreate + {{- if eq $mode "conflist" }} - name: cni-conf hostPath: path: {{ .Values.cni.confDir | quote }} type: Directory + {{- else }} + - name: cni-chain + hostPath: + path: {{ .Values.cni.chainDir | quote }} + type: DirectoryOrCreate + - name: cni-state + hostPath: + path: {{ $stateDir | quote }} + type: DirectoryOrCreate + {{- end }} - name: cni-log hostPath: path: {{ $cniLogDir | quote }} diff --git a/deploy/helm/openshell/tests/cni_daemonset_test.yaml b/deploy/helm/openshell/tests/cni_daemonset_test.yaml index d54f05ed39..0aab6bf1c0 100644 --- a/deploy/helm/openshell/tests/cni_daemonset_test.yaml +++ b/deploy/helm/openshell/tests/cni_daemonset_test.yaml @@ -69,3 +69,55 @@ tests: asserts: - failedTemplate: errorMessage: supervisor.topology=cni-sidecar requires cni.enabled=true so the OpenShell chained CNI plugin is installed on every node + + - it: renders multus-chain conf and state volumes when mode is multus-chain + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + cni.mode: multus-chain + cni.chainDir: /run/multus/cni/net.d/vendor-cni-chain + cni.stateDir: /etc/kubernetes/cni/openshell + asserts: + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-chain + mountPath: /host/run/multus/cni/net.d/vendor-cni-chain + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-state + mountPath: /host/etc/kubernetes/cni/openshell + - notContains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: cni-conf + mountPath: /host/etc/cni/net.d + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'multus-chain mode' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '"name": "openshell-cni"' + - matchRegex: + path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] + pattern: 'openshell-cni\.conf' + + - it: fails multus-chain mode without a chain dir + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + cni.mode: multus-chain + cni.chainDir: "" + asserts: + - failedTemplate: + errorMessage: cni.mode=multus-chain requires cni.chainDir + + - it: fails on an unknown cni mode + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + cni.mode: bogus + asserts: + - failedTemplate: + errorMessage: cni.mode must be "conflist" or "multus-chain" diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index c8535fb5ee..56e1c7e7e7 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -69,6 +69,11 @@ cni: # -- Install the OpenShell chained CNI plugin with a privileged node DaemonSet. # Required when supervisor.topology is "cni-sidecar". enabled: false + # -- CNI installer strategy. "conflist" appends the OpenShell plugin to an + # existing CNI .conflist (k3s / vanilla). "multus-chain" writes a standalone + # plugin .conf into a Multus vendor-cni-chain subdirectory (OpenShift), which + # never modifies a CNO-managed file. + mode: conflist image: # -- CNI installer image repository. Empty uses supervisor.image.repository. repository: "" @@ -86,6 +91,13 @@ cni: logFile: /var/log/openshell-cni.log # -- Host CNI conflist filename patched by the installer. Empty selects the first non-OpenShell .conflist. configFile: "" + # -- Host Multus vendor-cni-chain directory. Only used when mode is + # "multus-chain". The installer writes openshell-cni.conf here. + chainDir: "" + # -- Persistent host directory for plugin credentials (kubeconfig, token, + # ca.crt) in "multus-chain" mode. Empty falls back to confDir. Must not be a + # tmpfs path such as one under /run. + stateDir: "" resources: {} nodeSelector: {} tolerations: [] From b4a6e84843f07bd7a5975253d9fa9afc4ebe1b9f Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Thu, 30 Jul 2026 15:04:32 -0400 Subject: [PATCH 04/20] feat(cni): add gated privileged SCC for OpenShift Signed-off-by: Russell Bryant --- deploy/helm/openshell/README.md | 1 + deploy/helm/openshell/templates/cni-scc.yaml | 37 ++++++++++++++++ deploy/helm/openshell/tests/cni_scc_test.yaml | 44 +++++++++++++++++++ deploy/helm/openshell/values.yaml | 5 +++ 4 files changed, 87 insertions(+) create mode 100644 deploy/helm/openshell/templates/cni-scc.yaml create mode 100644 deploy/helm/openshell/tests/cni_scc_test.yaml diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index e8cfe2188d..aedf64be49 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -161,6 +161,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | cni.logLevel | string | `"info"` | Log level passed to the OpenShell CNI plugin. | | cni.mode | string | `"conflist"` | CNI installer strategy. "conflist" appends the OpenShell plugin to an existing CNI .conflist (k3s / vanilla). "multus-chain" writes a standalone plugin .conf into a Multus vendor-cni-chain subdirectory (OpenShift), which never modifies a CNO-managed file. | | cni.nodeSelector | object | `{}` | | +| cni.openshift.privilegedSCC | bool | `false` | Grant the privileged SecurityContextConstraints to the CNI ServiceAccount. Required on OpenShift for the privileged installer DaemonSet. Creates a ClusterRole + ClusterRoleBinding. | | cni.resources | object | `{}` | | | cni.stateDir | string | `""` | Persistent host directory for plugin credentials (kubeconfig, token, ca.crt) in "multus-chain" mode. Empty falls back to confDir. Must not be a tmpfs path such as one under /run. | | cni.tolerations | list | `[]` | | diff --git a/deploy/helm/openshell/templates/cni-scc.yaml b/deploy/helm/openshell/templates/cni-scc.yaml new file mode 100644 index 0000000000..ce7ed5876d --- /dev/null +++ b/deploy/helm/openshell/templates/cni-scc.yaml @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if and .Values.cni.enabled .Values.cni.openshift.privilegedSCC }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "openshell.fullname" . }}-cni-scc + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni +rules: + - apiGroups: + - security.openshift.io + resources: + - securitycontextconstraints + resourceNames: + - privileged + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "openshell.fullname" . }}-cni-scc + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "openshell.fullname" . }}-cni-scc +subjects: + - kind: ServiceAccount + name: {{ include "openshell.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/deploy/helm/openshell/tests/cni_scc_test.yaml b/deploy/helm/openshell/tests/cni_scc_test.yaml new file mode 100644 index 0000000000..325dc63889 --- /dev/null +++ b/deploy/helm/openshell/tests/cni_scc_test.yaml @@ -0,0 +1,44 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: cni scc +templates: + - templates/cni-scc.yaml +tests: + - it: does not render by default + asserts: + - hasDocuments: + count: 0 + + - it: does not render when cni enabled but scc disabled + set: + cni.enabled: true + asserts: + - hasDocuments: + count: 0 + + - it: renders clusterrole and binding when privilegedSCC enabled + set: + cni.enabled: true + cni.openshift.privilegedSCC: true + asserts: + - hasDocuments: + count: 2 + - documentIndex: 0 + isKind: + of: ClusterRole + - documentIndex: 0 + contains: + path: rules + content: + apiGroups: + - security.openshift.io + resources: + - securitycontextconstraints + resourceNames: + - privileged + verbs: + - use + - documentIndex: 1 + isKind: + of: ClusterRoleBinding diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 56e1c7e7e7..d419b969ea 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -102,6 +102,11 @@ cni: nodeSelector: {} tolerations: [] affinity: {} + openshift: + # -- Grant the privileged SecurityContextConstraints to the CNI + # ServiceAccount. Required on OpenShift for the privileged installer + # DaemonSet. Creates a ClusterRole + ClusterRoleBinding. + privilegedSCC: false # -- Image pull secrets attached to gateway and helper pods. imagePullSecrets: [] From 98110fb06be06c7251cc167f71eed1b40bbd09ef Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Thu, 30 Jul 2026 15:07:35 -0400 Subject: [PATCH 05/20] feat(helm): add OpenShift cni-sidecar values overlay Signed-off-by: Russell Bryant --- .../ci/values-cni-sidecar-openshift.yaml | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml diff --git a/deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml b/deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml new file mode 100644 index 0000000000..ed9e7dd987 --- /dev/null +++ b/deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Overlay for the Kubernetes CNI-sidecar topology on OpenShift (Multus / OVN-K). +# Injects the OpenShell chained plugin via the Multus vendor-cni-chain auxiliary +# chain instead of patching a CNI .conflist, and grants the privileged SCC. + +cni: + enabled: true + mode: multus-chain + # OpenShift cri-o plugin_dirs use /var/lib/cni/bin; /opt/cni/bin does not exist. + binDir: /var/lib/cni/bin + # Multus vendor-cni-chain subdirectory (derived from the clusterNetwork path in + # 00-multus.conf). Tmpfs, so it is re-asserted by the installer loop. + chainDir: /run/multus/cni/net.d/vendor-cni-chain + # Persistent directory for plugin credentials (chainDir is tmpfs). + stateDir: /etc/kubernetes/cni/openshell + openshift: + privilegedSCC: true + +supervisor: + topology: cni-sidecar From 36f910db0f24e737ae0a848956208e61412e73c5 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Thu, 30 Jul 2026 15:11:06 -0400 Subject: [PATCH 06/20] docs(cni): document OpenShift multus-chain mode Signed-off-by: Russell Bryant --- architecture/compute-runtimes.md | 7 +++++++ docs/kubernetes/topology.mdx | 20 ++++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 5c5a735749..2dacf9a8cc 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -249,6 +249,13 @@ annotates sandbox pods so the chained CNI plugin can read the proxy UID and enforcement mode, and both the agent container and long-running network sidecar stay non-root with no added Linux capabilities. +The CNI installer supports two modes. `conflist` (default) appends the +`openshell-cni` plugin to an existing CNI `.conflist` (k3s / vanilla). On +OpenShift (Multus / OVN-Kubernetes) there is no appendable `.conflist`, so +`multus-chain` writes a standalone plugin `.conf` into the Multus +`vendor-cni-chain` auxiliary-chain directory and stores plugin credentials in a +persistent `stateDir`. Neither mode modifies a CNO-managed file. + ## Images The gateway image and Helm chart are built from this repository. Sandbox images diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index 3d8024e736..716d61222b 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -332,6 +332,26 @@ supervisor: The OpenShell CNI installer requires privileged node access so it can copy the plugin into the host CNI binary directory and patch the node CNI conflist. +### OpenShift (Multus / OVN-Kubernetes) + +On OpenShift the default network is managed by the Cluster Network Operator and +there is no CNI `.conflist` to append to. Use `cni.mode: multus-chain`, which +injects the OpenShell plugin through the Multus `vendor-cni-chain` auxiliary chain +without modifying any operator-managed file. The installer also needs the +`privileged` SecurityContextConstraints. + +Install with the provided overlay: + +```shell +helm install openshell deploy/helm/openshell \ + -f deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml +``` + +This sets `cni.binDir: /var/lib/cni/bin`, `cni.chainDir: +/run/multus/cni/net.d/vendor-cni-chain`, a persistent `cni.stateDir`, and +`cni.openshift.privilegedSCC: true`. It requires a Multus thick daemon with +`auxiliaryCNIChainName` configured (default on OpenShift 4.x). + Leave `topology` unset, or set it to `combined`, to keep the original single-container supervisor path. For Helm installs, leave `supervisor.topology` unset or set it to `combined`. From 8b9cf8f40baa685bc38c1f4b48c10197a016a499 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Thu, 30 Jul 2026 15:27:03 -0400 Subject: [PATCH 07/20] fix(cni): scope privileged SCC to a dedicated CNI service account Signed-off-by: Russell Bryant --- deploy/helm/openshell/templates/_helpers.tpl | 9 +++ .../openshell/templates/cni-daemonset.yaml | 2 +- deploy/helm/openshell/templates/cni-rbac.yaml | 47 ++++++++++++++ deploy/helm/openshell/templates/cni-scc.yaml | 2 +- .../openshell/tests/cni_daemonset_test.yaml | 13 ++++ .../helm/openshell/tests/cni_rbac_test.yaml | 63 +++++++++++++++++++ deploy/helm/openshell/tests/cni_scc_test.yaml | 4 ++ 7 files changed, 138 insertions(+), 2 deletions(-) create mode 100644 deploy/helm/openshell/templates/cni-rbac.yaml create mode 100644 deploy/helm/openshell/tests/cni_rbac_test.yaml diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index b5619585dd..65563f152d 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -59,6 +59,15 @@ Create the name of the service account to use {{- end }} {{- end }} +{{/* +Name of the ServiceAccount used by the CNI installer DaemonSet. Dedicated so +the privileged SCC and pod-get RBAC stay scoped to the CNI installer rather +than the shared gateway service account. +*/}} +{{- define "openshell.cniServiceAccountName" -}} +{{- printf "%s-cni" (include "openshell.fullname" .) | trunc 63 | trimSuffix "-" }} +{{- end }} + {{/* Create the name of the service account assigned to sandbox pods */}} diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml index 8d2e6aff5b..273e43bb66 100644 --- a/deploy/helm/openshell/templates/cni-daemonset.yaml +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -39,7 +39,7 @@ spec: spec: hostNetwork: true dnsPolicy: ClusterFirstWithHostNet - serviceAccountName: {{ include "openshell.serviceAccountName" . }} + serviceAccountName: {{ include "openshell.cniServiceAccountName" . }} {{- with .Values.imagePullSecrets }} imagePullSecrets: {{- toYaml . | nindent 8 }} diff --git a/deploy/helm/openshell/templates/cni-rbac.yaml b/deploy/helm/openshell/templates/cni-rbac.yaml new file mode 100644 index 0000000000..468b27b2e0 --- /dev/null +++ b/deploy/helm/openshell/templates/cni-rbac.yaml @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if .Values.cni.enabled }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "openshell.cniServiceAccountName" . }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "openshell.cniServiceAccountName" . }} + namespace: {{ include "openshell.sandboxNamespace" . }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni +rules: + # The CNI plugin reads the sandbox pod's annotations during CNI ADD to + # decide whether to install bypass-prevention rules in the pod netns. + - apiGroups: + - "" + resources: + - pods + verbs: + - get +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "openshell.cniServiceAccountName" . }} + namespace: {{ include "openshell.sandboxNamespace" . }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "openshell.cniServiceAccountName" . }} +subjects: + - kind: ServiceAccount + name: {{ include "openshell.cniServiceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/deploy/helm/openshell/templates/cni-scc.yaml b/deploy/helm/openshell/templates/cni-scc.yaml index ce7ed5876d..62f7aa8dc5 100644 --- a/deploy/helm/openshell/templates/cni-scc.yaml +++ b/deploy/helm/openshell/templates/cni-scc.yaml @@ -32,6 +32,6 @@ roleRef: name: {{ include "openshell.fullname" . }}-cni-scc subjects: - kind: ServiceAccount - name: {{ include "openshell.serviceAccountName" . }} + name: {{ include "openshell.cniServiceAccountName" . }} namespace: {{ .Release.Namespace }} {{- end }} diff --git a/deploy/helm/openshell/tests/cni_daemonset_test.yaml b/deploy/helm/openshell/tests/cni_daemonset_test.yaml index 0aab6bf1c0..0bee0c9913 100644 --- a/deploy/helm/openshell/tests/cni_daemonset_test.yaml +++ b/deploy/helm/openshell/tests/cni_daemonset_test.yaml @@ -51,6 +51,9 @@ tests: - matchRegex: path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] pattern: 'select\(\.type != "openshell-cni"\)' + - matchRegex: + path: spec.template.spec.serviceAccountName + pattern: -cni$ - it: scopes cni annotation lookup to explicit sandbox namespace template: templates/cni-daemonset.yaml @@ -102,6 +105,16 @@ tests: - matchRegex: path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] pattern: 'openshell-cni\.conf' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'host_state_dir="/etc/kubernetes/cni/openshell"' + - contains: + path: spec.template.spec.volumes + content: + name: cni-state + hostPath: + path: /etc/kubernetes/cni/openshell + type: DirectoryOrCreate - it: fails multus-chain mode without a chain dir template: templates/cni-daemonset.yaml diff --git a/deploy/helm/openshell/tests/cni_rbac_test.yaml b/deploy/helm/openshell/tests/cni_rbac_test.yaml new file mode 100644 index 0000000000..c1759614a7 --- /dev/null +++ b/deploy/helm/openshell/tests/cni_rbac_test.yaml @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: cni rbac +templates: + - templates/cni-rbac.yaml +tests: + - it: does not render by default + asserts: + - hasDocuments: + count: 0 + + - it: renders service account, role and binding when cni enabled + set: + cni.enabled: true + asserts: + - hasDocuments: + count: 3 + - documentIndex: 0 + isKind: + of: ServiceAccount + - documentIndex: 0 + matchRegex: + path: metadata.name + pattern: -cni$ + - documentIndex: 1 + isKind: + of: Role + - documentIndex: 1 + contains: + path: rules + content: + apiGroups: + - "" + resources: + - pods + verbs: + - get + - documentIndex: 2 + isKind: + of: RoleBinding + - documentIndex: 2 + matchRegex: + path: subjects[0].name + pattern: -cni$ + - documentIndex: 2 + matchRegex: + path: roleRef.name + pattern: -cni$ + + - it: scopes cni pod-get role to the explicit sandbox namespace + set: + cni.enabled: true + server.sandboxNamespace: agents + asserts: + - documentIndex: 1 + equal: + path: metadata.namespace + value: agents + - documentIndex: 2 + equal: + path: metadata.namespace + value: agents diff --git a/deploy/helm/openshell/tests/cni_scc_test.yaml b/deploy/helm/openshell/tests/cni_scc_test.yaml index 325dc63889..e973bca4dc 100644 --- a/deploy/helm/openshell/tests/cni_scc_test.yaml +++ b/deploy/helm/openshell/tests/cni_scc_test.yaml @@ -42,3 +42,7 @@ tests: - documentIndex: 1 isKind: of: ClusterRoleBinding + - documentIndex: 1 + matchRegex: + path: subjects[0].name + pattern: -cni$ From 4aa2c2ee3f5e85dcf8e450ef811f20f8c7f016a4 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Thu, 30 Jul 2026 17:51:13 -0400 Subject: [PATCH 08/20] feat(helm): add minimal sandbox SCC for OpenShift binary-aware policy On OpenShift the binary-aware network sidecar must run as UID 0 with SYS_PTRACE and DAC_READ_SEARCH to inspect cross-UID /proc, which the restricted-v2 SCC forbids. Add a gated minimal SecurityContextConstraints (restricted-v2 baseline plus exactly those two capabilities and the image volume type) with a ClusterRole and ClusterRoleBinding granting it to the sandbox ServiceAccount. Controlled by sandboxServiceAccount.openshift.binaryAwareSCC (default false); the cni-sidecar OpenShift overlay enables it. Signed-off-by: Russell Bryant --- architecture/compute-runtimes.md | 9 ++ deploy/helm/openshell/README.md | 1 + .../ci/values-cni-sidecar-openshift.yaml | 7 ++ deploy/helm/openshell/templates/_helpers.tpl | 8 ++ .../helm/openshell/templates/sandbox-scc.yaml | 85 +++++++++++++++++ .../openshell/tests/sandbox_scc_test.yaml | 95 +++++++++++++++++++ deploy/helm/openshell/values.yaml | 11 +++ docs/kubernetes/topology.mdx | 28 +++++- 8 files changed, 241 insertions(+), 3 deletions(-) create mode 100644 deploy/helm/openshell/templates/sandbox-scc.yaml create mode 100644 deploy/helm/openshell/tests/sandbox_scc_test.yaml diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 2dacf9a8cc..af0453489d 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -256,6 +256,15 @@ OpenShift (Multus / OVN-Kubernetes) there is no appendable `.conflist`, so `vendor-cni-chain` auxiliary-chain directory and stores plugin credentials in a persistent `stateDir`. Neither mode modifies a CNO-managed file. +On OpenShift, binary-aware network policy also requires a purpose-built +SecurityContextConstraints for sandbox pods: the network sidecar runs as UID 0 +with `SYS_PTRACE` and `DAC_READ_SEARCH` to inspect cross-UID `/proc`, which +`restricted-v2` forbids. `sandboxServiceAccount.openshift.binaryAwareSCC` creates +a minimal SCC (the `restricted-v2` baseline plus only those two capabilities, UID +0, and the `image` volume type) and binds it to the sandbox ServiceAccount. +Disabling `processBinaryAwareNetworkPolicy` drops the capability requirement and +lets the stock `restricted-v2` SCC apply. + ## Images The gateway image and Helm chart are built from this repository. Sandbox images diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index aedf64be49..6fbb7e0abb 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -206,6 +206,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | sandboxServiceAccount.annotations | object | `{}` | Annotations to add to the generated sandbox service account. | | sandboxServiceAccount.create | bool | `true` | Create a service account for sandbox pods. | | sandboxServiceAccount.name | string | `""` | Existing service account name for sandbox pods when sandboxServiceAccount.create is false. | +| sandboxServiceAccount.openshift.binaryAwareSCC | bool | `false` | Create a minimal SecurityContextConstraints for sandbox pods and grant it to the sandbox ServiceAccount. Required on OpenShift for "sidecar" and "cni-sidecar" topologies when supervisor.sidecar.processBinaryAwareNetworkPolicy is true: the network sidecar must run as UID 0 with SYS_PTRACE and DAC_READ_SEARCH to inspect cross-UID /proc, which restricted-v2 forbids. The SCC is minimal — it adds only those two capabilities plus the image volume type over the restricted baseline; everything else stays locked down (no privileged, no host namespaces, drop ALL, seccomp runtime/default). Creates a SecurityContextConstraints + ClusterRole + ClusterRoleBinding. | | securityContext.allowPrivilegeEscalation | bool | `false` | Whether the gateway container can gain additional privileges. | | securityContext.capabilities.drop | list | `["ALL"]` | Linux capabilities dropped from the gateway container. | | securityContext.runAsNonRoot | bool | `true` | Require the gateway container to run as a non-root user. | diff --git a/deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml b/deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml index ed9e7dd987..36669ba5bd 100644 --- a/deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml +++ b/deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml @@ -20,3 +20,10 @@ cni: supervisor: topology: cni-sidecar + +sandboxServiceAccount: + openshift: + # Sandbox pods run the network sidecar as UID 0 with SYS_PTRACE and + # DAC_READ_SEARCH for binary-aware policy; restricted-v2 forbids this, so + # grant the sandbox SA the minimal purpose-built SCC. + binaryAwareSCC: true diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 65563f152d..e8c3290d5c 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -79,6 +79,14 @@ Create the name of the service account assigned to sandbox pods {{- end }} {{- end }} +{{/* +Name of the minimal SecurityContextConstraints (and its ClusterRole/binding) +granted to sandbox pods on OpenShift for binary-aware network policy. +*/}} +{{- define "openshell.sandboxSccName" -}} +{{- printf "%s-sandbox" (include "openshell.fullname" .) | trunc 63 | trimSuffix "-" }} +{{- end }} + {{/* Gateway image reference. Uses image.tag when set; falls back to .Chart.AppVersion so a released chart automatically pulls the matching image without extra overrides. diff --git a/deploy/helm/openshell/templates/sandbox-scc.yaml b/deploy/helm/openshell/templates/sandbox-scc.yaml new file mode 100644 index 0000000000..91b5287de6 --- /dev/null +++ b/deploy/helm/openshell/templates/sandbox-scc.yaml @@ -0,0 +1,85 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if .Values.sandboxServiceAccount.openshift.binaryAwareSCC }} +apiVersion: security.openshift.io/v1 +kind: SecurityContextConstraints +metadata: + name: {{ include "openshell.sandboxSccName" . }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: sandbox +# Minimal SCC for binary-aware network sidecars: the restricted-v2 baseline +# plus exactly what cross-UID /proc inspection requires — UID 0 and the two +# capabilities SYS_PTRACE and DAC_READ_SEARCH — and the image volume type used +# to sideload the supervisor binary. Everything else stays locked down. +allowPrivilegedContainer: false +allowPrivilegeEscalation: false +allowHostNetwork: false +allowHostPorts: false +allowHostPID: false +allowHostIPC: false +allowHostDirVolumePlugin: false +readOnlyRootFilesystem: false +defaultAddCapabilities: null +requiredDropCapabilities: + - ALL +allowedCapabilities: + - SYS_PTRACE + - DAC_READ_SEARCH +runAsUser: + type: RunAsAny +seLinuxContext: + type: MustRunAs +fsGroup: + type: RunAsAny +supplementalGroups: + type: RunAsAny +seccompProfiles: + - runtime/default +volumes: + - configMap + - csi + - downwardAPI + - emptyDir + - ephemeral + - image + - persistentVolumeClaim + - projected + - secret +users: [] +groups: [] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "openshell.sandboxSccName" . }}-scc + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: sandbox +rules: + - apiGroups: + - security.openshift.io + resources: + - securitycontextconstraints + resourceNames: + - {{ include "openshell.sandboxSccName" . }} + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "openshell.sandboxSccName" . }}-scc + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: sandbox +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "openshell.sandboxSccName" . }}-scc +subjects: + - kind: ServiceAccount + name: {{ include "openshell.sandboxServiceAccountName" . }} + namespace: {{ include "openshell.sandboxNamespace" . }} +{{- end }} diff --git a/deploy/helm/openshell/tests/sandbox_scc_test.yaml b/deploy/helm/openshell/tests/sandbox_scc_test.yaml new file mode 100644 index 0000000000..3c5fc6f93f --- /dev/null +++ b/deploy/helm/openshell/tests/sandbox_scc_test.yaml @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: sandbox scc +templates: + - templates/sandbox-scc.yaml +tests: + - it: does not render by default + asserts: + - hasDocuments: + count: 0 + + - it: renders scc, clusterrole and binding when binaryAwareSCC enabled + set: + sandboxServiceAccount.openshift.binaryAwareSCC: true + asserts: + - hasDocuments: + count: 3 + - documentIndex: 0 + isKind: + of: SecurityContextConstraints + - documentIndex: 0 + equal: + path: allowPrivilegedContainer + value: false + - documentIndex: 0 + equal: + path: allowPrivilegeEscalation + value: false + - documentIndex: 0 + equal: + path: requiredDropCapabilities + value: + - ALL + - documentIndex: 0 + equal: + path: allowedCapabilities + value: + - SYS_PTRACE + - DAC_READ_SEARCH + - documentIndex: 0 + equal: + path: runAsUser.type + value: RunAsAny + - documentIndex: 0 + contains: + path: volumes + content: image + - documentIndex: 1 + isKind: + of: ClusterRole + - documentIndex: 1 + contains: + path: rules + content: + apiGroups: + - security.openshift.io + resources: + - securitycontextconstraints + resourceNames: + - RELEASE-NAME-openshell-sandbox + verbs: + - use + - documentIndex: 2 + isKind: + of: ClusterRoleBinding + - documentIndex: 2 + matchRegex: + path: subjects[0].name + pattern: -sandbox$ + - documentIndex: 2 + matchRegex: + path: roleRef.name + pattern: -sandbox-scc$ + + - it: binds the scc to an existing sandbox service account name + set: + sandboxServiceAccount.openshift.binaryAwareSCC: true + sandboxServiceAccount.create: false + sandboxServiceAccount.name: custom-sandbox-sa + asserts: + - documentIndex: 2 + equal: + path: subjects[0].name + value: custom-sandbox-sa + + - it: scopes the scc binding subject to the explicit sandbox namespace + set: + sandboxServiceAccount.openshift.binaryAwareSCC: true + server.sandboxNamespace: agents + asserts: + - documentIndex: 2 + equal: + path: subjects[0].namespace + value: agents diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index d419b969ea..732683e8ec 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -130,6 +130,17 @@ sandboxServiceAccount: annotations: {} # -- Existing service account name for sandbox pods when sandboxServiceAccount.create is false. name: "" + openshift: + # -- Create a minimal SecurityContextConstraints for sandbox pods and grant + # it to the sandbox ServiceAccount. Required on OpenShift for "sidecar" and + # "cni-sidecar" topologies when supervisor.sidecar.processBinaryAwareNetworkPolicy + # is true: the network sidecar must run as UID 0 with SYS_PTRACE and + # DAC_READ_SEARCH to inspect cross-UID /proc, which restricted-v2 forbids. + # The SCC is minimal — it adds only those two capabilities plus the image + # volume type over the restricted baseline; everything else stays locked + # down (no privileged, no host namespaces, drop ALL, seccomp runtime/default). + # Creates a SecurityContextConstraints + ClusterRole + ClusterRoleBinding. + binaryAwareSCC: false # -- Extra annotations to add to the gateway pod. podAnnotations: {} diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index 716d61222b..9e4fb8a041 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -348,9 +348,31 @@ helm install openshell deploy/helm/openshell \ ``` This sets `cni.binDir: /var/lib/cni/bin`, `cni.chainDir: -/run/multus/cni/net.d/vendor-cni-chain`, a persistent `cni.stateDir`, and -`cni.openshift.privilegedSCC: true`. It requires a Multus thick daemon with -`auxiliaryCNIChainName` configured (default on OpenShift 4.x). +/run/multus/cni/net.d/vendor-cni-chain`, a persistent `cni.stateDir`, +`cni.openshift.privilegedSCC: true`, and +`sandboxServiceAccount.openshift.binaryAwareSCC: true`. It requires a Multus +thick daemon with `auxiliaryCNIChainName` configured (default on OpenShift 4.x). + +#### Sandbox pod SecurityContextConstraints + +With `supervisor.sidecar.processBinaryAwareNetworkPolicy` enabled (the default), +the network sidecar runs as UID 0 with the `SYS_PTRACE` and `DAC_READ_SEARCH` +capabilities so it can inspect the agent process's `/proc` entries across UID +boundaries. OpenShift's `restricted-v2` SCC forbids UID 0 and those +capabilities, so the sandbox ServiceAccount needs a purpose-built SCC. + +Setting `sandboxServiceAccount.openshift.binaryAwareSCC: true` creates a minimal +SCC — the `restricted-v2` baseline plus only UID 0, `SYS_PTRACE`, +`DAC_READ_SEARCH`, and the `image` volume type used to sideload the supervisor +binary — and a ClusterRole/ClusterRoleBinding granting it to the sandbox +ServiceAccount. Everything else stays locked down: no privileged container, no +host namespaces, `allowPrivilegeEscalation: false`, drop `ALL`, and the +`runtime/default` seccomp profile. + +To run with lower privileges at the cost of binary matching, set +`supervisor.sidecar.processBinaryAwareNetworkPolicy: false`; the sidecar then +runs as `proxyUid` without the extra capabilities, and the built-in +`restricted-v2` SCC suffices (leave `binaryAwareSCC` unset). Leave `topology` unset, or set it to `combined`, to keep the original single-container supervisor path. For Helm installs, leave From 84f82eda0353b611277958059b6b387db4754ad2 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Thu, 30 Jul 2026 20:15:29 -0400 Subject: [PATCH 09/20] docs(rfc): propose cni-sidecar supervisor topology Signed-off-by: Russell Bryant --- rfc/cni-sidecar-topology-DRAFT.md | 480 ++++++++++++++++++++++++++++++ 1 file changed, 480 insertions(+) create mode 100644 rfc/cni-sidecar-topology-DRAFT.md diff --git a/rfc/cni-sidecar-topology-DRAFT.md b/rfc/cni-sidecar-topology-DRAFT.md new file mode 100644 index 0000000000..4a66e2dc62 --- /dev/null +++ b/rfc/cni-sidecar-topology-DRAFT.md @@ -0,0 +1,480 @@ +--- +authors: + - "@russellb" +state: draft +links: + - (originating GitHub issue — RFC number to be assigned by maintainers) +--- + +# RFC NNNN - CNI-Sidecar Supervisor Topology (and OpenShift/Multus Enablement) + + + +## Summary + +This RFC proposes `cni-sidecar`, a third Kubernetes supervisor topology for +OpenShell sandbox pods. It keeps the split-supervisor runtime model of the +existing `sidecar` topology — a network-enforcement sidecar plus a +low-privilege process supervisor in the agent container — but moves pod-network +rule installation out of the sandbox pod entirely. A privileged, node-level +OpenShell CNI DaemonSet installs a chained CNI plugin on every node; during CNI +`ADD` the plugin reads OpenShell pod annotations and installs the +bypass-prevention rules in the pod's network namespace before the workload +starts. This removes the per-pod privileged network init container that +`sidecar` topology requires. + +The RFC also proposes the configuration surface needed to run this topology on +managed CNI platforms — specifically OpenShift with Multus and OVN-Kubernetes — +where there is no `.conflist` to append to and where pods run under restrictive +SecurityContextConstraints (SCC). This includes a second CNI install mode +(`multus-chain`), an accompanying set of host-path defaults, and two gated, +minimal SCC grants (one for the CNI DaemonSet, one for the sandbox pod). + +## Motivation + +OpenShell's `combined` topology runs the full supervisor — network, filesystem, +and process controls — inside the agent container, which requires that container +to carry elevated Linux capabilities (`SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, +and others). Many clusters will not admit a workload container with those +capabilities. The `sidecar` topology addresses part of this by moving network +enforcement into a dedicated sidecar and running the agent container as a +low-privilege, network-only process supervisor. But `sidecar` still needs a +**privileged network init container** in every sandbox pod to install the +pod-local nftables rules that fence egress through the sidecar. That init +container needs `NET_ADMIN`/`NET_RAW`, which is exactly the kind of per-pod +privilege stricter clusters — and stricter runtime classes like gVisor — want to +eliminate. + +The problem is worse on OpenShift. There, the sandbox pod init container's +capabilities collide with `restricted-v2`, and even if the sidecar model is +used, the network init container is a per-pod privileged surface that cluster +security teams object to. Operators who want OpenShell's network policy +enforcement on OpenShift currently have no clean path. + +If we leave the design unchanged, OpenShell's network enforcement remains +coupled to either a highly privileged agent container (`combined`) or a +per-pod privileged init container (`sidecar`). Neither fits clusters that push +network-privileged operations to the node/CNI layer, which is where cluster +admins already expect that privilege to live. Node-level CNI installation is a +one-time, admin-scoped grant; per-pod privileged init containers are a +recurring, workload-adjacent grant. Moving the rule installation to the CNI +layer aligns the privilege boundary with how clusters are actually governed. + +## Non-goals + +- **Fail-closed networking.** The chained-plugin model is fail-open: if the + OpenShell CNI conf is absent (e.g., briefly after a node reboot before the + DaemonSet re-asserts it), pod networking is not blocked. This is inherent to + the `cni-sidecar` approach and is left as a follow-up, not addressed here. +- **Configuring Multus.** This RFC consumes the Multus `vendor-cni-chain` + auxiliary chain when it is present; it does not propose configuring or + installing Multus, nor setting `auxiliaryCNIChainName`. +- **Replacing `combined` or `sidecar`.** Both remain; `combined` stays the + default and the only topology that provides the full supervisor contract + (filesystem policy, privilege drop, mount isolation). +- **Non-Kubernetes drivers.** Docker, Podman, and VM drivers are unaffected. +- **New CNI plugin behavior.** The `openshell-cni` plugin binary itself is + reused as-is; this RFC is about how it is installed and permitted, not what it + does during `ADD`. +- **gVisor/Kata certification.** `cni-sidecar` is proposed as experimental with + those runtime classes as validation targets, not as a supported guarantee. + +## Proposal + +### Topology overview + +`cni-sidecar` reuses the `sidecar` runtime split and changes only where the +network rules come from: + +```mermaid +flowchart TB + Sandbox["agents.x-k8s.io Sandbox"] + + subgraph Node["Kubernetes node"] + DaemonSet["OpenShell CNI DaemonSet
(privileged, hostNetwork)"] + CNIPlugin["OpenShell chained CNI plugin"] + Runtime["container runtime
CNI ADD / CHECK"] + + subgraph Pod["Sandbox pod"] + NetNS["pod network namespace"] + subgraph Agent["agent container"] + ProcessSupervisor["process supervisor
network-only, no added caps"] + Workload["Agent workload"] + end + NetworkSidecar["network supervisor sidecar"] + end + end + + Gateway["OpenShell Gateway"] + External["External services"] + + Sandbox --> Pod + DaemonSet -->|"installs plugin binary + conf on node"| CNIPlugin + Runtime -->|"invokes on pod setup"| CNIPlugin + CNIPlugin -->|"reads pod annotations (API lookup)"| Pod + CNIPlugin -->|"installs nftables/iptables rules"| NetNS + ProcessSupervisor --> Workload + Workload -->|"egress redirected on loopback"| NetworkSidecar + NetworkSidecar -->|"gateway forwarding"| Gateway + NetworkSidecar -->|"policy-enforced egress"| External +``` + +Key difference from `sidecar`: there is **no per-pod network init container**. +The DaemonSet's chained plugin installs the loopback-redirect fence during pod +network setup, so the pod itself never needs `NET_ADMIN`/`NET_RAW`. + +### Component privilege model + +The privilege of each component is the central design point. The goal is that +the only privileged surface is the node-level installer (admin-scoped), and the +sandbox pod carries the minimum needed for the enforcement mode in use. + +| Component | Scope | UID | Privilege escalation | Capabilities | Notes | +|---|---|---|---|---|---| +| CNI installer DaemonSet | Node | 0 | true | `privileged: true` | `hostNetwork`, host-path mounts into CNI dirs. One privileged surface, installed once per node by an admin. | +| Agent container (process supervisor, `network-only`) | Pod | `sandbox_uid:sandbox_gid` | false | drops `ALL` | The workload runs here with no added Linux capabilities. | +| Network sidecar — binary-aware mode (default) | Pod | `0:sandbox_gid` | false | drops `ALL`, adds `SYS_PTRACE` + `DAC_READ_SEARCH` | UID 0 is required to inspect the agent's `/proc` across UID boundaries (see below). | +| Network sidecar — endpoint/L7-only mode | Pod | `proxyUid:sandbox_gid` | false | drops `ALL` | Non-root; enforces endpoint/L7 policy without `policy.binaries` matching. | +| Network rule setup | — | N/A | N/A | N/A | Performed by the node CNI plugin; **no pod-local init container**. | + +**Why the binary-aware sidecar must be UID 0.** Kubernetes `SecurityContext` +has no ambient-capability field. A process that starts as non-root loses added +capabilities across `execve`, because ambient caps are what carry capabilities +into a non-root program's effective set. To keep `SYS_PTRACE` and +`DAC_READ_SEARCH` effective — which the sidecar needs to read the agent +process's `/proc` entries across the UID boundary for binary-aware policy — the +sidecar must run as UID 0. This is not an OpenShell preference; it is the only +configuration in which Kubernetes keeps those capabilities effective for +cross-UID `/proc` inspection. The nftables fence exempts UID 0, so operators +must not inject other root containers into these pods. + +Operators who do not need binary matching can set +`processBinaryAwareNetworkPolicy: false`. The sidecar then runs as the non-root +`proxyUid` with no added capabilities and enforces endpoint/L7 policy only. On +OpenShift this variant admits under the built-in `restricted-v2` SCC with no +custom SCC required. + +### CNI install modes + +The CNI DaemonSet installs the `openshell-cni` binary into the host CNI bin +directory in all cases. How it wires the plugin into the node's CNI +configuration is selected by `cni.mode`: + +- **`conflist`** (default; k3s / vanilla): find the first non-OpenShell + `*.conflist` in `cni.confDir`, back it up, and append `openshell-cni` to its + `.plugins` array. `preStop` removes the plugin entry. This is the pre-existing + behavior and is unchanged. +- **`multus-chain`** (OpenShift): write a standalone CNI `.conf` into a Multus + `vendor-cni-chain` subdirectory. Multus runs an isolated auxiliary chain for + every pod from that directory **without modifying any operator-managed file**. + `preStop` removes the `.conf` and credential files. + +The mode split isolates the OpenShift-specific difference to a script branch, +path defaults, and RBAC/SCC — no separate DaemonSet, no operator. + +### Why `multus-chain` on OpenShift + +On OpenShift 4.x (RHCOS, cri-o, OVN-Kubernetes + Multus), the CNI layout defeats +the `conflist` approach: + +- CNI bin dir is `/var/lib/cni/bin` (cri-o `plugin_dirs`); `/opt/cni/bin` does + not exist. +- CNI conf dir is `/etc/kubernetes/cni/net.d/`; the only file is + `00-multus.conf` (type `multus-shim`), a single `.conf` with **no `.plugins` + array** to append to. +- The delegated OVN config is CNO-managed and regenerated, so patching it would + be reverted and risks breaking cluster networking. + +Multus's thick daemon, when configured with `auxiliaryCNIChainName` +(`vendor-cni-chain`, default on OpenShift 4.x), loads additional per-pod CNI +configs from a `vendor-cni-chain/` subdirectory of the cluster-network config +path. Dropping a standalone `openshell-cni.conf` there runs our plugin as an +isolated auxiliary chain for all pods without touching CNO-managed files. The +plugin already no-ops for pods lacking the OpenShell annotation and passes +`prevResult` through, so it does not disturb other pods. + +### Configuration surface + +New/changed chart values. Chart defaults remain non-OpenShift; OpenShift +settings live in the overlay `ci/values-cni-sidecar-openshift.yaml`. + +| Value | Default | OpenShift overlay | Purpose | +|---|---|---|---| +| `supervisor.topology` | `combined` | `cni-sidecar` | Select the topology. | +| `supervisor.sidecar.proxyUid` | `1337` | `1337` | UID for the relaxed endpoint/L7 sidecar; must be non-root and != sandbox UID. | +| `supervisor.sidecar.processBinaryAwareNetworkPolicy` | `true` | `true` | Keep binary-aware policy (sidecar as UID 0). | +| `cni.enabled` | `false` | `true` | Install the node CNI DaemonSet. Required for `cni-sidecar`. | +| `cni.mode` | `conflist` | `multus-chain` | Install strategy. | +| `cni.binDir` | `/opt/cni/bin` | `/var/lib/cni/bin` | Host CNI binary directory. | +| `cni.chainDir` | `""` | `/run/multus/cni/net.d/vendor-cni-chain` | Multus aux-chain dir (multus-chain only). | +| `cni.stateDir` | `""` (falls back to `confDir`) | `/etc/kubernetes/cni/openshell` | **Persistent** credential dir (chainDir is tmpfs). | +| `cni.openshift.privilegedSCC` | `false` | `true` | Grant `privileged` SCC to the CNI ServiceAccount. | +| `sandboxServiceAccount.openshift.binaryAwareSCC` | `false` | `true` | Create + grant the minimal sandbox SCC. | + +Guardrails enforced at template render time: + +- `supervisor.topology=cni-sidecar` requires `cni.enabled=true` (existing + guard). +- `cni.mode` must be `conflist` or `multus-chain`. +- `cni.mode=multus-chain` requires `cni.chainDir`. + +Because `cni.chainDir` is tmpfs (under `/run`), credentials cannot live there. +The DaemonSet writes kubeconfig/token/ca.crt to the persistent `cni.stateDir`, +references those paths from the plugin `.conf`, and its periodic loop (already +refreshing the SA token every 300s) re-asserts the chain `.conf` if it goes +missing, so the config survives a Multus restart; a node reboot is covered by +the DaemonSet re-writing on startup. + +### SCC model (OpenShift) + +Two independent, gated, minimal SCC grants — each off by default so non-OpenShift +installs never reference OpenShift-only APIs: + +1. **CNI DaemonSet** (`cni.openshift.privilegedSCC`): a ClusterRole granting + `use` on the built-in `privileged` SCC, bound to the CNI ServiceAccount. The + installer genuinely needs `privileged` for host-path writes into node CNI + directories. + +2. **Sandbox pod** (`sandboxServiceAccount.openshift.binaryAwareSCC`): a + purpose-built minimal SCC, plus a ClusterRole/ClusterRoleBinding granting it + to the sandbox ServiceAccount. It is the `restricted-v2` baseline plus exactly + what binary-aware `/proc` inspection needs — `runAsUser: RunAsAny` (permits + UID 0), `SYS_PTRACE`, `DAC_READ_SEARCH`, and the `image` volume type used to + sideload the supervisor binary. Everything else stays locked down: + `allowPrivilegedContainer: false`, `allowPrivilegeEscalation: false`, no host + namespaces, `requiredDropCapabilities: [ALL]`, `seccompProfiles: + [runtime/default]`. When `processBinaryAwareNetworkPolicy: false`, this SCC is + unnecessary and `restricted-v2` suffices. + +Neither uses `anyuid` (grants UID 0 but adds no capabilities) or the full +`privileged` SCC for the sandbox pod (wildly overbroad). The custom minimal SCC +is the smallest grant that retains functionality. + +### How it works end to end + +1. Operator installs the chart with the OpenShift overlay. Helm renders the + gateway, the CNI DaemonSet (`multus-chain`), the CNI `privileged` SCC grant, + and the minimal sandbox SCC + grant. +2. The DaemonSet, on each node, copies `openshell-cni` into `/var/lib/cni/bin`, + writes credentials into the persistent `stateDir`, and drops + `openshell-cni.conf` into the Multus `vendor-cni-chain` dir. +3. A `cni-sidecar` sandbox is created. The agent-sandbox controller reconciles + the `Sandbox` CR into a pod. OpenShift SCC admission evaluates the sandbox SA + and admits the pod under the minimal SCC (recorded in the + `openshift.io/scc` annotation). +4. During pod network setup, the runtime invokes the Multus aux chain; the + OpenShell plugin reads the pod's OpenShell annotations (via API lookup, since + Multus does not pass annotations through `CNI_ARGS`) and installs the + nftables/iptables loopback-redirect fence in the pod netns. +5. The sidecar starts (UID 0, `SYS_PTRACE` + `DAC_READ_SEARCH`), the process + supervisor starts network-only in the agent container, and workload egress is + fenced through the sidecar to the gateway and policy-enforced destinations. + +### Sample manifests + +Install (OpenShift overlay): + +```shell +helm install openshell deploy/helm/openshell \ + -f deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml +``` + +Overlay values (abridged): + +```yaml +cni: + enabled: true + mode: multus-chain + binDir: /var/lib/cni/bin + chainDir: /run/multus/cni/net.d/vendor-cni-chain + stateDir: /etc/kubernetes/cni/openshell + openshift: + privilegedSCC: true +supervisor: + topology: cni-sidecar +sandboxServiceAccount: + openshift: + binaryAwareSCC: true +``` + +Equivalent gateway TOML for the driver: + +```toml +[openshell.drivers.kubernetes] +topology = "cni-sidecar" + +[openshell.drivers.kubernetes.sidecar] +proxy_uid = 1337 +``` + +Rendered minimal sandbox SCC (abridged): + +```yaml +apiVersion: security.openshift.io/v1 +kind: SecurityContextConstraints +metadata: + name: openshell-sandbox +allowPrivilegedContainer: false +allowPrivilegeEscalation: false +allowHostNetwork: false +allowHostPID: false +allowHostIPC: false +requiredDropCapabilities: + - ALL +allowedCapabilities: + - SYS_PTRACE + - DAC_READ_SEARCH +runAsUser: + type: RunAsAny # permits UID 0 for the binary-aware sidecar +seLinuxContext: + type: MustRunAs +seccompProfiles: + - runtime/default +volumes: + - configMap + - csi + - downwardAPI + - emptyDir + - ephemeral + - image # sideloads the supervisor binary + - persistentVolumeClaim + - projected + - secret +``` + +Standalone Multus aux-chain `.conf` the DaemonSet writes into `chainDir`: + +```json +{ + "cniVersion": "1.0.0", + "name": "openshell-cni", + "type": "openshell-cni", + "openshell": { + "kubeconfig": "/etc/kubernetes/cni/openshell/openshell-cni-kubeconfig", + "sandboxNamespaces": ["openshell"], + "logLevel": "info", + "logFile": "/var/log/openshell-cni.log" + } +} +``` + +## Implementation plan + +The work is incremental and gated so no phase changes non-OpenShift behavior +until explicitly enabled. + +1. **CNI install mode.** Add `cni.mode` (+ `chainDir`, `stateDir`) to the chart; + branch the DaemonSet install/`preStop` script on mode; add `multus-chain` + volumes/mounts and the conf re-assert loop. Keep all existing `conflist` + render output byte-for-byte (existing `cni_daemonset_test.yaml` stays green). +2. **CNI SCC.** Add `templates/cni-scc.yaml` gated on + `cni.openshift.privilegedSCC`, with unit tests. +3. **Sandbox SCC.** Add `templates/sandbox-scc.yaml` + + `openshell.sandboxSccName` helper gated on + `sandboxServiceAccount.openshift.binaryAwareSCC`, with unit tests. +4. **RBAC.** Grant the CNI ServiceAccount `pods: get` scoped to the sandbox + namespace(s), needed because Multus does not pass annotations via `CNI_ARGS`. +5. **Overlay + docs.** Add `ci/values-cni-sidecar-openshift.yaml`; document the + topology, privilege model, and SCCs in `docs/kubernetes/topology.mdx` and + `architecture/compute-runtimes.md`; regenerate the chart README. +6. **Validation.** `mise run pre-commit`, `mise run test`, `mise run helm:test`, + `mise run helm:lint`; then a live cluster verify: DaemonSet Running, plugin + binary and conf present, a throwaway pod confirms the aux chain does not break + normal networking, and a `cni-sidecar` sandbox demonstrates the fence + (allowed dest works, direct bypass blocked) with the sidecar performing a + cross-UID `/proc` read under the minimal SCC. + +Existing users are unaffected: `combined` stays the default, and every OpenShift +behavior is behind a default-off value. + +## Risks + +- **Fail-open window.** If the OpenShell CNI conf is missing (post-reboot before + the DaemonSet re-asserts), pods can network without the fence. Mitigation: the + re-assert loop and startup write shrink the window; true fail-closed is a + follow-up. Operators diagnose via `kubectl logs daemonset/openshell-cni` and + the tailed plugin log. +- **Dependency on Multus aux chain.** `multus-chain` requires the Multus thick + daemon to have `auxiliaryCNIChainName` set. It is default on OpenShift 4.x but + not guaranteed everywhere. Mitigation: document the requirement; the chart does + not attempt to configure Multus. +- **Privileged node DaemonSet.** The installer runs `privileged` with + host-path access to CNI directories — a real node-level surface. Mitigation: + it is admin-scoped, installed once per node, and its SCC grant is gated and + explicit; this is the standard privilege location for CNI components. +- **UID 0 sidecar.** The binary-aware sidecar runs as root in the sandbox pod. + Mitigation: it drops `ALL` and adds only two capabilities; the nftables fence + exempts UID 0 (so no other root container may be injected); operators can opt + down to the non-root endpoint/L7 mode. +- **Experimental runtime coverage.** `cni-sidecar` targets normal runtimes + first; Kata/gVisor must honor CNI-installed pod-network rules for enforcement + to hold. Mitigation: ship as experimental with those as validation targets. +- **SCC drift / cluster policy.** A cluster that further restricts SCC or blocks + custom SCCs could reject the sandbox pod. Mitigation: the SCC is minimal and + documented; the lower-privilege endpoint/L7 mode admits under `restricted-v2`. + +## Alternatives + +### Do nothing + +Operators keep choosing between a highly privileged agent container +(`combined`) or a per-pod privileged network init container (`sidecar`). On +OpenShift, neither is clean, and there is no supported path for CNI-layer +enforcement. Rejected: it leaves a real class of clusters unserved. + +### Separate OpenShift DaemonSet template + +Ship a dedicated OpenShift DaemonSet instead of a mode switch. Rejected: it +duplicates the pod spec, volumes, cert-refresh loop, and RBAC, doubling +maintenance for a difference that is really just install path + SCC. + +### Bespoke OpenShift operator + +Manage installation via a custom operator. Rejected as YAGNI for a +chained-plugin drop-in; it adds a large surface and a new support obligation for +no capability the DaemonSet lacks. + +### Patch the CNO-managed CNI config directly + +Append `openshell-cni` to the OVN/Multus config. Rejected: CNO regenerates and +reverts these files, and a bad edit risks cluster-wide networking outages. The +Multus aux chain exists precisely to avoid touching operator-managed files. + +### Broader SCC (`anyuid` or `privileged`) for the sandbox pod + +Use an existing SCC rather than a custom one. Rejected: `anyuid` permits UID 0 +but adds no capabilities (so binary-aware policy breaks), and `privileged` is +far broader than needed. The minimal custom SCC is the smallest grant that +retains functionality. + +## Prior art + +- **Multus auxiliary CNI chains** (`vendor-cni-chain`): the upstream mechanism + for injecting vendor plugins per-pod without modifying primary CNI config. + Lesson: use the platform's sanctioned extension point instead of patching + managed files. + +- **Chained CNI plugins** (CNI spec `plugins` arrays): the general pattern of + composing a plugin that reads prior results and adds behavior; the OpenShell + plugin no-ops for non-OpenShell pods and passes `prevResult` through. +- **OpenShell `sidecar` topology**: the split-supervisor and control-socket + bootstrap model this topology reuses; `cni-sidecar` changes only rule + installation. See `docs/kubernetes/topology.mdx`. +- **OpenShift SCC model**: `restricted-v2` as the secure baseline and minimal + custom SCCs as the sanctioned way to grant narrowly scoped extra privilege. + +## Open questions + +- Should fail-closed enforcement (deny pod networking when the OpenShell conf is + absent) be promoted from follow-up to a required part of graduating + `cni-sidecar` out of experimental? +- Is `pods: get` for the CNI ServiceAccount acceptable cluster-wide, or should + it be namespace-scoped via Role/RoleBinding per sandbox namespace by default? +- Should the chart detect Multus `auxiliaryCNIChainName` and fail fast (or warn) + when `multus-chain` is selected but the aux chain is not configured? +- What is the graduation criteria (which runtime classes, which CNIs) for + removing the experimental label? From 61b91ca7826a661acbf587a77078c5d45632e617 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Mon, 3 Aug 2026 13:49:20 -0400 Subject: [PATCH 10/20] docs(rfc): link topology PRs in cni-sidecar RFC Signed-off-by: Russell Bryant --- rfc/cni-sidecar-topology-DRAFT.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rfc/cni-sidecar-topology-DRAFT.md b/rfc/cni-sidecar-topology-DRAFT.md index 4a66e2dc62..41157073bc 100644 --- a/rfc/cni-sidecar-topology-DRAFT.md +++ b/rfc/cni-sidecar-topology-DRAFT.md @@ -3,7 +3,9 @@ authors: - "@russellb" state: draft links: - - (originating GitHub issue — RFC number to be assigned by maintainers) + - https://github.com/NVIDIA/OpenShell/pull/2074 - kubernetes combined topology + - https://github.com/NVIDIA/OpenShell/pull/2076 - kubernetes sidecar topology + - https://github.com/NVIDIA/OpenShell/pull/2078 - original cni-sidecar topology PR from TaylorMutch --- # RFC NNNN - CNI-Sidecar Supervisor Topology (and OpenShift/Multus Enablement) From 4bb0e7c08105da56bd05c6bfc951b19f7d5b6683 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Mon, 3 Aug 2026 19:11:54 -0400 Subject: [PATCH 11/20] fix(cni-sidecar): address review findings for OpenShift topology Remediate merge-blocking review findings on the cni-sidecar supervisor topology: - driver: mount the gateway client mTLS bundle directly onto the network sidecar (cni-sidecar omits network-init, so the bundle was never seeded and the sidecar crash-looped on policy fetch) - driver: annotate the effective sidecar proxy UID (0 in binary-aware mode) so the node CNI exempts the right identity instead of redirecting the sidecar's own egress - cni: fail closed when ip6tables is missing in the iptables fallback so a dual-stack sandbox cannot bypass policy over IPv6 - cni: emit a CNI-spec error object on stdout so the runtime surfaces plugin failures instead of an opaque crash - helm: re-patch the chained plugin on the reconcile tick when absent so a CNI config rewrite or restart cannot drop egress enforcement - helm: keep the component label out of the immutable gateway selector so existing releases can upgrade - helm: mirror supervisorImage repository/tag fallbacks for the CNI image - docs: document the sidecar privilege profile and cold-start window; trim RFC non-goals Signed-off-by: Russell Bryant --- architecture/compute-runtimes.md | 30 +++- crates/openshell-cni/src/lib.rs | 97 +++++++++-- .../openshell-driver-kubernetes/src/driver.rs | 152 +++++++++++++++++- deploy/helm/openshell/templates/_helpers.tpl | 9 +- .../openshell/templates/cni-daemonset.yaml | 50 ++++-- .../helm/openshell/templates/deployment.yaml | 1 - .../helm/openshell/templates/statefulset.yaml | 1 - .../openshell/tests/gateway_config_test.yaml | 12 +- rfc/cni-sidecar-topology-DRAFT.md | 5 - 9 files changed, 307 insertions(+), 50 deletions(-) diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index af0453489d..8f61a66a45 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -242,12 +242,18 @@ network sidecar can resolve workload process and binary identity through `/proc/`. The cni-sidecar topology keeps the sidecar runtime model and its shared-state -boundary, but removes the privileged `openshell-network-init` init container. -Instead, the privileged OpenShell CNI DaemonSet installs the pod-network -bypass-prevention rules during CNI `ADD` using nftables or iptables. The driver -annotates sandbox pods so the chained CNI plugin can read the proxy UID and -enforcement mode, and both the agent container and long-running network sidecar -stay non-root with no added Linux capabilities. +boundary, but removes the privileged `openshell-network-init` init container and +its `NET_ADMIN`. Instead, the privileged OpenShell CNI DaemonSet installs the +pod-network bypass-prevention rules during CNI `ADD` using nftables or iptables. +The driver annotates sandbox pods so the chained CNI plugin can read the proxy +UID and enforcement mode. The network sidecar keeps the same privilege profile +as the other sidecar topologies: in the default binary-aware mode it runs as UID +0 with `SYS_PTRACE` and `DAC_READ_SEARCH` (but no `NET_ADMIN`) to resolve +cross-UID `/proc`, and only stays non-root with no added capabilities when +`process_binary_aware_network_policy` is disabled. The agent container stays +non-root with no added Linux capabilities in either mode. Because the node CNI — +not an in-pod container — programs the firewall, no container in the sandbox pod +holds `NET_ADMIN`. The CNI installer supports two modes. `conflist` (default) appends the `openshell-cni` plugin to an existing CNI `.conflist` (k3s / vanilla). On @@ -256,6 +262,18 @@ OpenShift (Multus / OVN-Kubernetes) there is no appendable `.conflist`, so `vendor-cni-chain` auxiliary-chain directory and stores plugin credentials in a persistent `stateDir`. Neither mode modifies a CNO-managed file. +The installer patches the chained plugin at startup and then re-verifies it on a +reconcile tick, re-patching when the plugin is missing. This keeps enforcement in +place across CNI config rewrites (for example a CNO reconcile) and DaemonSet +restarts, bounding any such gap to one reconcile interval. It does not cover the +cold-start scheduling race: a sandbox pod scheduled on a node before that node's +CNI DaemonSet pod has completed its first patch is admitted with unrestricted +egress, because the cni-sidecar topology deliberately omits the in-pod +fail-closed network-init backstop. Closing that window requires gating sandbox +scheduling on verified per-node CNI readiness (tracked as a follow-up); until +then, operators should ensure the CNI DaemonSet is Ready on every node that can +schedule sandbox pods before enabling the topology in production. + On OpenShift, binary-aware network policy also requires a purpose-built SecurityContextConstraints for sandbox pods: the network sidecar runs as UID 0 with `SYS_PTRACE` and `DAC_READ_SEARCH` to inspect cross-UID `/proc`, which diff --git a/crates/openshell-cni/src/lib.rs b/crates/openshell-cni/src/lib.rs index 27c2f8e1b7..92abbcb1f0 100644 --- a/crates/openshell-cni/src/lib.rs +++ b/crates/openshell-cni/src/lib.rs @@ -158,24 +158,58 @@ struct InstallReport { backend: &'static str, } +/// CNI spec reserves error codes >= 100 for plugin-specific failures. +const CNI_PLUGIN_ERROR_CODE: u32 = 100; + pub fn run() -> Result<()> { - let mut input = String::new(); - std::io::stdin() - .read_to_string(&mut input) - .into_diagnostic()?; let env = CniEnv::from_process(); - let runtime = Runtime; - let output = match handle_command(&input, &env, &runtime, &runtime) { - Ok(output) => output, + let mut input = String::new(); + let result = (|| -> Result> { + std::io::stdin() + .read_to_string(&mut input) + .into_diagnostic() + .wrap_err("failed to read CNI config from stdin")?; + let runtime = Runtime; + handle_command(&input, &env, &runtime, &runtime) + })(); + + match result { + Ok(output) => { + if let Some(output) = output { + println!("{}", serde_json::to_string(&output).into_diagnostic()?); + } + Ok(()) + } Err(error) => { log_cni_error(&input, &env, &error); - return Err(error); + // Per the CNI spec a failing plugin must print a structured error object + // on stdout and exit non-zero; the runtime parses this to surface the + // failure instead of treating stderr text as an opaque crash. + emit_cni_error(&input, &error); + Err(error) } - }; - if let Some(output) = output { - println!("{}", serde_json::to_string(&output).into_diagnostic()?); } - Ok(()) +} + +/// Builds the CNI-spec error object (`cniVersion`, `code`, `msg`, `details`). The +/// version echoes the request when parseable so the runtime accepts the reply. +fn cni_error_payload(input: &str, error: &miette::Report) -> Value { + let cni_version = serde_json::from_str::(input) + .ok() + .and_then(|config| config.cni_version) + .unwrap_or_else(|| DEFAULT_CNI_VERSION.to_string()); + serde_json::json!({ + "cniVersion": cni_version, + "code": CNI_PLUGIN_ERROR_CODE, + "msg": "OpenShell CNI plugin error", + "details": one_line_error(error), + }) +} + +fn emit_cni_error(input: &str, error: &miette::Report) { + if let Ok(serialized) = serde_json::to_string(&cni_error_payload(input, error)) { + println!("{serialized}"); + } } fn log_cni_error(input: &str, env: &CniEnv, error: &miette::Report) { @@ -672,13 +706,22 @@ struct IptablesBackend { #[cfg(target_os = "linux")] fn install_iptables_rules(netns: &Path, proxy_uid: u32, backend: &IptablesBackend) -> Result<()> { + // Fail closed on IPv6 first: without ip6tables we cannot prove IPv6 egress is + // blocked, so a dual-stack sandbox could bypass policy over IPv6. Refuse before + // installing any rules rather than leaving IPv4-only enforcement in place. The + // preferred nft backend always programs both families in one ruleset; this path + // only runs when nft is unavailable. + let ipv6 = backend.ipv6.as_deref().ok_or_else(|| { + miette::miette!( + "ip6tables not found on node; OpenShell CNI requires it to enforce IPv6 egress in the iptables fallback (install ip6tables or nft)" + ) + })?; + cleanup_iptables_family(netns, &backend.ipv4); install_iptables_family(netns, &backend.ipv4, proxy_uid, "icmp-port-unreachable")?; - if let Some(ipv6) = backend.ipv6.as_deref() { - cleanup_iptables_family(netns, ipv6); - install_iptables_family(netns, ipv6, proxy_uid, "icmp6-port-unreachable")?; - } + cleanup_iptables_family(netns, ipv6); + install_iptables_family(netns, ipv6, proxy_uid, "icmp6-port-unreachable")?; Ok(()) } @@ -1085,4 +1128,26 @@ mod tests { fn nft_search_path_includes_k3s_aux_path() { assert!(NFT_SEARCH_PATHS.contains(&"/bin/aux/nft")); } + + #[test] + fn cni_error_payload_is_spec_compliant() { + let error = miette::miette!("boom failure"); + let payload = cni_error_payload(&cni_input(), &error); + assert_eq!(payload["cniVersion"], "1.0.0"); + assert_eq!(payload["code"], serde_json::json!(CNI_PLUGIN_ERROR_CODE)); + assert_eq!(payload["msg"], "OpenShell CNI plugin error"); + assert!( + payload["details"] + .as_str() + .unwrap() + .contains("boom failure") + ); + } + + #[test] + fn cni_error_payload_defaults_cni_version_on_unparseable_input() { + let error = miette::miette!("boom"); + let payload = cni_error_payload("not valid json", &error); + assert_eq!(payload["cniVersion"], DEFAULT_CNI_VERSION); + } } diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index 7cc20fd65b..128b6025e7 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -1787,6 +1787,22 @@ fn supervisor_sidecar_container( } ] }); + // In the CNI-sidecar topology there is no network-init container to copy the + // gateway client mTLS bundle into the shared TLS emptyDir, because CNI-sidecar + // omits network-init (the node CNI plugin programs egress redirection instead). + // Mount the client TLS secret directly at the client subdir so the network + // supervisor can authenticate to the gateway (policy fetch, callbacks). The + // secret volume is provisioned by the base template with fsGroup-readable mode. + if topology == SupervisorTopology::CniSidecar && !params.client_tls_secret_name.is_empty() { + container["volumeMounts"] + .as_array_mut() + .expect("volumeMounts is an array") + .push(serde_json::json!({ + "name": CLIENT_TLS_VOLUME_NAME, + "mountPath": SIDECAR_CLIENT_TLS_MOUNT_PATH, + "readOnly": true + })); + } if !params.supervisor_image_pull_policy.is_empty() { container["imagePullPolicy"] = serde_json::json!(params.supervisor_image_pull_policy); } @@ -2467,9 +2483,15 @@ fn sandbox_template_to_k8s_with_validated_config( serde_json::Value::String(params.sandbox_id.to_string()), ); } + // Annotate the UID the sidecar proxy actually runs as so the node CNI + // plugin exempts the right identity from egress redirection. In + // binary-aware mode the sidecar runs as UID 0 (see + // effective_sidecar_proxy_uid); annotating the raw params.proxy_uid + // (e.g. 1337) would make the CNI redirect the sidecar's own egress and + // block it from reaching the gateway. pod_annotations.insert( OPENSHELL_CNI_PROXY_UID_ANNOTATION.to_string(), - serde_json::Value::String(params.proxy_uid.to_string()), + serde_json::Value::String(effective_sidecar_proxy_uid(params).to_string()), ); pod_annotations.insert( OPENSHELL_CNI_PROXY_PORT_ANNOTATION.to_string(), @@ -4612,6 +4634,9 @@ mod tests { proxy_uid: 2200, sandbox_uid: 1500, sandbox_gid: 1500, + // Non-binary-aware: the sidecar runs as proxy_uid, so the CNI + // proxy-uid annotation should reflect that raw UID. + process_binary_aware_network_policy: false, ..SandboxPodParams::default() }; let pod_template = sandbox_template_to_k8s( @@ -4682,6 +4707,131 @@ mod tests { ); } + /// Regression test for the cni-sidecar proxy-uid annotation. + /// + /// In binary-aware mode the network sidecar runs as UID 0 (see + /// `effective_sidecar_proxy_uid`), not `params.proxy_uid`. The CNI proxy-uid + /// annotation tells the node plugin which UID to exempt from egress + /// redirection; annotating the raw `params.proxy_uid` would make the CNI + /// redirect the sidecar's own egress and block it from reaching the gateway. + /// The annotation must reflect the effective (runAsUser) UID. + #[test] + fn cni_sidecar_binary_aware_annotates_effective_proxy_uid() { + let params = SandboxPodParams { + topology: SupervisorTopology::CniSidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-cni-ba", + proxy_uid: 1337, + sandbox_uid: 1500, + sandbox_gid: 1500, + process_binary_aware_network_policy: true, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + false, + ¶ms, + ); + + // The sidecar runs as UID 0 in binary-aware mode; the annotation must + // match so the node CNI exempts the sidecar's own egress. + let annotations = pod_template["metadata"]["annotations"].as_object().unwrap(); + assert_eq!( + annotations[OPENSHELL_CNI_PROXY_UID_ANNOTATION], + serde_json::json!(BINARY_AWARE_SIDECAR_PROXY_UID.to_string()) + ); + + let sidecar = pod_template["spec"]["containers"] + .as_array() + .unwrap() + .iter() + .find(|container| container["name"] == SUPERVISOR_NETWORK_SIDECAR_NAME) + .unwrap(); + assert_eq!( + sidecar["securityContext"]["runAsUser"], + serde_json::json!(BINARY_AWARE_SIDECAR_PROXY_UID) + ); + } + + /// Regression test for the cni-sidecar gateway client mTLS bundle. + /// + /// The cni-sidecar topology omits the network-init container (the node CNI + /// plugin programs egress redirection instead). In the standard sidecar + /// topology, network-init also copies the gateway client mTLS bundle into + /// the shared TLS emptyDir at `/client`; dropping network-init would + /// otherwise leave that path empty, so the network supervisor cannot read + /// its client cert and crash-loops on policy fetch. cni-sidecar must instead + /// mount the client TLS secret directly onto the network sidecar so the + /// bundle is present at the path the sidecar env points at. + #[test] + fn cni_sidecar_topology_mounts_client_tls_on_network_sidecar() { + let params = SandboxPodParams { + topology: SupervisorTopology::CniSidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-cni-tls", + proxy_uid: 2200, + sandbox_uid: 1500, + sandbox_gid: 1500, + client_tls_secret_name: "openshell-client-tls", + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + false, + ¶ms, + ); + + // The client TLS secret volume must be present in the pod spec. + let volumes = pod_template["spec"]["volumes"].as_array().unwrap(); + let client_tls_volume = volumes + .iter() + .find(|volume| volume["name"] == CLIENT_TLS_VOLUME_NAME) + .expect("client TLS secret volume must be provisioned"); + assert_eq!( + client_tls_volume["secret"]["secretName"], + serde_json::json!("openshell-client-tls") + ); + + // The network sidecar must mount that secret at the client subdir the + // sidecar env points OPENSHELL_TLS_CA/CERT/KEY at, read-only. + let containers = pod_template["spec"]["containers"].as_array().unwrap(); + let sidecar = containers + .iter() + .find(|container| container["name"] == SUPERVISOR_NETWORK_SIDECAR_NAME) + .unwrap(); + let mounts = sidecar["volumeMounts"].as_array().unwrap(); + let client_tls_mount = mounts + .iter() + .find(|mount| mount["name"] == CLIENT_TLS_VOLUME_NAME) + .expect("network sidecar must mount the client TLS secret"); + assert_eq!( + client_tls_mount["mountPath"], + serde_json::json!(SIDECAR_CLIENT_TLS_MOUNT_PATH) + ); + assert_eq!(client_tls_mount["readOnly"], serde_json::json!(true)); + + // The sidecar env must reference that same mount path so the mount and + // the consuming paths stay in lockstep. + assert_eq!( + rendered_env(sidecar, "OPENSHELL_TLS_CA"), + Some(format!("{SIDECAR_CLIENT_TLS_MOUNT_PATH}/ca.crt").as_str()) + ); + } + #[test] fn sidecar_topology_rejects_proxy_uid_matching_sandbox_uid() { let params = SandboxPodParams { diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index e8c3290d5c..259c679675 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -122,11 +122,14 @@ a repository-only override uses the effective gateway image tag. {{/* CNI installer image reference. Defaults to the supervisor image because the -supervisor image carries both openshell-sandbox and openshell-cni. +supervisor image carries both openshell-sandbox and openshell-cni. The +repository and tag fallbacks mirror openshell.supervisorImage so a +gateway-only tag override (image.tag) keeps the CNI image in lockstep with +the supervisor instead of silently pinning to the chart AppVersion. */}} {{- define "openshell.cniImage" -}} -{{- $repository := .Values.cni.image.repository | default .Values.supervisor.image.repository -}} -{{- $tag := .Values.cni.image.tag | default .Values.supervisor.image.tag | default .Chart.AppVersion -}} +{{- $repository := .Values.cni.image.repository | default .Values.supervisor.image.repository | default (include "openshell.defaultSupervisorRepository" .) -}} +{{- $tag := .Values.cni.image.tag | default .Values.supervisor.image.tag | default .Values.image.tag | default .Chart.AppVersion -}} {{- printf "%s:%s" $repository $tag }} {{- end }} diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml index 273e43bb66..6bc0b6c1c8 100644 --- a/deploy/helm/openshell/templates/cni-daemonset.yaml +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -100,23 +100,41 @@ spec: EOF {{- if eq $mode "conflist" }} conf_dir="/host{{ .Values.cni.confDir }}" + resolve_conflist_target() { {{- if .Values.cni.configFile }} - target="${conf_dir}/{{ .Values.cni.configFile }}" + printf '%s' "${conf_dir}/{{ .Values.cni.configFile }}" {{- else }} - target="$(find "${conf_dir}" -maxdepth 1 -type f -name '*.conflist' ! -name '*openshell*' | sort | head -n 1)" + find "${conf_dir}" -maxdepth 1 -type f -name '*.conflist' ! -name '*openshell*' | sort | head -n 1 {{- end }} - if [ -z "${target}" ] || [ ! -f "${target}" ]; then - echo "OpenShell CNI requires an existing CNI .conflist in ${conf_dir}" >&2 - exit 1 - fi - if [ ! -f "${target}.openshell-backup" ]; then - cp "${target}" "${target}.openshell-backup" - fi - tmp="$(mktemp "${conf_dir}/openshell-cni.XXXXXX")" - jq --slurpfile openshell "${plugin_config}" ' - .plugins = ((.plugins // []) | map(select(.type != "openshell-cni")) + [$openshell[0]]) - ' "${target}" > "${tmp}" - mv "${tmp}" "${target}" + } + patch_conflist() { + target="$(resolve_conflist_target)" + if [ -z "${target}" ] || [ ! -f "${target}" ]; then + echo "OpenShell CNI requires an existing CNI .conflist in ${conf_dir}" >&2 + return 1 + fi + if [ ! -f "${target}.openshell-backup" ]; then + cp "${target}" "${target}.openshell-backup" + fi + tmp="$(mktemp "${conf_dir}/openshell-cni.XXXXXX")" + jq --slurpfile openshell "${plugin_config}" ' + .plugins = ((.plugins // []) | map(select(.type != "openshell-cni")) + [$openshell[0]]) + ' "${target}" > "${tmp}" + mv "${tmp}" "${target}" + } + # Re-patch only when the chained plugin is absent, so a CNI config + # rewrite (CNO reconcile, node reimage) or a jq no-op cannot leave the + # node forwarding sandbox traffic without OpenShell egress enforcement. + ensure_conflist() { + target="$(resolve_conflist_target)" + if [ -n "${target}" ] && [ -f "${target}" ] \ + && jq -e 'any((.plugins // [])[]; .type == "openshell-cni")' "${target}" >/dev/null 2>&1; then + return 0 + fi + echo "OpenShell CNI re-patching ${target:-}; chained plugin missing (CNI config rewrite or restart)" >&2 + patch_conflist + } + patch_conflist || exit 1 echo "OpenShell CNI installed (conflist mode); plugin log ${log_file}; firewall backend selected during CNI ADD" {{- else }} chain_dir="/host{{ .Values.cni.chainDir }}" @@ -138,7 +156,9 @@ spec: while true; do cp /var/run/secrets/kubernetes.io/serviceaccount/token "${token_file}" chmod 0600 "${token_file}" - {{- if eq $mode "multus-chain" }} + {{- if eq $mode "conflist" }} + ensure_conflist || true + {{- else }} [ -f "${chain_conf}" ] || write_chain_conf {{- end }} sleep 300 diff --git a/deploy/helm/openshell/templates/deployment.yaml b/deploy/helm/openshell/templates/deployment.yaml index de9ae012df..e937979370 100644 --- a/deploy/helm/openshell/templates/deployment.yaml +++ b/deploy/helm/openshell/templates/deployment.yaml @@ -13,7 +13,6 @@ spec: selector: matchLabels: {{- include "openshell.selectorLabels" . | nindent 6 }} - app.kubernetes.io/component: gateway template: {{- include "openshell.gatewayPodTemplate" . | nindent 4 }} {{- end }} diff --git a/deploy/helm/openshell/templates/statefulset.yaml b/deploy/helm/openshell/templates/statefulset.yaml index 205f9a8daa..30571f80ba 100644 --- a/deploy/helm/openshell/templates/statefulset.yaml +++ b/deploy/helm/openshell/templates/statefulset.yaml @@ -14,7 +14,6 @@ spec: selector: matchLabels: {{- include "openshell.selectorLabels" . | nindent 6 }} - app.kubernetes.io/component: gateway template: {{- include "openshell.gatewayPodTemplate" . | nindent 4 }} volumeClaimTemplates: diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index 563942fe16..03009bb6e7 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -25,8 +25,12 @@ tests: - equal: path: kind value: StatefulSet - - equal: + # The component label must stay out of the immutable selector so existing + # releases can upgrade; it lives on the pod template instead. + - notExists: path: spec.selector.matchLabels["app.kubernetes.io/component"] + - equal: + path: spec.template.metadata.labels["app.kubernetes.io/component"] value: gateway - it: treats a null workload map as the default StatefulSet @@ -386,8 +390,12 @@ tests: - equal: path: spec.replicas value: 2 - - equal: + # The component label must stay out of the immutable selector so existing + # releases can upgrade; it lives on the pod template instead. + - notExists: path: spec.selector.matchLabels["app.kubernetes.io/component"] + - equal: + path: spec.template.metadata.labels["app.kubernetes.io/component"] value: gateway - equal: path: spec.template.spec.containers[0].name diff --git a/rfc/cni-sidecar-topology-DRAFT.md b/rfc/cni-sidecar-topology-DRAFT.md index 41157073bc..c1eba458b9 100644 --- a/rfc/cni-sidecar-topology-DRAFT.md +++ b/rfc/cni-sidecar-topology-DRAFT.md @@ -79,11 +79,6 @@ layer aligns the privilege boundary with how clusters are actually governed. default and the only topology that provides the full supervisor contract (filesystem policy, privilege drop, mount isolation). - **Non-Kubernetes drivers.** Docker, Podman, and VM drivers are unaffected. -- **New CNI plugin behavior.** The `openshell-cni` plugin binary itself is - reused as-is; this RFC is about how it is installed and permitted, not what it - does during `ADD`. -- **gVisor/Kata certification.** `cni-sidecar` is proposed as experimental with - those runtime classes as validation targets, not as a supported guarantee. ## Proposal From efe37f2f913279d7667dbfce14598ad4270c7ef8 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Mon, 3 Aug 2026 20:09:13 -0400 Subject: [PATCH 12/20] fix(cni-sidecar): close egress-enforcement gaps from re-review Address the second-round review of the cni-sidecar topology: - cni/driver: add a per-node readiness scheduling gate. The installer labels its node openshell.ai/cni-ready via a new `openshell-cni node-ready` subcommand (minimal cluster-scoped nodes get/patch RBAC), clearing it on shutdown or when a reconcile tick cannot restore the plugin; the driver sets a required nodeAffinity on that label for cni-sidecar sandbox pods, so a pod cannot schedule before per-node egress enforcement is active. - driver: run the untrusted workspace-init container as the sandbox UID under binary-aware policy instead of root, so it cannot inherit the CNI's UID-0 egress exemption and bypass policy before the sidecar enforces it. - helm: gate the conflist/chain-conf mv on jq success and non-empty output so a jq failure cannot truncate the host CNI config and break node networking. - helm: give the CNI DaemonSet a distinct app name (openshell-cni) and revert the gateway Service selector to base-only, so a helm upgrade no longer blanks gateway endpoints for pods that predate the component label. - cni: only require ip6tables in the iptables fallback when the pod has a routable IPv6 address (from prevResult), so IPv4-only nodes are not rejected. - build: add crates/openshell-cni/BUILD.bazel so Bazel and the aggregate rustfmt suite cover the crate. - docs: document the scheduling gate and corrected sidecar privilege profile across architecture, published docs, crate READMEs, values, AGENTS, and the debug-openshell-cluster skill. Signed-off-by: Russell Bryant --- .../skills/debug-openshell-cluster/SKILL.md | 9 + AGENTS.md | 1 + architecture/compute-runtimes.md | 21 +- crates/openshell-cni/BUILD.bazel | 41 +++ crates/openshell-cni/src/lib.rs | 253 ++++++++++++++++-- crates/openshell-cni/src/main.rs | 11 +- crates/openshell-driver-kubernetes/README.md | 19 +- .../openshell-driver-kubernetes/src/driver.rs | 242 ++++++++++++++++- deploy/helm/openshell/README.md | 4 +- deploy/helm/openshell/templates/_helpers.tpl | 27 ++ .../openshell/templates/cni-daemonset.yaml | 73 +++-- deploy/helm/openshell/templates/cni-rbac.yaml | 36 +++ deploy/helm/openshell/templates/service.yaml | 1 - .../openshell/tests/cni_daemonset_test.yaml | 31 +++ .../helm/openshell/tests/cni_rbac_test.yaml | 25 +- deploy/helm/openshell/tests/service_test.yaml | 10 +- deploy/helm/openshell/values.yaml | 4 +- docs/kubernetes/topology.mdx | 9 + 18 files changed, 754 insertions(+), 63 deletions(-) create mode 100644 crates/openshell-cni/BUILD.bazel diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index bbef67f855..d6bbf3c9ff 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -408,6 +408,15 @@ sandbox pod should carry `openshell.ai/cni=enabled`, network setup, inspect the DaemonSet logs, the host CNI config, and whether the cluster actually invokes chained CNI plugins for the sandbox runtime class. +A per-node scheduling gate can also keep cni-sidecar sandbox pods `Pending`. The +CNI DaemonSet labels each node `openshell.ai/cni-ready=true` after installing the +plugin, and the gateway sets a required `nodeAffinity` on that label. If a +sandbox pod stays `Pending` with an "unmatched nodeAffinity" event, check that +the CNI DaemonSet is Ready on schedulable nodes and that the label is present +(`kubectl get nodes -L openshell.ai/cni-ready`). A missing label means the +installer has not completed its first patch or a reconcile tick cleared it after +a plugin-restore failure. + Inspect all three when sandbox registration or egress enforcement fails: ```bash diff --git a/AGENTS.md b/AGENTS.md index 540a8d9207..44fa97c924 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,6 +33,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files | `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary | | `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing | | `crates/openshell-policy/` | Policy engine | Filesystem, network, process, and inference constraints | +| `crates/openshell-cni/` | Chained CNI plugin | Node-level egress bypass-prevention for the cni-sidecar topology; also sets the per-node readiness label | | `crates/openshell-router/` | Privacy router | Privacy-aware LLM routing | | `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage | | `crates/openshell-gateway-interceptors/` | Gateway interceptors | Intercepts and transforms configured gRPC requests at the gateway routing boundary | diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 8f61a66a45..f9b14d6551 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -265,14 +265,19 @@ persistent `stateDir`. Neither mode modifies a CNO-managed file. The installer patches the chained plugin at startup and then re-verifies it on a reconcile tick, re-patching when the plugin is missing. This keeps enforcement in place across CNI config rewrites (for example a CNO reconcile) and DaemonSet -restarts, bounding any such gap to one reconcile interval. It does not cover the -cold-start scheduling race: a sandbox pod scheduled on a node before that node's -CNI DaemonSet pod has completed its first patch is admitted with unrestricted -egress, because the cni-sidecar topology deliberately omits the in-pod -fail-closed network-init backstop. Closing that window requires gating sandbox -scheduling on verified per-node CNI readiness (tracked as a follow-up); until -then, operators should ensure the CNI DaemonSet is Ready on every node that can -schedule sandbox pods before enabling the topology in production. +restarts, bounding any such gap to one reconcile interval. + +A per-node scheduling gate closes the cold-start race. Once the chained plugin is +installed, the installer labels its node `openshell.ai/cni-ready=true`; it clears +the label on shutdown and whenever a reconcile tick cannot restore the plugin. +The gateway sets a required `nodeAffinity` on that label for every cni-sidecar +sandbox pod, so a pod cannot schedule onto a node before that node's egress +enforcement is active, and a node whose enforcement later breaks stops accepting +new sandbox pods. The label is set through a minimal cluster-scoped grant +(`nodes` `get`/`patch` only) bound to the dedicated CNI ServiceAccount. One +residual limitation: an ungraceful DaemonSet pod deletion (no `preStop`) leaves a +stale `cni-ready=true` until the pod is rescheduled and the next reconcile tick +re-evaluates it. On OpenShift, binary-aware network policy also requires a purpose-built SecurityContextConstraints for sandbox pods: the network sidecar runs as UID 0 diff --git a/crates/openshell-cni/BUILD.bazel b/crates/openshell-cni/BUILD.bazel new file mode 100644 index 0000000000..78705e644c --- /dev/null +++ b/crates/openshell-cni/BUILD.bazel @@ -0,0 +1,41 @@ +load("@crates//:defs.bzl", "aliases", "all_crate_deps") +load("@rules_rs//rs:rust_binary.bzl", "rust_binary") +load("@rules_rs//rs:rust_library.bzl", "rust_library") +load("@rules_rs//rs:rust_test.bzl", "rust_test") +load("@rules_rust//rust:defs.bzl", "rustfmt_test") + +rust_library( + name = "openshell-cni", + srcs = glob( + ["src/**/*.rs"], + exclude = ["src/main.rs"], + ), + aliases = aliases(), + visibility = ["//visibility:public"], + deps = all_crate_deps(normal = True), +) + +rust_binary( + name = "openshell-cni_bin", + srcs = ["src/main.rs"], + aliases = aliases(), + binary_name = "openshell-cni", + visibility = ["//visibility:public"], + deps = all_crate_deps(normal = True) + [":openshell-cni"], +) + +rust_test( + name = "openshell-cni_test", + crate = ":openshell-cni", + deps = all_crate_deps(normal_dev = True), +) + +rustfmt_test( + name = "rustfmt_test", + targets = [ + ":openshell-cni", + ":openshell-cni_bin", + ":openshell-cni_test", + ], + visibility = ["//crates:__pkg__"], +) diff --git a/crates/openshell-cni/src/lib.rs b/crates/openshell-cni/src/lib.rs index 92abbcb1f0..18e1e6c2b4 100644 --- a/crates/openshell-cni/src/lib.rs +++ b/crates/openshell-cni/src/lib.rs @@ -19,6 +19,12 @@ const OPENSHELL_CNI_PROXY_UID_ANNOTATION: &str = "openshell.ai/proxy-uid"; const OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION: &str = "openshell.ai/network-enforcement-mode"; const CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE: &str = "cni-sidecar"; +/// Node label set by the CNI installer once the chained plugin is in place and +/// cleared when it is not, so the gateway can gate sandbox scheduling on +/// per-node egress-enforcement readiness. Must stay in sync with the identical +/// constant in `openshell-driver-kubernetes` (used for the sandbox pod +/// nodeAffinity) and with the CNI `DaemonSet`'s node-patch RBAC. +const NODE_READY_LABEL: &str = "openshell.ai/cni-ready"; #[allow(dead_code)] const OPENSHELL_TABLE: &str = "openshell_sidecar_bypass"; #[allow(dead_code)] @@ -149,10 +155,37 @@ trait PodReader { } trait RuleInstaller { - fn install(&self, netns: &Path, proxy_uid: u32) -> Result; + fn install(&self, netns: &Path, proxy_uid: u32, enforce_ipv6: bool) -> Result; fn cleanup(&self, netns: &Path) -> Result<()>; } +/// Returns true when the chained CNI `prevResult` reports at least one routable +/// IPv6 address for the pod. Loopback and link-local (`fe80::/10`) addresses are +/// ignored because they cannot carry egress off the node, so a pod with only +/// those needs no IPv6 firewall enforcement. IPv4-only pods return false, which +/// lets the iptables fallback skip the (possibly absent) ip6tables binary. +fn prev_result_has_ipv6(config: &CniConfig) -> bool { + let Some(prev) = config.prev_result.as_ref() else { + return false; + }; + let Some(ips) = prev.get("ips").and_then(Value::as_array) else { + return false; + }; + ips.iter().any(|entry| { + entry + .get("address") + .and_then(Value::as_str) + .and_then(|addr| addr.split('/').next()) + .and_then(|ip| ip.parse::().ok()) + .is_some_and(|ip| match ip { + std::net::IpAddr::V6(v6) => { + !v6.is_loopback() && (v6.segments()[0] & 0xffc0) != 0xfe80 + } + std::net::IpAddr::V4(_) => false, + }) + }) +} + #[derive(Debug, Clone, Copy, PartialEq, Eq)] struct InstallReport { backend: &'static str, @@ -161,6 +194,37 @@ struct InstallReport { /// CNI spec reserves error codes >= 100 for plugin-specific failures. const CNI_PLUGIN_ERROR_CODE: u32 = 100; +/// Parses the `node-ready` subcommand flags into the desired readiness state. +/// `--set` marks the node ready; `--clear` fences it. +fn parse_node_ready_args(args: &[String]) -> Result { + let mut ready = None; + for arg in args { + match arg.as_str() { + "--set" => ready = Some(true), + "--clear" => ready = Some(false), + other => return Err(miette::miette!("unknown node-ready argument '{other}'")), + } + } + ready.ok_or_else(|| miette::miette!("node-ready requires --set or --clear")) +} + +/// Entry point for `openshell-cni node-ready`, the CNI installer's readiness gate. +/// +/// Invoked by the installer `DaemonSet` to gate sandbox scheduling on per-node +/// enforcement readiness. `--set` labels the node ready once the chained plugin +/// is installed; `--clear` removes the label on shutdown or repair failure so +/// new sandbox pods will not schedule on a node where egress enforcement is +/// absent. Reads the target node from `NODE_NAME` and authenticates with the +/// pod's in-cluster service account. +pub fn node_ready(args: &[String]) -> Result<()> { + let ready = parse_node_ready_args(args)?; + let node = std::env::var("NODE_NAME") + .map_err(|_| miette::miette!("NODE_NAME env var is required for node-ready"))?; + let client = KubeApiClient::from_in_cluster()?; + let value = if ready { Some("true") } else { None }; + client.patch_node_label(&node, NODE_READY_LABEL, value) +} + pub fn run() -> Result<()> { let env = CniEnv::from_process(); let mut input = String::new(); @@ -270,7 +334,8 @@ fn handle_command( let netns = env.netns.as_deref().ok_or_else(|| { miette::miette!("CNI_NETNS is required for OpenShell CNI ADD") })?; - let report = installer.install(netns, workload.proxy_uid)?; + let report = + installer.install(netns, workload.proxy_uid, prev_result_has_ipv6(&config))?; log_cni_info( &config, env, @@ -288,7 +353,8 @@ fn handle_command( let netns = env.netns.as_deref().ok_or_else(|| { miette::miette!("CNI_NETNS is required for OpenShell CNI CHECK") })?; - let report = installer.install(netns, workload.proxy_uid)?; + let report = + installer.install(netns, workload.proxy_uid, prev_result_has_ipv6(&config))?; log_cni_info( &config, env, @@ -455,6 +521,66 @@ impl KubeApiClient { }) } + /// Builds a client from the pod's mounted in-cluster service account, + /// independent of the plugin kubeconfig (whose token-file path is a host + /// path that does not resolve inside the installer container). + fn from_in_cluster() -> Result { + const TOKEN_PATH: &str = "/var/run/secrets/kubernetes.io/serviceaccount/token"; + const CA_PATH: &str = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"; + let host = std::env::var("KUBERNETES_SERVICE_HOST").map_err(|_| { + miette::miette!("KUBERNETES_SERVICE_HOST is required for in-cluster access") + })?; + let port = std::env::var("KUBERNETES_SERVICE_PORT").map_err(|_| { + miette::miette!("KUBERNETES_SERVICE_PORT is required for in-cluster access") + })?; + let token = std::fs::read_to_string(TOKEN_PATH) + .into_diagnostic() + .wrap_err("failed to read in-cluster service account token")? + .trim() + .to_string(); + let ca_pem = std::fs::read(CA_PATH) + .into_diagnostic() + .wrap_err("failed to read in-cluster CA certificate")?; + let ca = reqwest::Certificate::from_pem(&ca_pem).into_diagnostic()?; + let client = reqwest::blocking::Client::builder() + .add_root_certificate(ca) + .build() + .into_diagnostic()?; + Ok(Self { + server: format!("https://{host}:{port}"), + token, + client, + }) + } + + /// Sets (`value = Some`) or removes (`value = None`) a single node label via + /// a JSON merge patch. A null value in a merge patch deletes the key. + fn patch_node_label(&self, node: &str, key: &str, value: Option<&str>) -> Result<()> { + let url = format!("{}/api/v1/nodes/{}", self.server, node); + let label_value = value.map_or(Value::Null, |v| Value::String(v.to_string())); + let body = serde_json::json!({ "metadata": { "labels": { key: label_value } } }); + let response = self + .client + .patch(url) + .bearer_auth(&self.token) + .header( + reqwest::header::CONTENT_TYPE, + "application/merge-patch+json", + ) + .body(serde_json::to_vec(&body).into_diagnostic()?) + .send() + .into_diagnostic() + .wrap_err("failed to patch node label")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while patching node {}", + response.status(), + node + )); + } + Ok(()) + } + fn pod_annotations(&self, pod: &PodRef) -> Result> { let url = format!( "{}/api/v1/namespaces/{}/pods/{}", @@ -511,8 +637,8 @@ fn cluster_certificate_authority( } impl RuleInstaller for Runtime { - fn install(&self, netns: &Path, proxy_uid: u32) -> Result { - install_rules(netns, proxy_uid) + fn install(&self, netns: &Path, proxy_uid: u32, enforce_ipv6: bool) -> Result { + install_rules(netns, proxy_uid, enforce_ipv6) } fn cleanup(&self, netns: &Path) -> Result<()> { @@ -556,7 +682,11 @@ fn generate_sidecar_bypass_ruleset(proxy_uid: u32, log_prefix: Option<&str>) -> } #[cfg(target_os = "linux")] -fn install_rules(netns: &Path, proxy_uid: u32) -> Result { +fn install_rules(netns: &Path, proxy_uid: u32, enforce_ipv6: bool) -> Result { + // The preferred nft backend programs both families in one inet ruleset, so + // its IPv6 reject rules are harmless no-ops on IPv4-only pods; enforce_ipv6 + // only gates the iptables fallback, where a missing ip6tables binary would + // otherwise reject an IPv4-only node outright. let nft_error = if let Some(nft) = find_nft() { match install_nft_rules(netns, proxy_uid, &nft) { Ok(()) => { @@ -569,7 +699,8 @@ fn install_rules(netns: &Path, proxy_uid: u32) -> Result { }; if let Some(iptables) = find_iptables() { - install_iptables_rules(netns, proxy_uid, &iptables).wrap_err("iptables fallback failed")?; + install_iptables_rules(netns, proxy_uid, &iptables, enforce_ipv6) + .wrap_err("iptables fallback failed")?; return Ok(InstallReport { backend: "iptables", }); @@ -594,8 +725,8 @@ fn install_nft_rules(netns: &Path, proxy_uid: u32, nft: &str) -> Result<()> { } #[cfg(not(target_os = "linux"))] -fn install_rules(netns: &Path, proxy_uid: u32) -> Result { - let _ = (netns, proxy_uid); +fn install_rules(netns: &Path, proxy_uid: u32, enforce_ipv6: bool) -> Result { + let _ = (netns, proxy_uid, enforce_ipv6); Err(miette::miette!( "OpenShell CNI rule installation is supported only on Linux nodes" )) @@ -705,23 +836,28 @@ struct IptablesBackend { } #[cfg(target_os = "linux")] -fn install_iptables_rules(netns: &Path, proxy_uid: u32, backend: &IptablesBackend) -> Result<()> { - // Fail closed on IPv6 first: without ip6tables we cannot prove IPv6 egress is - // blocked, so a dual-stack sandbox could bypass policy over IPv6. Refuse before - // installing any rules rather than leaving IPv4-only enforcement in place. The - // preferred nft backend always programs both families in one ruleset; this path - // only runs when nft is unavailable. - let ipv6 = backend.ipv6.as_deref().ok_or_else(|| { - miette::miette!( - "ip6tables not found on node; OpenShell CNI requires it to enforce IPv6 egress in the iptables fallback (install ip6tables or nft)" - ) - })?; - +fn install_iptables_rules( + netns: &Path, + proxy_uid: u32, + backend: &IptablesBackend, + enforce_ipv6: bool, +) -> Result<()> { cleanup_iptables_family(netns, &backend.ipv4); install_iptables_family(netns, &backend.ipv4, proxy_uid, "icmp-port-unreachable")?; - cleanup_iptables_family(netns, ipv6); - install_iptables_family(netns, ipv6, proxy_uid, "icmp6-port-unreachable")?; + if enforce_ipv6 { + // The pod has a routable IPv6 address, so unenforced IPv6 would be a + // policy bypass. Fail closed if ip6tables is missing rather than leaving + // IPv4-only enforcement in place. IPv4-only pods skip this entirely so + // nodes without ip6tables still work. + let ipv6 = backend.ipv6.as_deref().ok_or_else(|| { + miette::miette!( + "pod has IPv6 connectivity but ip6tables was not found on node; OpenShell CNI requires it to enforce IPv6 egress in the iptables fallback (install ip6tables or nft)" + ) + })?; + cleanup_iptables_family(netns, ipv6); + install_iptables_family(netns, ipv6, proxy_uid, "icmp6-port-unreachable")?; + } Ok(()) } @@ -890,7 +1026,12 @@ mod tests { } impl RuleInstaller for TestInstaller { - fn install(&self, _netns: &Path, proxy_uid: u32) -> Result { + fn install( + &self, + _netns: &Path, + proxy_uid: u32, + _enforce_ipv6: bool, + ) -> Result { self.installed.lock().unwrap().push(proxy_uid); Ok(InstallReport { backend: "test" }) } @@ -1150,4 +1291,68 @@ mod tests { let payload = cni_error_payload("not valid json", &error); assert_eq!(payload["cniVersion"], DEFAULT_CNI_VERSION); } + + fn config_with_prev_result(prev_result: Value) -> CniConfig { + CniConfig { + cni_version: Some("1.0.0".to_string()), + prev_result: Some(prev_result), + openshell: OpenShellConfig::default(), + } + } + + #[test] + fn prev_result_has_ipv6_true_for_routable_ipv6() { + let config = config_with_prev_result(serde_json::json!({ + "ips": [ + { "address": "10.1.2.3/24" }, + { "address": "fd00:10:244::5/64" } + ] + })); + assert!(prev_result_has_ipv6(&config)); + } + + #[test] + fn prev_result_has_ipv6_false_for_ipv4_only() { + let config = config_with_prev_result(serde_json::json!({ + "ips": [{ "address": "10.1.2.3/24" }] + })); + assert!(!prev_result_has_ipv6(&config)); + } + + #[test] + fn prev_result_has_ipv6_ignores_link_local_and_loopback() { + let config = config_with_prev_result(serde_json::json!({ + "ips": [ + { "address": "fe80::1/64" }, + { "address": "::1/128" } + ] + })); + assert!(!prev_result_has_ipv6(&config)); + } + + #[test] + fn prev_result_has_ipv6_false_when_prev_result_absent() { + let config = CniConfig { + cni_version: Some("1.0.0".to_string()), + prev_result: None, + openshell: OpenShellConfig::default(), + }; + assert!(!prev_result_has_ipv6(&config)); + } + + #[test] + fn parse_node_ready_args_set_and_clear() { + assert!(parse_node_ready_args(&["--set".to_string()]).unwrap()); + assert!(!parse_node_ready_args(&["--clear".to_string()]).unwrap()); + } + + #[test] + fn parse_node_ready_args_requires_a_flag() { + assert!(parse_node_ready_args(&[]).is_err()); + } + + #[test] + fn parse_node_ready_args_rejects_unknown_flag() { + assert!(parse_node_ready_args(&["--bogus".to_string()]).is_err()); + } } diff --git a/crates/openshell-cni/src/main.rs b/crates/openshell-cni/src/main.rs index ab22358268..b30589b13f 100644 --- a/crates/openshell-cni/src/main.rs +++ b/crates/openshell-cni/src/main.rs @@ -2,7 +2,16 @@ // SPDX-License-Identifier: Apache-2.0 fn main() { - if let Err(error) = openshell_cni::run() { + // The container runtime invokes the CNI plugin with no positional args + // (config on stdin, CNI_COMMAND in env). The `node-ready` subcommand is only + // reached when the installer DaemonSet calls the binary explicitly. + let args: Vec = std::env::args().collect(); + let result = if args.get(1).map(String::as_str) == Some("node-ready") { + openshell_cni::node_ready(&args[2..]) + } else { + openshell_cni::run() + }; + if let Err(error) = result { eprintln!("{error:?}"); std::process::exit(1); } diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 9b03707d4e..b9668e88ee 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -99,11 +99,20 @@ supervisors exit if the control connection closes, coupling their container restart lifecycle before a new authoritative client can be established. The `cni-sidecar` supervisor topology keeps the sidecar runtime model, but -removes the pod-local network init container. The driver annotates sandbox pods -for the OpenShell chained CNI plugin, and the privileged OpenShell CNI -DaemonSet installs the sidecar bypass-prevention rules during CNI `ADD` before -the workload starts. The agent container and long-running network sidecar remain -non-root with no added Linux capabilities. +removes the pod-local network init container and its `NET_ADMIN`. The driver +annotates sandbox pods for the OpenShell chained CNI plugin, and the privileged +OpenShell CNI DaemonSet installs the sidecar bypass-prevention rules during CNI +`ADD` before the workload starts. Because the node CNI programs the firewall, no +container in the sandbox pod holds `NET_ADMIN`. The network sidecar otherwise +keeps the same privilege profile as the other sidecar topologies: in the default +binary-aware mode it runs as UID 0 with `SYS_PTRACE` and `DAC_READ_SEARCH` to +resolve cross-UID `/proc`, and is non-root with no added capabilities only when +`process_binary_aware_network_policy` is disabled. The agent container is non-root +with no added Linux capabilities in either mode. + +The driver also sets a required `nodeAffinity` on the CNI installer's +`openshell.ai/cni-ready` node label for cni-sidecar pods, so a sandbox cannot +schedule onto a node before that node's chained plugin is active. Sidecar and cni-sidecar modes use the pod `fsGroup` to make the projected service-account token and sandbox client TLS secret group-readable so the diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index 128b6025e7..b8cbadb2d7 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -1454,6 +1454,12 @@ const OPENSHELL_CNI_PROXY_PORT_ANNOTATION: &str = "openshell.ai/proxy-port"; const OPENSHELL_CNI_NETWORK_ENFORCEMENT_MODE_ANNOTATION: &str = "openshell.ai/network-enforcement-mode"; const CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE: &str = "cni-sidecar"; +/// Node label the CNI installer sets once per-node egress enforcement is in +/// place. Sandbox pods in the cni-sidecar topology require it via nodeAffinity +/// so they cannot schedule onto a node before the chained plugin is active +/// (closing the cold-start fail-open window). Must stay in sync with +/// `NODE_READY_LABEL` in `openshell-cni` and the CNI `DaemonSet`'s node-patch RBAC. +const OPENSHELL_CNI_READY_NODE_LABEL: &str = "openshell.ai/cni-ready"; /// Build the emptyDir volume that holds the supervisor binary. /// @@ -2065,6 +2071,7 @@ fn apply_workspace_persistence( image: &str, image_pull_policy: &str, sandbox_gid: u32, + workspace_init_uid: u32, ) { let Some(spec) = pod_template.get_mut("spec").and_then(|v| v.as_object_mut()) else { return; @@ -2141,12 +2148,19 @@ fn apply_workspace_persistence( fi" ); + // The workspace-init container runs the user-selected (untrusted) agent + // image. Running it as UID 0 is a policy-bypass risk under binary-aware + // network policy, where the CNI firewall exempts UID 0 so the network + // sidecar can reach the gateway: an init running as root would inherit + // that exemption and egress freely before the sidecar enforces policy. + // Callers pass the sandbox UID in that mode (non-exempt, still able to + // seed the PVC via fsGroup) and 0 otherwise (where UID 0 is not exempt). let mut init_spec = serde_json::json!({ "name": WORKSPACE_INIT_CONTAINER_NAME, "image": image, "command": ["sh", "-c", copy_cmd], "securityContext": { - "runAsUser": 0, + "runAsUser": workspace_init_uid, }, "volumeMounts": [{ "name": WORKSPACE_VOLUME_NAME, @@ -2533,6 +2547,14 @@ fn sandbox_template_to_k8s_with_validated_config( } apply_pod_driver_config(&mut spec, &driver_config.pod); + // In the cni-sidecar topology, egress enforcement is programmed by the node + // CNI plugin, not an in-pod init container. Require the CNI installer's + // node-ready label so a sandbox pod cannot schedule onto a node before that + // node's chained plugin is active — closing the cold-start fail-open window. + if params.topology == SupervisorTopology::CniSidecar { + apply_cni_ready_node_affinity(&mut spec); + } + // Per-sandbox platform_config.host_users overrides the cluster-wide default. let use_user_namespaces = platform_config_bool(template, "host_users") .map_or(params.enable_user_namespaces, |host_users| !host_users); @@ -2776,17 +2798,74 @@ fn sandbox_template_to_k8s_with_validated_config( // that /sandbox data survives pod rescheduling. Skipped when the user // provides custom storage through driver_config. if inject_workspace { + // Under binary-aware network policy the CNI firewall exempts UID 0, so + // the untrusted workspace-init must not run as root or it could bypass + // egress policy. Run it as the (non-exempt) sandbox UID in that mode; + // otherwise UID 0 is already fenced and root keeps the widest read + // access to the image's /sandbox contents. + let workspace_init_uid = if params.process_binary_aware_network_policy { + params.sandbox_uid + } else { + 0 + }; apply_workspace_persistence( &mut result, image, params.image_pull_policy, params.sandbox_gid, + workspace_init_uid, ); } result } +/// Adds a required nodeAffinity term that gates scheduling on the CNI +/// installer's node-ready label. The requirement is added to every existing +/// nodeSelectorTerm (or a fresh one when none exist) — a match expression is +/// conjunctive within a term — so it composes with any operator-supplied +/// affinity without weakening it. `IgnoredDuringExecution` matches upstream +/// behavior: it gates placement, not eviction of running pods. +fn apply_cni_ready_node_affinity(spec: &mut serde_json::Map) { + let requirement = serde_json::json!({ + "key": OPENSHELL_CNI_READY_NODE_LABEL, + "operator": "In", + "values": ["true"], + }); + + let terms = spec + .entry("affinity") + .or_insert_with(|| serde_json::json!({})) + .as_object_mut() + .expect("affinity is an object") + .entry("nodeAffinity") + .or_insert_with(|| serde_json::json!({})) + .as_object_mut() + .expect("nodeAffinity is an object") + .entry("requiredDuringSchedulingIgnoredDuringExecution") + .or_insert_with(|| serde_json::json!({ "nodeSelectorTerms": [] })) + .as_object_mut() + .expect("nodeSelector is an object") + .entry("nodeSelectorTerms") + .or_insert_with(|| serde_json::json!([])) + .as_array_mut() + .expect("nodeSelectorTerms is an array"); + + if terms.is_empty() { + terms.push(serde_json::json!({ "matchExpressions": [requirement] })); + return; + } + for term in terms.iter_mut() { + term.as_object_mut() + .expect("nodeSelectorTerm is an object") + .entry("matchExpressions") + .or_insert_with(|| serde_json::json!([])) + .as_array_mut() + .expect("matchExpressions is an array") + .push(requirement.clone()); + } +} + fn apply_pod_driver_config( spec: &mut serde_json::Map, config: &KubernetesPodDriverConfig, @@ -4832,6 +4911,163 @@ mod tests { ); } + /// Regression test for the cold-start scheduling gate. + /// + /// cni-sidecar pods must carry a required nodeAffinity on the CNI installer's + /// node-ready label so they cannot land on a node before that node's chained + /// plugin is active. + #[test] + fn cni_sidecar_requires_cni_ready_node_affinity() { + let params = SandboxPodParams { + topology: SupervisorTopology::CniSidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-cni-gate", + proxy_uid: 1337, + sandbox_uid: 1500, + sandbox_gid: 1500, + process_binary_aware_network_policy: true, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + false, + ¶ms, + ); + + let terms = &pod_template["spec"]["affinity"]["nodeAffinity"]["requiredDuringSchedulingIgnoredDuringExecution"] + ["nodeSelectorTerms"]; + let expressions = terms[0]["matchExpressions"].as_array().unwrap(); + let requirement = expressions + .iter() + .find(|expr| expr["key"] == OPENSHELL_CNI_READY_NODE_LABEL) + .expect("cni-sidecar pod must require the cni-ready node label"); + assert_eq!(requirement["operator"], "In"); + assert_eq!(requirement["values"], serde_json::json!(["true"])); + } + + /// The scheduling gate is cni-sidecar-specific; other topologies enforce in + /// pod and must not carry the node-ready affinity. + #[test] + fn non_cni_sidecar_has_no_cni_ready_node_affinity() { + let params = SandboxPodParams { + topology: SupervisorTopology::Sidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-no-gate", + proxy_uid: 2200, + sandbox_uid: 1500, + sandbox_gid: 1500, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + false, + ¶ms, + ); + assert!( + pod_template["spec"]["affinity"]["nodeAffinity"].is_null(), + "non-cni-sidecar pods must not carry cni-ready nodeAffinity" + ); + } + + /// Regression test for the workspace-init policy-bypass hole. + /// + /// The workspace-init container runs the untrusted user image. Under + /// binary-aware network policy the CNI firewall exempts UID 0 so the network + /// sidecar can reach the gateway; an init running as root would inherit that + /// exemption and egress before policy is enforced. It must run as the + /// non-exempt sandbox UID instead. + #[test] + fn binary_aware_workspace_init_runs_as_sandbox_uid() { + let params = SandboxPodParams { + topology: SupervisorTopology::CniSidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-cni-init", + proxy_uid: 1337, + sandbox_uid: 1_000_790_000, + sandbox_gid: 1_000_790_000, + process_binary_aware_network_policy: true, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + true, // inject_workspace + ¶ms, + ); + + let init = pod_template["spec"]["initContainers"] + .as_array() + .unwrap() + .iter() + .find(|container| container["name"] == WORKSPACE_INIT_CONTAINER_NAME) + .expect("workspace-init container must be present"); + assert_eq!( + init["securityContext"]["runAsUser"], + serde_json::json!(1_000_790_000), + "workspace-init must run as the non-exempt sandbox UID in binary-aware mode" + ); + } + + /// Companion: without binary-aware policy, UID 0 is not exempt, so the init + /// keeps root for the widest read access to the image's /sandbox contents. + #[test] + fn non_binary_aware_workspace_init_runs_as_root() { + let params = SandboxPodParams { + topology: SupervisorTopology::Sidecar, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-init-root", + proxy_uid: 2200, + sandbox_uid: 1500, + sandbox_gid: 1500, + process_binary_aware_network_policy: false, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + true, // inject_workspace + ¶ms, + ); + + let init = pod_template["spec"]["initContainers"] + .as_array() + .unwrap() + .iter() + .find(|container| container["name"] == WORKSPACE_INIT_CONTAINER_NAME) + .expect("workspace-init container must be present"); + assert_eq!( + init["securityContext"]["runAsUser"], + serde_json::json!(0), + "workspace-init keeps root when UID 0 is not CNI-exempt" + ); + } + #[test] fn sidecar_topology_rejects_proxy_uid_matching_sandbox_uid() { let params = SandboxPodParams { @@ -5329,6 +5565,7 @@ mod tests { "openshell/sandbox:latest", "IfNotPresent", 1000, // sandbox_gid + 0, // workspace_init_uid (non-binary-aware: root) ); // Init container @@ -5388,6 +5625,7 @@ mod tests { "my-custom-image:v2", "IfNotPresent", 1000, + 0, ); let init_image = pod_template["spec"]["initContainers"][0]["image"] @@ -5410,7 +5648,7 @@ mod tests { } }); - apply_workspace_persistence(&mut pod_template, "img:latest", "Always", 1000); + apply_workspace_persistence(&mut pod_template, "img:latest", "Always", 1000, 0); let cmd = pod_template["spec"]["initContainers"][0]["command"] .as_array() diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 6fbb7e0abb..5d8f8870de 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -155,8 +155,8 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | cni.configFile | string | `""` | Host CNI conflist filename patched by the installer. Empty selects the first non-OpenShell .conflist. | | cni.enabled | bool | `false` | Install the OpenShell chained CNI plugin with a privileged node DaemonSet. Required when supervisor.topology is "cni-sidecar". | | cni.image.pullPolicy | string | `""` | CNI installer image pull policy. Empty uses supervisor.image.pullPolicy, then image.pullPolicy. | -| cni.image.repository | string | `""` | CNI installer image repository. Empty uses supervisor.image.repository. | -| cni.image.tag | string | `""` | CNI installer image tag. Empty uses supervisor.image.tag, then chart appVersion. | +| cni.image.repository | string | `""` | CNI installer image repository. Empty falls back to supervisor.image.repository, then the official supervisor repository. | +| cni.image.tag | string | `""` | CNI installer image tag. Empty falls back to supervisor.image.tag, then image.tag, then chart appVersion. | | cni.logFile | string | `"/var/log/openshell-cni.log"` | Host log file written by the OpenShell CNI plugin and tailed by the installer DaemonSet. | | cni.logLevel | string | `"info"` | Log level passed to the OpenShell CNI plugin. | | cni.mode | string | `"conflist"` | CNI installer strategy. "conflist" appends the OpenShell plugin to an existing CNI .conflist (k3s / vanilla). "multus-chain" writes a standalone plugin .conf into a Multus vendor-cni-chain subdirectory (OpenShift), which never modifies a CNO-managed file. | diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 259c679675..93a572ecbb 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -48,6 +48,33 @@ app.kubernetes.io/name: {{ include "openshell.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} {{- end }} +{{/* +CNI DaemonSet selector labels. The CNI installer runs a distinct +`app.kubernetes.io/name` (`-cni`) so it does NOT share the gateway's base +selector labels. This lets the gateway Service and workload selectors stay +base-only (name + instance) without also matching CNI pods — which in turn +avoids requiring an immutable-selector or component-label change on existing +gateway releases during upgrade. +*/}} +{{- define "openshell.cniSelectorLabels" -}} +app.kubernetes.io/name: {{ include "openshell.name" . }}-cni +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Full label set for the CNI DaemonSet and its pods, mirroring openshell.labels +but keyed on the distinct CNI selector labels. +*/}} +{{- define "openshell.cniLabels" -}} +helm.sh/chart: {{ include "openshell.chart" . }} +{{ include "openshell.cniSelectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +app.kubernetes.io/component: cni +{{- end }} + {{/* Create the name of the service account to use */}} diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml index 6bc0b6c1c8..133d273167 100644 --- a/deploy/helm/openshell/templates/cni-daemonset.yaml +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -24,18 +24,15 @@ kind: DaemonSet metadata: name: {{ include "openshell.fullname" . }}-cni labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni + {{- include "openshell.cniLabels" . | nindent 4 }} spec: selector: matchLabels: - {{- include "openshell.selectorLabels" . | nindent 6 }} - app.kubernetes.io/component: cni + {{- include "openshell.cniSelectorLabels" . | nindent 6 }} template: metadata: labels: - {{- include "openshell.selectorLabels" . | nindent 8 }} - app.kubernetes.io/component: cni + {{- include "openshell.cniLabels" . | nindent 8 }} spec: hostNetwork: true dnsPolicy: ClusterFirstWithHostNet @@ -48,6 +45,13 @@ spec: - name: install-cni image: {{ include "openshell.cniImage" . | quote }} imagePullPolicy: {{ include "openshell.cniImagePullPolicy" . | quote }} + env: + # Used by `openshell-cni node-ready` to label this node ready/unready + # so the gateway can gate sandbox scheduling on per-node enforcement. + - name: NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName command: - sh - -ec @@ -98,6 +102,20 @@ spec: } } EOF + # Per-node readiness gate. The gateway sets a required nodeAffinity + # on sandbox pods for this label, so a node is only eligible once + # the chained plugin is installed. mark_ready/mark_unready are + # best-effort: a failed set fails closed (pods keep avoiding the + # node); a failed clear is retried on the next reconcile tick. + openshell_cni="/openshell-cni" + mark_ready() { + "${openshell_cni}" node-ready --set \ + || echo "OpenShell CNI: failed to mark node ready" >&2 + } + mark_unready() { + "${openshell_cni}" node-ready --clear \ + || echo "OpenShell CNI: failed to mark node unready" >&2 + } {{- if eq $mode "conflist" }} conf_dir="/host{{ .Values.cni.confDir }}" resolve_conflist_target() { @@ -117,10 +135,20 @@ spec: cp "${target}" "${target}.openshell-backup" fi tmp="$(mktemp "${conf_dir}/openshell-cni.XXXXXX")" - jq --slurpfile openshell "${plugin_config}" ' + # Only replace the live conflist when jq succeeded AND wrote a + # non-empty file. This function is invoked through `||`, which + # disables `set -e` inside it, so a bare `jq; mv` would clobber a + # working host conflist with an empty file on jq failure and break + # node networking. Fail closed instead: leave the target intact. + if jq --slurpfile openshell "${plugin_config}" ' .plugins = ((.plugins // []) | map(select(.type != "openshell-cni")) + [$openshell[0]]) - ' "${target}" > "${tmp}" - mv "${tmp}" "${target}" + ' "${target}" > "${tmp}" && [ -s "${tmp}" ]; then + mv "${tmp}" "${target}" + else + rm -f "${tmp}" + echo "OpenShell CNI conflist patch failed (jq error); left ${target} unchanged" >&2 + return 1 + fi } # Re-patch only when the chained plugin is absent, so a CNI config # rewrite (CNO reconcile, node reimage) or a jq no-op cannot leave the @@ -135,6 +163,7 @@ spec: patch_conflist } patch_conflist || exit 1 + mark_ready echo "OpenShell CNI installed (conflist mode); plugin log ${log_file}; firewall backend selected during CNI ADD" {{- else }} chain_dir="/host{{ .Values.cni.chainDir }}" @@ -142,11 +171,19 @@ spec: mkdir -p "${chain_dir}" write_chain_conf() { _tmp="$(mktemp "${chain_dir}/openshell-cni.XXXXXX")" - jq -n --slurpfile openshell "${plugin_config}" \ - '{"cniVersion": "1.0.0", "name": "openshell-cni"} + $openshell[0]' > "${_tmp}" - mv "${_tmp}" "${chain_conf}" + # Same fail-closed contract as patch_conflist: never install a + # truncated chain config if jq fails. + if jq -n --slurpfile openshell "${plugin_config}" \ + '{"cniVersion": "1.0.0", "name": "openshell-cni"} + $openshell[0]' > "${_tmp}" && [ -s "${_tmp}" ]; then + mv "${_tmp}" "${chain_conf}" + else + rm -f "${_tmp}" + echo "OpenShell CNI chain conf generation failed (jq error); left ${chain_conf} unchanged" >&2 + return 1 + fi } - write_chain_conf + write_chain_conf || exit 1 + mark_ready echo "OpenShell CNI installed (multus-chain mode) at ${chain_conf}; plugin log ${log_file}; firewall backend selected during CNI ADD" {{- end }} tail -n 0 -F "${host_log_file}" & @@ -157,9 +194,12 @@ spec: cp /var/run/secrets/kubernetes.io/serviceaccount/token "${token_file}" chmod 0600 "${token_file}" {{- if eq $mode "conflist" }} - ensure_conflist || true + # Keep the node's readiness label in lockstep with enforcement: + # mark unready if the chained plugin cannot be restored, so no new + # sandbox pods schedule here until it is. + if ensure_conflist; then mark_ready; else mark_unready; fi {{- else }} - [ -f "${chain_conf}" ] || write_chain_conf + if { [ -f "${chain_conf}" ] || write_chain_conf; }; then mark_ready; else mark_unready; fi {{- end }} sleep 300 done @@ -173,6 +213,9 @@ spec: - sh - -ec - | + # Fence the node before removing enforcement so no sandbox pod + # schedules here during teardown. + /openshell-cni node-ready --clear || true {{- if eq $mode "conflist" }} conf_dir="/host{{ .Values.cni.confDir }}" for target in "${conf_dir}"/*.conflist; do diff --git a/deploy/helm/openshell/templates/cni-rbac.yaml b/deploy/helm/openshell/templates/cni-rbac.yaml index 468b27b2e0..b81cac83e1 100644 --- a/deploy/helm/openshell/templates/cni-rbac.yaml +++ b/deploy/helm/openshell/templates/cni-rbac.yaml @@ -44,4 +44,40 @@ subjects: - kind: ServiceAccount name: {{ include "openshell.cniServiceAccountName" . }} namespace: {{ .Release.Namespace }} +--- +# Nodes are cluster-scoped, so the readiness-label grant must be a ClusterRole. +# It is deliberately minimal: only get/patch on nodes, so the installer can +# label its own node ready/unready for the sandbox scheduling gate. No create, +# delete, or list, and no other resources. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "openshell.cniServiceAccountName" . }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni +rules: + - apiGroups: + - "" + resources: + - nodes + verbs: + - get + - patch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "openshell.cniServiceAccountName" . }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "openshell.cniServiceAccountName" . }} +subjects: + - kind: ServiceAccount + name: {{ include "openshell.cniServiceAccountName" . }} + namespace: {{ .Release.Namespace }} {{- end }} diff --git a/deploy/helm/openshell/templates/service.yaml b/deploy/helm/openshell/templates/service.yaml index 921eec4ee1..ebad42eab1 100644 --- a/deploy/helm/openshell/templates/service.yaml +++ b/deploy/helm/openshell/templates/service.yaml @@ -26,4 +26,3 @@ spec: {{- end }} selector: {{- include "openshell.selectorLabels" . | nindent 4 }} - app.kubernetes.io/component: gateway diff --git a/deploy/helm/openshell/tests/cni_daemonset_test.yaml b/deploy/helm/openshell/tests/cni_daemonset_test.yaml index 0bee0c9913..c66bc5151e 100644 --- a/deploy/helm/openshell/tests/cni_daemonset_test.yaml +++ b/deploy/helm/openshell/tests/cni_daemonset_test.yaml @@ -54,6 +54,37 @@ tests: - matchRegex: path: spec.template.spec.serviceAccountName pattern: -cni$ + # The CNI DaemonSet uses a distinct app name so the gateway Service and + # workload base selectors (name + instance) never match CNI pods. + - equal: + path: spec.selector.matchLabels["app.kubernetes.io/name"] + value: openshell-cni + - notExists: + path: spec.selector.matchLabels["app.kubernetes.io/component"] + - equal: + path: spec.template.metadata.labels["app.kubernetes.io/name"] + value: openshell-cni + - equal: + path: spec.template.metadata.labels["app.kubernetes.io/component"] + value: cni + # Readiness gate: the installer needs its node name and marks the node + # ready/unready so the gateway can gate sandbox scheduling. + - contains: + path: spec.template.spec.containers[0].env + content: + name: NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'node-ready --set' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'node-ready --clear' + - matchRegex: + path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] + pattern: 'node-ready --clear' - it: scopes cni annotation lookup to explicit sandbox namespace template: templates/cni-daemonset.yaml diff --git a/deploy/helm/openshell/tests/cni_rbac_test.yaml b/deploy/helm/openshell/tests/cni_rbac_test.yaml index c1759614a7..90f84cb895 100644 --- a/deploy/helm/openshell/tests/cni_rbac_test.yaml +++ b/deploy/helm/openshell/tests/cni_rbac_test.yaml @@ -15,7 +15,7 @@ tests: cni.enabled: true asserts: - hasDocuments: - count: 3 + count: 5 - documentIndex: 0 isKind: of: ServiceAccount @@ -47,6 +47,29 @@ tests: matchRegex: path: roleRef.name pattern: -cni$ + # ClusterRole/ClusterRoleBinding grant the minimal cluster-scoped node + # get/patch the installer needs to set the readiness label. + - documentIndex: 3 + isKind: + of: ClusterRole + - documentIndex: 3 + contains: + path: rules + content: + apiGroups: + - "" + resources: + - nodes + verbs: + - get + - patch + - documentIndex: 4 + isKind: + of: ClusterRoleBinding + - documentIndex: 4 + matchRegex: + path: roleRef.name + pattern: -cni$ - it: scopes cni pod-get role to the explicit sandbox namespace set: diff --git a/deploy/helm/openshell/tests/service_test.yaml b/deploy/helm/openshell/tests/service_test.yaml index 16eca90cfd..2d346c00ec 100644 --- a/deploy/helm/openshell/tests/service_test.yaml +++ b/deploy/helm/openshell/tests/service_test.yaml @@ -5,9 +5,15 @@ suite: gateway service templates: - templates/service.yaml tests: - - it: selects only gateway component pods + # The Service selector stays base-only (name + instance) so a helm upgrade + # from a release whose gateway pods predate the component label does not blank + # the Service endpoints. The CNI DaemonSet uses a distinct app name + # (-cni), so base-only selection never matches CNI pods. + - it: selects gateway pods by base labels template: templates/service.yaml asserts: - equal: + path: spec.selector["app.kubernetes.io/name"] + value: openshell + - notExists: path: spec.selector["app.kubernetes.io/component"] - value: gateway diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 732683e8ec..ac3c39c4e2 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -75,11 +75,11 @@ cni: # never modifies a CNO-managed file. mode: conflist image: - # -- CNI installer image repository. Empty uses supervisor.image.repository. + # -- CNI installer image repository. Empty falls back to supervisor.image.repository, then the official supervisor repository. repository: "" # -- CNI installer image pull policy. Empty uses supervisor.image.pullPolicy, then image.pullPolicy. pullPolicy: "" - # -- CNI installer image tag. Empty uses supervisor.image.tag, then chart appVersion. + # -- CNI installer image tag. Empty falls back to supervisor.image.tag, then image.tag, then chart appVersion. tag: "" # -- Host CNI binary directory. binDir: /opt/cni/bin diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index 9e4fb8a041..48908069c5 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -217,6 +217,15 @@ file, so `kubectl logs daemonset/openshell-cni -c install-cni` includes CNI `ADD` and `CHECK` errors that would otherwise only appear in pod events or node logs. +A per-node scheduling gate ensures enforcement is in place before a sandbox +runs. Once the chained plugin is installed, the installer labels its node +`openshell.ai/cni-ready=true`, and clears the label on shutdown or when a +reconcile tick cannot restore the plugin. The gateway sets a required +`nodeAffinity` on that label for every cni-sidecar sandbox pod, so a pod cannot +schedule onto a node whose egress enforcement is not active. An ungraceful CNI +DaemonSet pod deletion can briefly leave a stale `cni-ready=true` until the pod +is rescheduled and the next reconcile tick re-evaluates it. + `cni-sidecar` is experimental. It currently targets normal Kubernetes runtimes first. Kata Containers and gVisor remain validation targets, because each From e37129cb192a765e26902d7354552b803ef22323 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Mon, 3 Aug 2026 20:57:05 -0400 Subject: [PATCH 13/20] fix(cni-sidecar): harden readiness lifecycle, init, and IPv6 detection Address the third review round of the cni-sidecar topology: - cni/helm: fence before repair and stop failing open on teardown. The reconcile loop clears the readiness label the instant enforcement is not verifiably present, then repairs and re-marks. Enforcement lives in the host CNI config and survives pod restarts, so preStop no longer strips it on an ordinary restart/rolling update: a new `openshell-cni daemonset-active` check removes it only when the owning DaemonSet is confirmed terminating, and fences the node first. Reconcile interval 300s -> 30s. - cni/helm: stamp an owner (/) on the chained plugin entry and refuse to overwrite an entry owned by a different release (fail closed). Only one OpenShell release per cluster is supported for cni-sidecar today; documented in the RFC. - helm: validate the multus chain conf content (type, owner, kubeconfig, non-empty namespaces) before marking the node ready, instead of accepting any existing file. - driver: harden the untrusted workspace-init container (allowPrivilegeEscalation=false, drop ALL capabilities, runAsNonRoot when non-root) so a setuid binary cannot regain the exempt UID 0; restrict the non-root init to the Sidecar/CniSidecar topologies so combined keeps root. - cni: determine IPv6 enforcement by probing /proc/net/if_inet6 in the pod netns (fail closed on indeterminate; link-local counts) instead of trusting prevResult, keeping IPv4-only nodes working without ip6tables. - build: stage openshell-cni in the prebuilt 'all' target. - rbac: add get on the installer's own DaemonSet (resourceNames-scoped). Signed-off-by: Russell Bryant --- architecture/compute-runtimes.md | 34 ++- crates/openshell-cni/src/lib.rs | 227 ++++++++++++------ crates/openshell-cni/src/main.rs | 11 +- .../openshell-driver-kubernetes/src/driver.rs | 87 ++++++- .../openshell/templates/cni-daemonset.yaml | 110 +++++++-- deploy/helm/openshell/templates/cni-rbac.yaml | 10 + .../openshell/tests/cni_daemonset_test.yaml | 19 ++ .../helm/openshell/tests/cni_rbac_test.yaml | 15 ++ docs/kubernetes/topology.mdx | 8 + rfc/cni-sidecar-topology-DRAFT.md | 24 +- tasks/scripts/stage-prebuilt-binaries.sh | 4 +- 11 files changed, 422 insertions(+), 127 deletions(-) diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index f9b14d6551..c286a1a174 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -265,20 +265,34 @@ persistent `stateDir`. Neither mode modifies a CNO-managed file. The installer patches the chained plugin at startup and then re-verifies it on a reconcile tick, re-patching when the plugin is missing. This keeps enforcement in place across CNI config rewrites (for example a CNO reconcile) and DaemonSet -restarts, bounding any such gap to one reconcile interval. +restarts, bounding any such gap to one reconcile interval. The chained plugin +lives in the host CNI config and survives installer pod restarts, so an ordinary +DaemonSet restart or rolling update never strips enforcement: the `preStop` hook +removes it only when the owning DaemonSet is actually being deleted (helm +uninstall), and fences the node before doing so. A per-node scheduling gate closes the cold-start race. Once the chained plugin is -installed, the installer labels its node `openshell.ai/cni-ready=true`; it clears -the label on shutdown and whenever a reconcile tick cannot restore the plugin. -The gateway sets a required `nodeAffinity` on that label for every cni-sidecar -sandbox pod, so a pod cannot schedule onto a node before that node's egress -enforcement is active, and a node whose enforcement later breaks stops accepting -new sandbox pods. The label is set through a minimal cluster-scoped grant -(`nodes` `get`/`patch` only) bound to the dedicated CNI ServiceAccount. One -residual limitation: an ungraceful DaemonSet pod deletion (no `preStop`) leaves a -stale `cni-ready=true` until the pod is rescheduled and the next reconcile tick +installed, the installer labels its node `openshell.ai/cni-ready=true`. Each +reconcile tick fences before it repairs: the instant enforcement is not +verifiably in place it clears the label, attempts repair, and only re-marks the +node ready once the plugin is healthy again. The gateway sets a required +`nodeAffinity` on that label for every cni-sidecar sandbox pod, so a pod cannot +schedule onto a node before that node's egress enforcement is active, and a node +whose enforcement later breaks stops accepting new sandbox pods. The label is set +through a minimal cluster-scoped grant (`nodes` `get`/`patch`, plus `get` on the +installer's own DaemonSet) bound to the dedicated CNI ServiceAccount. One residual +limitation: an ungraceful DaemonSet pod deletion (no `preStop`) leaves a stale +`cni-ready=true` until the pod is rescheduled and the next reconcile tick re-evaluates it. +The installer stamps each chained plugin entry with an owner (`/`) +and refuses to overwrite an entry owned by a different release, failing closed +rather than silently unrestricting another release's sandboxes. Because the host +CNI config and the readiness label are cluster-global, **only one OpenShell +release per cluster is supported for the cni-sidecar topology today**; aggregating +multiple releases (or a namespace-changing upgrade) is future work noted in the +cni-sidecar RFC. + On OpenShift, binary-aware network policy also requires a purpose-built SecurityContextConstraints for sandbox pods: the network sidecar runs as UID 0 with `SYS_PTRACE` and `DAC_READ_SEARCH` to inspect cross-UID `/proc`, which diff --git a/crates/openshell-cni/src/lib.rs b/crates/openshell-cni/src/lib.rs index 18e1e6c2b4..f7328f4370 100644 --- a/crates/openshell-cni/src/lib.rs +++ b/crates/openshell-cni/src/lib.rs @@ -100,6 +100,17 @@ struct PodMetadata { annotations: BTreeMap, } +#[derive(Debug, Deserialize)] +struct DaemonSetResponse { + metadata: ObjectMeta, +} + +#[derive(Debug, Default, Deserialize)] +struct ObjectMeta { + #[serde(rename = "deletionTimestamp")] + deletion_timestamp: Option, +} + #[derive(Debug, Deserialize)] struct KubeConfig { #[serde(rename = "current-context")] @@ -155,34 +166,20 @@ trait PodReader { } trait RuleInstaller { - fn install(&self, netns: &Path, proxy_uid: u32, enforce_ipv6: bool) -> Result; + fn install(&self, netns: &Path, proxy_uid: u32) -> Result; fn cleanup(&self, netns: &Path) -> Result<()>; } -/// Returns true when the chained CNI `prevResult` reports at least one routable -/// IPv6 address for the pod. Loopback and link-local (`fe80::/10`) addresses are -/// ignored because they cannot carry egress off the node, so a pod with only -/// those needs no IPv6 firewall enforcement. IPv4-only pods return false, which -/// lets the iptables fallback skip the (possibly absent) ip6tables binary. -fn prev_result_has_ipv6(config: &CniConfig) -> bool { - let Some(prev) = config.prev_result.as_ref() else { - return false; - }; - let Some(ips) = prev.get("ips").and_then(Value::as_array) else { - return false; - }; - ips.iter().any(|entry| { - entry - .get("address") - .and_then(Value::as_str) - .and_then(|addr| addr.split('/').next()) - .and_then(|ip| ip.parse::().ok()) - .is_some_and(|ip| match ip { - std::net::IpAddr::V6(v6) => { - !v6.is_loopback() && (v6.segments()[0] & 0xffc0) != 0xfe80 - } - std::net::IpAddr::V4(_) => false, - }) +/// Returns true when `/proc/net/if_inet6` contents report at least one +/// non-loopback IPv6 address. Link-local (`fe80::/10`) counts because it can +/// still reach on-link peers and node services; only loopback (`::1`) is +/// excluded. Empty contents (IPv6 disabled in the kernel) return false. +fn if_inet6_has_non_loopback_ipv6(contents: &str) -> bool { + contents.lines().any(|line| { + // Format: <32-hex-addr> + line.split_whitespace() + .next() + .is_some_and(|addr| addr != "00000000000000000000000000000001") }) } @@ -225,6 +222,36 @@ pub fn node_ready(args: &[String]) -> Result<()> { client.patch_node_label(&node, NODE_READY_LABEL, value) } +/// Entry point for `openshell-cni daemonset-active`. +/// +/// Used by the installer `preStop` hook to distinguish an ordinary pod +/// restart/rolling update from a real teardown (helm uninstall / `DaemonSet` +/// delete). +/// +/// Returns `Ok(())` (exit 0) whenever enforcement should be **preserved** — the +/// owning `DaemonSet` still exists and is not being deleted, OR the state cannot +/// be determined (missing env, API error). Enforcement lives in the host CNI +/// config and survives pod restarts, so preserving is the fail-safe default. +/// Returns `Err` (non-zero) only when the `DaemonSet` is positively confirmed +/// gone or terminating, signalling the caller that cleanup is appropriate. +pub fn daemonset_active() -> Result<()> { + let (Ok(name), Ok(namespace)) = (std::env::var("DS_NAME"), std::env::var("DS_NAMESPACE")) + else { + // Cannot identify the owning DaemonSet; preserve enforcement. + return Ok(()); + }; + let Ok(client) = KubeApiClient::from_in_cluster() else { + return Ok(()); + }; + match client.daemonset_terminating(&namespace, &name) { + Ok(true) => Err(miette::miette!( + "owning DaemonSet {namespace}/{name} is gone or terminating" + )), + // Active, or an API error we cannot interpret: preserve enforcement. + Ok(false) | Err(_) => Ok(()), + } +} + pub fn run() -> Result<()> { let env = CniEnv::from_process(); let mut input = String::new(); @@ -334,8 +361,7 @@ fn handle_command( let netns = env.netns.as_deref().ok_or_else(|| { miette::miette!("CNI_NETNS is required for OpenShell CNI ADD") })?; - let report = - installer.install(netns, workload.proxy_uid, prev_result_has_ipv6(&config))?; + let report = installer.install(netns, workload.proxy_uid)?; log_cni_info( &config, env, @@ -353,8 +379,7 @@ fn handle_command( let netns = env.netns.as_deref().ok_or_else(|| { miette::miette!("CNI_NETNS is required for OpenShell CNI CHECK") })?; - let report = - installer.install(netns, workload.proxy_uid, prev_result_has_ipv6(&config))?; + let report = installer.install(netns, workload.proxy_uid)?; log_cni_info( &config, env, @@ -581,6 +606,36 @@ impl KubeApiClient { Ok(()) } + /// Returns true when the named `DaemonSet` is gone (404) or has a + /// `deletionTimestamp` set (being deleted). A non-404 HTTP error is + /// propagated so the caller can treat it as indeterminate. + fn daemonset_terminating(&self, namespace: &str, name: &str) -> Result { + let url = format!( + "{}/apis/apps/v1/namespaces/{}/daemonsets/{}", + self.server, namespace, name + ); + let response = self + .client + .get(url) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to query Kubernetes API for DaemonSet state")?; + if response.status().as_u16() == 404 { + return Ok(true); + } + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while reading DaemonSet {}/{}", + response.status(), + namespace, + name + )); + } + let ds = response.json::().into_diagnostic()?; + Ok(ds.metadata.deletion_timestamp.is_some()) + } + fn pod_annotations(&self, pod: &PodRef) -> Result> { let url = format!( "{}/api/v1/namespaces/{}/pods/{}", @@ -637,8 +692,8 @@ fn cluster_certificate_authority( } impl RuleInstaller for Runtime { - fn install(&self, netns: &Path, proxy_uid: u32, enforce_ipv6: bool) -> Result { - install_rules(netns, proxy_uid, enforce_ipv6) + fn install(&self, netns: &Path, proxy_uid: u32) -> Result { + install_rules(netns, proxy_uid) } fn cleanup(&self, netns: &Path) -> Result<()> { @@ -682,11 +737,11 @@ fn generate_sidecar_bypass_ruleset(proxy_uid: u32, log_prefix: Option<&str>) -> } #[cfg(target_os = "linux")] -fn install_rules(netns: &Path, proxy_uid: u32, enforce_ipv6: bool) -> Result { +fn install_rules(netns: &Path, proxy_uid: u32) -> Result { // The preferred nft backend programs both families in one inet ruleset, so - // its IPv6 reject rules are harmless no-ops on IPv4-only pods; enforce_ipv6 - // only gates the iptables fallback, where a missing ip6tables binary would - // otherwise reject an IPv4-only node outright. + // its IPv6 reject rules are harmless no-ops on IPv4-only pods and it needs no + // IPv6 probe. Only the iptables fallback must decide whether ip6tables is + // required, and it does so by inspecting the pod netns (fail-closed). let nft_error = if let Some(nft) = find_nft() { match install_nft_rules(netns, proxy_uid, &nft) { Ok(()) => { @@ -699,6 +754,7 @@ fn install_rules(netns: &Path, proxy_uid: u32, enforce_ipv6: bool) -> Result Result<()> { } #[cfg(not(target_os = "linux"))] -fn install_rules(netns: &Path, proxy_uid: u32, enforce_ipv6: bool) -> Result { - let _ = (netns, proxy_uid, enforce_ipv6); +fn install_rules(netns: &Path, proxy_uid: u32) -> Result { + let _ = (netns, proxy_uid); Err(miette::miette!( "OpenShell CNI rule installation is supported only on Linux nodes" )) } +/// Decides whether the iptables fallback must enforce IPv6, by probing the pod +/// netns for any non-loopback IPv6 address. Fails closed: any error (setns/exec +/// failure, or the probe reporting IPv6 present) returns true, so a pod with +/// IPv6 is never left unenforced just because detection was inconclusive. Only a +/// clean "no IPv6" result (probe exit 0) returns false, keeping IPv4-only nodes +/// working without ip6tables. +#[cfg(target_os = "linux")] +fn netns_requires_ipv6_enforcement(netns: &Path) -> bool { + // Re-exec this binary inside the target netns; its exit status encodes the + // result (exit 0 = no IPv6 → Ok, non-zero = IPv6/undetermined → Err). + run_command_in_netns(netns, "/proc/self/exe", &[NETNS_IPV6_PROBE_ARG]).is_err() +} + +/// Argument that runs the in-netns IPv6 probe (see `netns_probe_ipv6`). +#[cfg(target_os = "linux")] +const NETNS_IPV6_PROBE_ARG: &str = "__netns-probe-ipv6"; + +/// In-netns IPv6 probe entry point, re-exec'd inside the pod netns. +/// +/// Reports via exit status whether IPv6 enforcement is needed: success (exit 0) +/// means no non-loopback IPv6 is present; a non-success exit means IPv6 is +/// present or could not be determined, so the caller fails closed. A missing +/// `/proc/net/if_inet6` means IPv6 is disabled in the kernel (no enforcement +/// needed); any other read error is treated as indeterminate. +pub fn netns_probe_ipv6() -> Result<()> { + match std::fs::read_to_string("/proc/net/if_inet6") { + Ok(contents) => { + if if_inet6_has_non_loopback_ipv6(&contents) { + Err(miette::miette!("pod netns has non-loopback IPv6")) + } else { + Ok(()) + } + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(error) => Err(error) + .into_diagnostic() + .wrap_err("failed to read /proc/net/if_inet6 for IPv6 probe"), + } +} + #[cfg(target_os = "linux")] fn cleanup_rules(netns: &Path) -> Result<()> { if let Some(nft) = find_nft() { @@ -1026,12 +1122,7 @@ mod tests { } impl RuleInstaller for TestInstaller { - fn install( - &self, - _netns: &Path, - proxy_uid: u32, - _enforce_ipv6: bool, - ) -> Result { + fn install(&self, _netns: &Path, proxy_uid: u32) -> Result { self.installed.lock().unwrap().push(proxy_uid); Ok(InstallReport { backend: "test" }) } @@ -1292,52 +1383,30 @@ mod tests { assert_eq!(payload["cniVersion"], DEFAULT_CNI_VERSION); } - fn config_with_prev_result(prev_result: Value) -> CniConfig { - CniConfig { - cni_version: Some("1.0.0".to_string()), - prev_result: Some(prev_result), - openshell: OpenShellConfig::default(), - } - } - #[test] - fn prev_result_has_ipv6_true_for_routable_ipv6() { - let config = config_with_prev_result(serde_json::json!({ - "ips": [ - { "address": "10.1.2.3/24" }, - { "address": "fd00:10:244::5/64" } - ] - })); - assert!(prev_result_has_ipv6(&config)); + fn if_inet6_detects_global_and_ula() { + // Global address on eth0. + let contents = "fd0010244000000000000000000000005 03 40 00 80 eth0\n"; + assert!(if_inet6_has_non_loopback_ipv6(contents)); } #[test] - fn prev_result_has_ipv6_false_for_ipv4_only() { - let config = config_with_prev_result(serde_json::json!({ - "ips": [{ "address": "10.1.2.3/24" }] - })); - assert!(!prev_result_has_ipv6(&config)); + fn if_inet6_counts_link_local() { + // Link-local fe80::/10 can still reach on-link peers and node services. + let contents = "fe800000000000000042acfffe110002 02 40 20 80 eth0\n"; + assert!(if_inet6_has_non_loopback_ipv6(contents)); } #[test] - fn prev_result_has_ipv6_ignores_link_local_and_loopback() { - let config = config_with_prev_result(serde_json::json!({ - "ips": [ - { "address": "fe80::1/64" }, - { "address": "::1/128" } - ] - })); - assert!(!prev_result_has_ipv6(&config)); + fn if_inet6_ignores_loopback_only() { + // Only ::1 on lo → no enforcement needed. + let contents = "00000000000000000000000000000001 01 80 10 80 lo\n"; + assert!(!if_inet6_has_non_loopback_ipv6(contents)); } #[test] - fn prev_result_has_ipv6_false_when_prev_result_absent() { - let config = CniConfig { - cni_version: Some("1.0.0".to_string()), - prev_result: None, - openshell: OpenShellConfig::default(), - }; - assert!(!prev_result_has_ipv6(&config)); + fn if_inet6_empty_means_no_ipv6() { + assert!(!if_inet6_has_non_loopback_ipv6("")); } #[test] diff --git a/crates/openshell-cni/src/main.rs b/crates/openshell-cni/src/main.rs index b30589b13f..f4ded8faeb 100644 --- a/crates/openshell-cni/src/main.rs +++ b/crates/openshell-cni/src/main.rs @@ -6,10 +6,13 @@ fn main() { // (config on stdin, CNI_COMMAND in env). The `node-ready` subcommand is only // reached when the installer DaemonSet calls the binary explicitly. let args: Vec = std::env::args().collect(); - let result = if args.get(1).map(String::as_str) == Some("node-ready") { - openshell_cni::node_ready(&args[2..]) - } else { - openshell_cni::run() + let result = match args.get(1).map(String::as_str) { + Some("node-ready") => openshell_cni::node_ready(&args[2..]), + Some("daemonset-active") => openshell_cni::daemonset_active(), + // Internal: re-exec'd inside a pod netns to probe for IPv6 (see + // netns_requires_ipv6_enforcement). Not part of the public CLI surface. + Some("__netns-probe-ipv6") => openshell_cni::netns_probe_ipv6(), + _ => openshell_cni::run(), }; if let Err(error) = result { eprintln!("{error:?}"); diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index b8cbadb2d7..df334fea1f 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -2155,13 +2155,25 @@ fn apply_workspace_persistence( // that exemption and egress freely before the sidecar enforces policy. // Callers pass the sandbox UID in that mode (non-exempt, still able to // seed the PVC via fsGroup) and 0 otherwise (where UID 0 is not exempt). + // + // Harden the securityContext regardless: dropping all capabilities and + // disabling privilege escalation stops a setuid binary in the untrusted + // image from regaining the exempt UID 0 and bypassing egress. When the + // container runs non-root, also assert runAsNonRoot so the kubelet + // refuses an image that would otherwise force UID 0. + let mut security_context = serde_json::json!({ + "runAsUser": workspace_init_uid, + "allowPrivilegeEscalation": false, + "capabilities": { "drop": ["ALL"] }, + }); + if workspace_init_uid != 0 { + security_context["runAsNonRoot"] = serde_json::json!(true); + } let mut init_spec = serde_json::json!({ "name": WORKSPACE_INIT_CONTAINER_NAME, "image": image, "command": ["sh", "-c", copy_cmd], - "securityContext": { - "runAsUser": workspace_init_uid, - }, + "securityContext": security_context, "volumeMounts": [{ "name": WORKSPACE_VOLUME_NAME, "mountPath": WORKSPACE_INIT_MOUNT_PATH @@ -2800,10 +2812,17 @@ fn sandbox_template_to_k8s_with_validated_config( if inject_workspace { // Under binary-aware network policy the CNI firewall exempts UID 0, so // the untrusted workspace-init must not run as root or it could bypass - // egress policy. Run it as the (non-exempt) sandbox UID in that mode; - // otherwise UID 0 is already fenced and root keeps the widest read - // access to the image's /sandbox contents. - let workspace_init_uid = if params.process_binary_aware_network_policy { + // egress policy. This exemption only exists in the sidecar topologies + // (Sidecar / CniSidecar), which run a separate network supervisor at the + // exempt UID; the combined topology has no such exemption, so it keeps + // root to preserve the widest read access to the image's /sandbox + // contents and PVC data. Run as the (non-exempt) sandbox UID only where + // the exemption applies. + let workspace_init_uid = if matches!( + params.topology, + SupervisorTopology::Sidecar | SupervisorTopology::CniSidecar + ) && params.process_binary_aware_network_policy + { params.sandbox_uid } else { 0 @@ -5021,11 +5040,63 @@ mod tests { .iter() .find(|container| container["name"] == WORKSPACE_INIT_CONTAINER_NAME) .expect("workspace-init container must be present"); + let sc = &init["securityContext"]; assert_eq!( - init["securityContext"]["runAsUser"], + sc["runAsUser"], serde_json::json!(1_000_790_000), "workspace-init must run as the non-exempt sandbox UID in binary-aware mode" ); + // Hardening: a setuid binary in the untrusted image must not be able to + // regain the exempt UID 0. + assert_eq!(sc["allowPrivilegeEscalation"], serde_json::json!(false)); + assert_eq!(sc["runAsNonRoot"], serde_json::json!(true)); + assert_eq!(sc["capabilities"]["drop"], serde_json::json!(["ALL"])); + } + + /// The UID-0 exemption only exists in the sidecar topologies. A combined + /// deployment with binary-aware policy on must keep root for workspace-init + /// so it retains access to root-owned image and PVC contents. + #[test] + fn combined_topology_workspace_init_keeps_root() { + let params = SandboxPodParams { + topology: SupervisorTopology::Combined, + supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, + supervisor_image: "supervisor-image:latest", + grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", + sandbox_id: "sb-combined-init", + sandbox_uid: 1500, + sandbox_gid: 1500, + process_binary_aware_network_policy: true, + ..SandboxPodParams::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate { + image: "agent-image:latest".to_string(), + ..SandboxTemplate::default() + }, + false, + &std::collections::HashMap::new(), + true, // inject_workspace + ¶ms, + ); + + let init = pod_template["spec"]["initContainers"] + .as_array() + .unwrap() + .iter() + .find(|container| container["name"] == WORKSPACE_INIT_CONTAINER_NAME) + .expect("workspace-init container must be present"); + assert_eq!( + init["securityContext"]["runAsUser"], + serde_json::json!(0), + "combined topology keeps root workspace-init even with binary-aware policy" + ); + // Hardening still applies, but runAsNonRoot must not be asserted for root. + assert_eq!( + init["securityContext"]["allowPrivilegeEscalation"], + serde_json::json!(false) + ); + assert!(init["securityContext"]["runAsNonRoot"].is_null()); } /// Companion: without binary-aware policy, UID 0 is not exempt, so the init diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml index 133d273167..398111640d 100644 --- a/deploy/helm/openshell/templates/cni-daemonset.yaml +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -52,6 +52,12 @@ spec: valueFrom: fieldRef: fieldPath: spec.nodeName + # Used by `openshell-cni daemonset-active` so the preStop hook can tell + # an ordinary pod restart from a real teardown (helm uninstall). + - name: DS_NAME + value: {{ include "openshell.fullname" . }}-cni + - name: DS_NAMESPACE + value: {{ .Release.Namespace | quote }} command: - sh - -ec @@ -91,12 +97,21 @@ spec: cluster: cluster user: openshell-cni EOF + # Ownership stamp so a second OpenShell release (or a namespace- + # changing upgrade) cannot silently overwrite this release's chained + # plugin and leave its sandboxes unenforced. Install/repair refuse + # to clobber an entry owned by a different release (fail closed). + # NOTE: one OpenShell release per cluster is the supported model for + # cni-sidecar today (see the cni-sidecar RFC for the multi-release + # follow-up). + owner="{{ .Release.Namespace }}/{{ .Release.Name }}" cat > "${plugin_config}" </dev/null 2>&1 + } + # True when the target carries an openshell-cni plugin owned by a + # DIFFERENT release. We must never overwrite it (that would silently + # unrestrict the other release's sandboxes). + conflist_foreign() { + _t="$1" + [ -n "${_t}" ] && [ -f "${_t}" ] && jq -e --arg o "${owner}" \ + 'any((.plugins // [])[]; .type == "openshell-cni" and (.openshell.owner // "") != $o)' \ + "${_t}" >/dev/null 2>&1 + } patch_conflist() { target="$(resolve_conflist_target)" if [ -z "${target}" ] || [ ! -f "${target}" ]; then echo "OpenShell CNI requires an existing CNI .conflist in ${conf_dir}" >&2 return 1 fi + if conflist_foreign "${target}"; then + echo "OpenShell CNI refuses to overwrite a chained plugin owned by another release in ${target}; only one OpenShell release per cluster is supported for cni-sidecar" >&2 + return 1 + fi if [ ! -f "${target}.openshell-backup" ]; then cp "${target}" "${target}.openshell-backup" fi @@ -150,18 +186,6 @@ spec: return 1 fi } - # Re-patch only when the chained plugin is absent, so a CNI config - # rewrite (CNO reconcile, node reimage) or a jq no-op cannot leave the - # node forwarding sandbox traffic without OpenShell egress enforcement. - ensure_conflist() { - target="$(resolve_conflist_target)" - if [ -n "${target}" ] && [ -f "${target}" ] \ - && jq -e 'any((.plugins // [])[]; .type == "openshell-cni")' "${target}" >/dev/null 2>&1; then - return 0 - fi - echo "OpenShell CNI re-patching ${target:-}; chained plugin missing (CNI config rewrite or restart)" >&2 - patch_conflist - } patch_conflist || exit 1 mark_ready echo "OpenShell CNI installed (conflist mode); plugin log ${log_file}; firewall backend selected during CNI ADD" @@ -169,7 +193,28 @@ spec: chain_dir="/host{{ .Values.cni.chainDir }}" chain_conf="${chain_dir}/openshell-cni.conf" mkdir -p "${chain_dir}" + # True only when the chain conf is a well-formed openshell-cni config + # owned by this release. Mere file existence is not enough — an empty, + # corrupt, stale, or foreign file must NOT mark the node ready. + chain_conf_healthy() { + [ -f "${chain_conf}" ] && jq -e --arg o "${owner}" ' + .type == "openshell-cni" + and .openshell.owner == $o + and (.openshell.kubeconfig // "") != "" + and ((.openshell.sandboxNamespaces // []) | length) > 0 + ' "${chain_conf}" >/dev/null 2>&1 + } + # True when a chain conf owned by a different release is present. + chain_conf_foreign() { + [ -f "${chain_conf}" ] && jq -e --arg o "${owner}" \ + '.type == "openshell-cni" and (.openshell.owner // "") != $o' \ + "${chain_conf}" >/dev/null 2>&1 + } write_chain_conf() { + if chain_conf_foreign; then + echo "OpenShell CNI refuses to overwrite ${chain_conf} owned by another release; only one OpenShell release per cluster is supported for cni-sidecar" >&2 + return 1 + fi _tmp="$(mktemp "${chain_dir}/openshell-cni.XXXXXX")" # Same fail-closed contract as patch_conflist: never install a # truncated chain config if jq fails. @@ -193,15 +238,25 @@ spec: while true; do cp /var/run/secrets/kubernetes.io/serviceaccount/token "${token_file}" chmod 0600 "${token_file}" + # Fence before repair: the instant enforcement is not verifiably in + # place, clear readiness so no new sandbox pod schedules here, then + # attempt repair and only re-mark ready once it is healthy again. {{- if eq $mode "conflist" }} - # Keep the node's readiness label in lockstep with enforcement: - # mark unready if the chained plugin cannot be restored, so no new - # sandbox pods schedule here until it is. - if ensure_conflist; then mark_ready; else mark_unready; fi + if conflist_ours "$(resolve_conflist_target)"; then + mark_ready + else + mark_unready + if patch_conflist; then mark_ready; fi + fi {{- else }} - if { [ -f "${chain_conf}" ] || write_chain_conf; }; then mark_ready; else mark_unready; fi + if chain_conf_healthy; then + mark_ready + else + mark_unready + if write_chain_conf; then mark_ready; fi + fi {{- end }} - sleep 300 + sleep 30 done securityContext: privileged: true @@ -213,9 +268,22 @@ spec: - sh - -ec - | - # Fence the node before removing enforcement so no sandbox pod - # schedules here during teardown. - /openshell-cni node-ready --clear || true + # Enforcement lives in the host CNI config and survives pod + # restarts, so an ordinary restart/rolling update must NOT strip + # it (that would fail open for running sandboxes). Only tear it + # down on a real teardown: daemonset-active exits 0 (preserve) + # unless the owning DaemonSet is confirmed gone/terminating. + if /openshell-cni daemonset-active; then + echo "OpenShell CNI: DaemonSet still active; preserving enforcement across restart" >&2 + exit 0 + fi + # Teardown: fence the node first and only remove enforcement if + # fencing succeeded, so we never strip enforcement while the node + # still advertises readiness. + if ! /openshell-cni node-ready --clear; then + echo "OpenShell CNI: fencing failed during teardown; leaving enforcement in place" >&2 + exit 0 + fi {{- if eq $mode "conflist" }} conf_dir="/host{{ .Values.cni.confDir }}" for target in "${conf_dir}"/*.conflist; do diff --git a/deploy/helm/openshell/templates/cni-rbac.yaml b/deploy/helm/openshell/templates/cni-rbac.yaml index b81cac83e1..3ec5fcbf6e 100644 --- a/deploy/helm/openshell/templates/cni-rbac.yaml +++ b/deploy/helm/openshell/templates/cni-rbac.yaml @@ -64,6 +64,16 @@ rules: verbs: - get - patch + # get on this DaemonSet only, so the preStop hook can tell an ordinary restart + # from a real teardown. resourceNames keeps the grant scoped to our object. + - apiGroups: + - apps + resources: + - daemonsets + resourceNames: + - {{ include "openshell.fullname" . }}-cni + verbs: + - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding diff --git a/deploy/helm/openshell/tests/cni_daemonset_test.yaml b/deploy/helm/openshell/tests/cni_daemonset_test.yaml index c66bc5151e..62673c23c0 100644 --- a/deploy/helm/openshell/tests/cni_daemonset_test.yaml +++ b/deploy/helm/openshell/tests/cni_daemonset_test.yaml @@ -4,6 +4,8 @@ suite: cni daemonset templates: - templates/cni-daemonset.yaml +release: + name: openshell tests: - it: does not render by default template: templates/cni-daemonset.yaml @@ -85,6 +87,23 @@ tests: - matchRegex: path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] pattern: 'node-ready --clear' + # preStop preserves enforcement across ordinary restarts (only tears down + # on real teardown, gated by daemonset-active). + - contains: + path: spec.template.spec.containers[0].env + content: + name: DS_NAME + value: openshell-cni + - matchRegex: + path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] + pattern: 'daemonset-active' + # Ownership stamp + fence-before-repair reconcile. + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '"owner": "' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'refuses to overwrite' - it: scopes cni annotation lookup to explicit sandbox namespace template: templates/cni-daemonset.yaml diff --git a/deploy/helm/openshell/tests/cni_rbac_test.yaml b/deploy/helm/openshell/tests/cni_rbac_test.yaml index 90f84cb895..471a0930f3 100644 --- a/deploy/helm/openshell/tests/cni_rbac_test.yaml +++ b/deploy/helm/openshell/tests/cni_rbac_test.yaml @@ -4,6 +4,8 @@ suite: cni rbac templates: - templates/cni-rbac.yaml +release: + name: openshell tests: - it: does not render by default asserts: @@ -63,6 +65,19 @@ tests: verbs: - get - patch + # daemonsets get, scoped by resourceNames to this release's DaemonSet. + - documentIndex: 3 + contains: + path: rules + content: + apiGroups: + - apps + resources: + - daemonsets + resourceNames: + - openshell-cni + verbs: + - get - documentIndex: 4 isKind: of: ClusterRoleBinding diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index 48908069c5..1e74b77e86 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -233,6 +233,14 @@ runtime must honor the CNI-installed pod-network rules for this topology to provide the expected network enforcement. + +Install only **one** OpenShell release per cluster when using `cni-sidecar`. The +chained CNI plugin and the readiness label are cluster-global host state; the +installer stamps an owner on its plugin entry and refuses to overwrite another +release's entry, so a second release (or a `sandboxNamespace`-changing upgrade) +fails closed rather than silently unrestricting sandboxes. + + ## Credential Exposure Sidecar and cni-sidecar topologies keep gateway credentials in the network diff --git a/rfc/cni-sidecar-topology-DRAFT.md b/rfc/cni-sidecar-topology-DRAFT.md index c1eba458b9..78699fc5ad 100644 --- a/rfc/cni-sidecar-topology-DRAFT.md +++ b/rfc/cni-sidecar-topology-DRAFT.md @@ -391,10 +391,26 @@ behavior is behind a default-off value. ## Risks - **Fail-open window.** If the OpenShell CNI conf is missing (post-reboot before - the DaemonSet re-asserts), pods can network without the fence. Mitigation: the - re-assert loop and startup write shrink the window; true fail-closed is a - follow-up. Operators diagnose via `kubectl logs daemonset/openshell-cni` and - the tailed plugin log. + the DaemonSet re-asserts), pods could network without the fence. Mitigation: a + per-node readiness gate — the installer labels its node `openshell.ai/cni-ready` + and the gateway requires that label via `nodeAffinity`, so sandbox pods cannot + schedule before enforcement is verified. The reconcile loop fences before it + repairs (clears the label the instant enforcement is not verifiably present), + and enforcement lives in the host CNI config so ordinary DaemonSet restarts + never strip it — `preStop` removes it only on a real teardown, fencing first. + Residual: an ungraceful installer-pod deletion (no `preStop`) can leave a stale + `cni-ready=true` until the pod is rescheduled and the next reconcile re-checks. + Operators diagnose via `kubectl logs daemonset/openshell-cni` and the tailed + plugin log. +- **Single release per cluster.** The chained plugin and readiness label are + cluster-global host state keyed to one `sandboxNamespaces` list, so two + OpenShell releases (or a `sandboxNamespace`-changing upgrade) would otherwise + overwrite each other and leave one release's sandboxes unenforced. Mitigation: + the installer stamps an owner (`/`) on its plugin entry and + refuses to overwrite an entry owned by a different release (fail closed); only + one OpenShell release per cluster is supported for `cni-sidecar` today. + Aggregating multiple releases (a cluster-singleton installer, or release-scoped + config and readiness ownership) is future work. - **Dependency on Multus aux chain.** `multus-chain` requires the Multus thick daemon to have `auxiliaryCNIChainName` set. It is default on OpenShift 4.x but not guaranteed everywhere. Mitigation: document the requirement; the chart does diff --git a/tasks/scripts/stage-prebuilt-binaries.sh b/tasks/scripts/stage-prebuilt-binaries.sh index b153bdbbe8..28a860fb95 100755 --- a/tasks/scripts/stage-prebuilt-binaries.sh +++ b/tasks/scripts/stage-prebuilt-binaries.sh @@ -97,7 +97,9 @@ components_for_target() { echo "cni" ;; all) - echo "gateway supervisor" + # The supervisor image bundles openshell-cni, so the aggregate prebuilt + # target must stage it too (kept in sync with the supervisor target above). + echo "gateway supervisor cni" ;; *) usage From 6dfd16bfca7c9c4ab9efb63a916ea58c4d8e88d4 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Tue, 4 Aug 2026 09:27:47 -0400 Subject: [PATCH 14/20] fix(cni-sidecar): make the CNI installer a cluster singleton Address the fourth review round of the cni-sidecar topology: - cni/helm: convert the CNI installer to a cluster singleton. The chained plugin enforces any pod carrying the OpenShell annotations (set only by a gateway on its own sandbox pods), so a single installation serves every release across all namespaces. The plugin config uses an empty sandboxNamespaces allowlist and a fixed 'openshell' owner; the installer resources use release-independent names; pods get RBAC becomes cluster-scoped. This fixes the multi-release fail-open: a second release no longer relies on a namespace list that another release could overwrite. Additional gateway releases set cni.enabled=false + cni.external=true to share the singleton. - helm: bind node readiness to a configVersion stamp so a stale-version entry is repaired before the node is re-marked ready; fence before the initial patch on startup; retry mark_unready so a transient API error does not leave stale readiness. - helm: create the host SA token atomically with install -m 0600 (no briefly world-readable node-patch credential). - driver: keep default root capabilities for the combined-topology workspace init (UID 0 is not CNI-exempt there) so it retains DAC read access; apply the drop-ALL/no-privilege-escalation/runAsNonRoot hardening only to the non-root exempt (Sidecar/CniSidecar binary-aware) case. - docs: add a cni-sidecar enable-and-verify walkthrough to the OpenShift page, add user-facing example values files under deploy/helm/openshell/examples/ (replacing the ci/ fixture in docs), and update topology, compute-runtimes, and the RFC to the singleton model. Signed-off-by: Russell Bryant --- architecture/compute-runtimes.md | 20 ++-- crates/openshell-cni/src/lib.rs | 6 ++ .../openshell-driver-kubernetes/src/driver.rs | 31 +++--- deploy/helm/openshell/README.md | 3 +- .../openshell/examples/cni-sidecar-k3s.yaml | 24 +++++ .../cni-sidecar-openshift-extra-release.yaml | 23 +++++ .../examples/cni-sidecar-openshift.yaml | 39 ++++++++ deploy/helm/openshell/templates/_helpers.tpl | 20 +++- .../openshell/templates/cni-daemonset.yaml | 94 ++++++++++++------- deploy/helm/openshell/templates/cni-rbac.yaml | 63 ++++--------- deploy/helm/openshell/templates/cni-scc.yaml | 12 +-- .../openshell/tests/cni_daemonset_test.yaml | 37 ++++++-- .../helm/openshell/tests/cni_rbac_test.yaml | 66 ++++++------- deploy/helm/openshell/values.yaml | 12 ++- docs/kubernetes/openshift.mdx | 57 +++++++++++ docs/kubernetes/topology.mdx | 21 +++-- rfc/cni-sidecar-topology-DRAFT.md | 21 +++-- 17 files changed, 375 insertions(+), 174 deletions(-) create mode 100644 deploy/helm/openshell/examples/cni-sidecar-k3s.yaml create mode 100644 deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml create mode 100644 deploy/helm/openshell/examples/cni-sidecar-openshift.yaml diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index c286a1a174..b3958329e4 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -285,13 +285,19 @@ limitation: an ungraceful DaemonSet pod deletion (no `preStop`) leaves a stale `cni-ready=true` until the pod is rescheduled and the next reconcile tick re-evaluates it. -The installer stamps each chained plugin entry with an owner (`/`) -and refuses to overwrite an entry owned by a different release, failing closed -rather than silently unrestricting another release's sandboxes. Because the host -CNI config and the readiness label are cluster-global, **only one OpenShell -release per cluster is supported for the cni-sidecar topology today**; aggregating -multiple releases (or a namespace-changing upgrade) is future work noted in the -cni-sidecar RFC. +The CNI installer is a **cluster singleton**. The chained plugin enforces any pod +carrying the OpenShell annotations (`openshell.ai/cni=enabled` plus the proxy-UID +and enforcement-mode annotations, set only by a gateway on its own sandbox pods), +so a single installation serves every OpenShell release across all namespaces — +enforcement is gated by per-pod annotations, not a namespace list. The installer +resources use release-independent names, the plugin config carries a fixed +`openshell` owner and an empty namespace allowlist, and a `configVersion` binds +readiness to the desired config so a stale-version entry is repaired before the +node is re-marked ready. Install the singleton (`cni.enabled=true`) in one release +per cluster; additional gateway releases set `cni.enabled=false` + +`cni.external=true` to share it, and the installer refuses to overwrite a chained +plugin entry with a non-OpenShell owner. Because the plugin serves all sandbox +namespaces, its `pods get` grant is cluster-scoped. On OpenShift, binary-aware network policy also requires a purpose-built SecurityContextConstraints for sandbox pods: the network sidecar runs as UID 0 diff --git a/crates/openshell-cni/src/lib.rs b/crates/openshell-cni/src/lib.rs index f7328f4370..095d9e3128 100644 --- a/crates/openshell-cni/src/lib.rs +++ b/crates/openshell-cni/src/lib.rs @@ -408,6 +408,12 @@ fn workload_from_config( let Some(pod) = env.pod_ref() else { return Ok(None); }; + // sandbox_namespaces is an OPTIONAL allowlist. The cluster-singleton + // installer leaves it empty, which means "enforce any pod that carries the + // OpenShell annotations, in any namespace" — enforcement is gated by the + // per-pod annotations below (set only by a gateway on its own sandbox pods), + // so one installation safely serves every release. A non-empty list narrows + // enforcement to those namespaces. if !config.openshell.sandbox_namespaces.is_empty() && !config .openshell diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index df334fea1f..b4324027aa 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -2156,18 +2156,20 @@ fn apply_workspace_persistence( // Callers pass the sandbox UID in that mode (non-exempt, still able to // seed the PVC via fsGroup) and 0 otherwise (where UID 0 is not exempt). // - // Harden the securityContext regardless: dropping all capabilities and - // disabling privilege escalation stops a setuid binary in the untrusted - // image from regaining the exempt UID 0 and bypassing egress. When the - // container runs non-root, also assert runAsNonRoot so the kubelet - // refuses an image that would otherwise force UID 0. - let mut security_context = serde_json::json!({ - "runAsUser": workspace_init_uid, - "allowPrivilegeEscalation": false, - "capabilities": { "drop": ["ALL"] }, - }); + // Hardening applies only when the init runs as the non-root exempt UID + // (the Sidecar/CniSidecar binary-aware case). There, dropping all + // capabilities and disabling privilege escalation stops a setuid binary + // in the untrusted image from regaining the exempt UID 0 and bypassing + // egress, and runAsNonRoot makes the kubelet refuse an image that would + // force UID 0. When the init runs as root (combined topology, where UID 0 + // is not exempt), keep the default root capabilities so it retains the + // broad DAC read access needed to seed the PVC from image files owned by + // other UIDs with restrictive modes. + let mut security_context = serde_json::json!({ "runAsUser": workspace_init_uid }); if workspace_init_uid != 0 { + security_context["allowPrivilegeEscalation"] = serde_json::json!(false); security_context["runAsNonRoot"] = serde_json::json!(true); + security_context["capabilities"] = serde_json::json!({ "drop": ["ALL"] }); } let mut init_spec = serde_json::json!({ "name": WORKSPACE_INIT_CONTAINER_NAME, @@ -5091,12 +5093,11 @@ mod tests { serde_json::json!(0), "combined topology keeps root workspace-init even with binary-aware policy" ); - // Hardening still applies, but runAsNonRoot must not be asserted for root. - assert_eq!( - init["securityContext"]["allowPrivilegeEscalation"], - serde_json::json!(false) - ); + // Root init keeps default capabilities (UID 0 is not CNI-exempt here) so + // it retains broad DAC read access; the setuid-hardening must NOT apply. + assert!(init["securityContext"]["allowPrivilegeEscalation"].is_null()); assert!(init["securityContext"]["runAsNonRoot"].is_null()); + assert!(init["securityContext"]["capabilities"].is_null()); } /// Companion: without binary-aware policy, UID 0 is not exempt, so the init diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 5d8f8870de..00f7a05516 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -153,7 +153,8 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | cni.chainDir | string | `""` | Host Multus vendor-cni-chain directory. Only used when mode is "multus-chain". The installer writes openshell-cni.conf here. | | cni.confDir | string | `"/etc/cni/net.d"` | Host CNI config directory. | | cni.configFile | string | `""` | Host CNI conflist filename patched by the installer. Empty selects the first non-OpenShell .conflist. | -| cni.enabled | bool | `false` | Install the OpenShell chained CNI plugin with a privileged node DaemonSet. Required when supervisor.topology is "cni-sidecar". | +| cni.enabled | bool | `false` | Install the OpenShell chained CNI plugin as a privileged node DaemonSet. The installer is a CLUSTER SINGLETON: one installation serves every OpenShell release's sandbox pods across all namespaces (enforcement is keyed on pod annotations the gateway sets). Enable it in exactly one release per cluster. Required for supervisor.topology="cni-sidecar" unless cni.external=true. | +| cni.external | bool | `false` | Use a CNI singleton installed by another release/chart instead of installing one here. Set this (with supervisor.topology="cni-sidecar" and cni.enabled=false) on additional gateway releases that share the cluster's existing OpenShell CNI installation. | | cni.image.pullPolicy | string | `""` | CNI installer image pull policy. Empty uses supervisor.image.pullPolicy, then image.pullPolicy. | | cni.image.repository | string | `""` | CNI installer image repository. Empty falls back to supervisor.image.repository, then the official supervisor repository. | | cni.image.tag | string | `""` | CNI installer image tag. Empty falls back to supervisor.image.tag, then image.tag, then chart appVersion. | diff --git a/deploy/helm/openshell/examples/cni-sidecar-k3s.yaml b/deploy/helm/openshell/examples/cni-sidecar-k3s.yaml new file mode 100644 index 0000000000..acbfbf8ed8 --- /dev/null +++ b/deploy/helm/openshell/examples/cni-sidecar-k3s.yaml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Example values for the cni-sidecar topology on vanilla Kubernetes / k3s. +# +# The OpenShell CNI installer is a CLUSTER SINGLETON: one installation enforces +# every OpenShell release's sandbox pods across all namespaces. Enable it in +# exactly one release per cluster. +# +# Copy and adjust the host CNI paths for your distro, then: +# helm install openshell deploy/helm/openshell -n openshell \ +# -f deploy/helm/openshell/examples/cni-sidecar-k3s.yaml + +cni: + enabled: true + # conflist (default) appends the OpenShell plugin to an existing CNI .conflist. + mode: conflist + # k3s configures containerd with k3s-specific CNI paths. On upstream Kubernetes + # the defaults (/opt/cni/bin, /etc/cni/net.d) usually apply — drop these two. + binDir: /bin + confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d + +supervisor: + topology: cni-sidecar diff --git a/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml b/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml new file mode 100644 index 0000000000..9995479cb1 --- /dev/null +++ b/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml @@ -0,0 +1,23 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Example values for an ADDITIONAL gateway release on a cluster that already has +# the OpenShell CNI singleton installed by another release. This release does not +# install its own CNI; it reuses the cluster's existing installation. Its sandbox +# pods are enforced by the shared plugin (enforcement is keyed on pod annotations +# the gateway sets) and gated on the same per-node readiness label. +# +# helm install openshell-team-b deploy/helm/openshell -n team-b \ +# -f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml + +cni: + # Do not install a second CNI installer; use the cluster singleton. + enabled: false + external: true + +supervisor: + topology: cni-sidecar + +sandboxServiceAccount: + openshift: + binaryAwareSCC: true diff --git a/deploy/helm/openshell/examples/cni-sidecar-openshift.yaml b/deploy/helm/openshell/examples/cni-sidecar-openshift.yaml new file mode 100644 index 0000000000..6be2a659ad --- /dev/null +++ b/deploy/helm/openshell/examples/cni-sidecar-openshift.yaml @@ -0,0 +1,39 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Example values for the cni-sidecar topology on OpenShift (Multus / OVN-K). +# +# The OpenShell CNI installer is a CLUSTER SINGLETON: one installation enforces +# every OpenShell release's sandbox pods across all namespaces. Enable it +# (cni.enabled=true) in exactly one release per cluster. Additional gateway +# releases that share it set cni.enabled=false and cni.external=true (see +# cni-sidecar-openshift-extra-release.yaml). +# +# Copy and adjust, then: +# helm install openshell deploy/helm/openshell -n openshell \ +# -f deploy/helm/openshell/examples/cni-sidecar-openshift.yaml + +cni: + enabled: true + # OpenShift's default network is managed by the Cluster Network Operator; there + # is no appendable CNI .conflist, so chain the plugin via Multus instead. + mode: multus-chain + # OpenShift cri-o resolves plugins from /var/lib/cni/bin (/opt/cni/bin is absent). + binDir: /var/lib/cni/bin + # Multus vendor-cni-chain directory (tmpfs; the installer re-asserts it). + chainDir: /run/multus/cni/net.d/vendor-cni-chain + # Persistent directory for plugin credentials (chainDir is tmpfs). + stateDir: /etc/kubernetes/cni/openshell + openshift: + # Grant the privileged SCC to the dedicated CNI ServiceAccount only. + privilegedSCC: true + +supervisor: + topology: cni-sidecar + +sandboxServiceAccount: + openshift: + # The network sidecar runs as UID 0 with SYS_PTRACE + DAC_READ_SEARCH for + # binary-aware policy; restricted-v2 forbids this, so grant the sandbox SA a + # minimal purpose-built SCC. + binaryAwareSCC: true diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 93a572ecbb..a487f24bbf 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -87,12 +87,24 @@ Create the name of the service account to use {{- end }} {{/* -Name of the ServiceAccount used by the CNI installer DaemonSet. Dedicated so -the privileged SCC and pod-get RBAC stay scoped to the CNI installer rather -than the shared gateway service account. +Cluster-singleton name for the CNI installer and its RBAC/SCC. The CNI installer +is a per-cluster singleton (one chained plugin serves every OpenShell release's +sandbox pods, keyed on pod annotations), so its name is release-independent +(chart name, not fullname). This lets additional gateway releases share one CNI +installation and keeps the host CNI config, ClusterRole, and readiness label +unambiguous across releases. +*/}} +{{- define "openshell.cniName" -}} +{{- printf "%s-cni" (include "openshell.name" .) | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +ServiceAccount used by the CNI installer DaemonSet. Same singleton name so the +privileged SCC and CNI RBAC stay scoped to the CNI installer rather than the +shared gateway service account. */}} {{- define "openshell.cniServiceAccountName" -}} -{{- printf "%s-cni" (include "openshell.fullname" .) | trunc 63 | trimSuffix "-" }} +{{- include "openshell.cniName" . }} {{- end }} {{/* diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml index 398111640d..e015ea204d 100644 --- a/deploy/helm/openshell/templates/cni-daemonset.yaml +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -2,8 +2,8 @@ # SPDX-License-Identifier: Apache-2.0 {{- $mode := .Values.cni.mode | default "conflist" -}} -{{- if and (eq (.Values.supervisor.topology | default "combined") "cni-sidecar") (not .Values.cni.enabled) -}} -{{- fail "supervisor.topology=cni-sidecar requires cni.enabled=true so the OpenShell chained CNI plugin is installed on every node" -}} +{{- if and (eq (.Values.supervisor.topology | default "combined") "cni-sidecar") (not .Values.cni.enabled) (not .Values.cni.external) -}} +{{- fail "supervisor.topology=cni-sidecar requires the OpenShell CNI installer: set cni.enabled=true to install the cluster singleton here, or cni.external=true if it is installed by another release" -}} {{- end }} {{- if .Values.cni.enabled }} {{- if not (has $mode (list "conflist" "multus-chain")) -}} @@ -22,7 +22,7 @@ apiVersion: apps/v1 kind: DaemonSet metadata: - name: {{ include "openshell.fullname" . }}-cni + name: {{ include "openshell.cniName" . }} labels: {{- include "openshell.cniLabels" . | nindent 4 }} spec: @@ -55,7 +55,7 @@ spec: # Used by `openshell-cni daemonset-active` so the preStop hook can tell # an ordinary pod restart from a real teardown (helm uninstall). - name: DS_NAME - value: {{ include "openshell.fullname" . }}-cni + value: {{ include "openshell.cniName" . }} - name: DS_NAMESPACE value: {{ .Release.Namespace | quote }} command: @@ -75,9 +75,10 @@ spec: touch "${host_log_file}" chmod 0644 "${host_log_file}" install -m 0755 /openshell-cni "${bin_dir}/openshell-cni" - cp /var/run/secrets/kubernetes.io/serviceaccount/ca.crt "${ca_file}" - cp /var/run/secrets/kubernetes.io/serviceaccount/token "${token_file}" - chmod 0600 "${token_file}" + # Create the token atomically at mode 0600 (install, not cp+chmod) so + # this node-patch-capable credential is never briefly world-readable. + install -m 0644 /var/run/secrets/kubernetes.io/serviceaccount/ca.crt "${ca_file}" + install -m 0600 /var/run/secrets/kubernetes.io/serviceaccount/token "${token_file}" cat > "${kubeconfig}" < "${plugin_config}" <&2 } + # Fencing (clearing readiness) is security-critical: while enforcement + # is not in place the node must not advertise readiness. Retry a few + # times so a transient API error does not silently leave stale + # readiness; log loudly if it ultimately fails. mark_unready() { - "${openshell_cni}" node-ready --clear \ - || echo "OpenShell CNI: failed to mark node unready" >&2 + _i=0 + while [ "${_i}" -lt 5 ]; do + if "${openshell_cni}" node-ready --clear; then + return 0 + fi + _i=$((_i + 1)) + sleep 2 + done + echo "OpenShell CNI: FAILED to fence node after retries; readiness may be stale" >&2 + return 1 } {{- if eq $mode "conflist" }} conf_dir="/host{{ .Values.cni.confDir }}" @@ -140,17 +156,20 @@ spec: find "${conf_dir}" -maxdepth 1 -type f -name '*.conflist' ! -name '*openshell*' | sort | head -n 1 {{- end }} } - # True when the target already carries our openshell-cni plugin - # (matching owner). Used to decide whether enforcement is in place. + # True when the target already carries our openshell-cni plugin at the + # desired config version. Readiness is bound to configVersion so a + # stale same-owner entry (e.g. after a config change) is treated as + # unhealthy and repaired before the node is re-marked ready. conflist_ours() { _t="$1" - [ -n "${_t}" ] && [ -f "${_t}" ] && jq -e --arg o "${owner}" \ - 'any((.plugins // [])[]; .type == "openshell-cni" and .openshell.owner == $o)' \ + [ -n "${_t}" ] && [ -f "${_t}" ] && jq -e --arg o "${owner}" --arg v "${config_version}" \ + 'any((.plugins // [])[]; .type == "openshell-cni" and .openshell.owner == $o and .openshell.configVersion == $v)' \ "${_t}" >/dev/null 2>&1 } - # True when the target carries an openshell-cni plugin owned by a - # DIFFERENT release. We must never overwrite it (that would silently - # unrestrict the other release's sandboxes). + # True when the target carries an openshell-cni plugin with a + # non-OpenShell owner. The installer is a cluster singleton owned by + # "openshell"; refuse to overwrite an entry owned by anything else + # (fail closed rather than clobber a foreign plugin of the same type). conflist_foreign() { _t="$1" [ -n "${_t}" ] && [ -f "${_t}" ] && jq -e --arg o "${owner}" \ @@ -164,7 +183,7 @@ spec: return 1 fi if conflist_foreign "${target}"; then - echo "OpenShell CNI refuses to overwrite a chained plugin owned by another release in ${target}; only one OpenShell release per cluster is supported for cni-sidecar" >&2 + echo "OpenShell CNI refuses to overwrite a chained plugin with a non-OpenShell owner in ${target}" >&2 return 1 fi if [ ! -f "${target}.openshell-backup" ]; then @@ -186,6 +205,12 @@ spec: return 1 fi } + # Fence before the initial patch too: if a stale readiness label + # survives from a previous incarnation and this patch fails, the node + # must not keep advertising readiness. + if ! conflist_ours "$(resolve_conflist_target)"; then + mark_unready + fi patch_conflist || exit 1 mark_ready echo "OpenShell CNI installed (conflist mode); plugin log ${log_file}; firewall backend selected during CNI ADD" @@ -194,17 +219,18 @@ spec: chain_conf="${chain_dir}/openshell-cni.conf" mkdir -p "${chain_dir}" # True only when the chain conf is a well-formed openshell-cni config - # owned by this release. Mere file existence is not enough — an empty, - # corrupt, stale, or foreign file must NOT mark the node ready. + # at the desired config version. Mere file existence is not enough — + # an empty, corrupt, stale-version, or foreign file must NOT mark the + # node ready. chain_conf_healthy() { - [ -f "${chain_conf}" ] && jq -e --arg o "${owner}" ' + [ -f "${chain_conf}" ] && jq -e --arg o "${owner}" --arg v "${config_version}" ' .type == "openshell-cni" and .openshell.owner == $o + and .openshell.configVersion == $v and (.openshell.kubeconfig // "") != "" - and ((.openshell.sandboxNamespaces // []) | length) > 0 ' "${chain_conf}" >/dev/null 2>&1 } - # True when a chain conf owned by a different release is present. + # True when a chain conf with a non-OpenShell owner is present. chain_conf_foreign() { [ -f "${chain_conf}" ] && jq -e --arg o "${owner}" \ '.type == "openshell-cni" and (.openshell.owner // "") != $o' \ @@ -212,7 +238,7 @@ spec: } write_chain_conf() { if chain_conf_foreign; then - echo "OpenShell CNI refuses to overwrite ${chain_conf} owned by another release; only one OpenShell release per cluster is supported for cni-sidecar" >&2 + echo "OpenShell CNI refuses to overwrite ${chain_conf}: it carries a non-OpenShell owner" >&2 return 1 fi _tmp="$(mktemp "${chain_dir}/openshell-cni.XXXXXX")" @@ -227,6 +253,10 @@ spec: return 1 fi } + # Fence before the initial write too (see conflist path). + if ! chain_conf_healthy; then + mark_unready + fi write_chain_conf || exit 1 mark_ready echo "OpenShell CNI installed (multus-chain mode) at ${chain_conf}; plugin log ${log_file}; firewall backend selected during CNI ADD" diff --git a/deploy/helm/openshell/templates/cni-rbac.yaml b/deploy/helm/openshell/templates/cni-rbac.yaml index 3ec5fcbf6e..2fa2bd5030 100644 --- a/deploy/helm/openshell/templates/cni-rbac.yaml +++ b/deploy/helm/openshell/templates/cni-rbac.yaml @@ -6,57 +6,31 @@ apiVersion: v1 kind: ServiceAccount metadata: name: {{ include "openshell.cniServiceAccountName" . }} + namespace: {{ .Release.Namespace }} labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni + {{- include "openshell.cniLabels" . | nindent 4 }} --- +# The CNI installer is a cluster singleton: one chained plugin enforces every +# OpenShell release's sandbox pods (keyed on pod annotations) across all +# namespaces. Its RBAC is therefore cluster-scoped. It is still minimal: +# - pods get: read a sandbox pod's annotations during CNI ADD (any namespace, +# because the singleton serves all sandbox namespaces). +# - nodes get/patch: set/clear the per-node readiness label for the gate. +# - daemonsets get (this DaemonSet only): let preStop tell an ordinary restart +# from a real teardown. apiVersion: rbac.authorization.k8s.io/v1 -kind: Role +kind: ClusterRole metadata: - name: {{ include "openshell.cniServiceAccountName" . }} - namespace: {{ include "openshell.sandboxNamespace" . }} + name: {{ include "openshell.cniName" . }} labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni + {{- include "openshell.cniLabels" . | nindent 4 }} rules: - # The CNI plugin reads the sandbox pod's annotations during CNI ADD to - # decide whether to install bypass-prevention rules in the pod netns. - apiGroups: - "" resources: - pods verbs: - get ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: {{ include "openshell.cniServiceAccountName" . }} - namespace: {{ include "openshell.sandboxNamespace" . }} - labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: {{ include "openshell.cniServiceAccountName" . }} -subjects: - - kind: ServiceAccount - name: {{ include "openshell.cniServiceAccountName" . }} - namespace: {{ .Release.Namespace }} ---- -# Nodes are cluster-scoped, so the readiness-label grant must be a ClusterRole. -# It is deliberately minimal: only get/patch on nodes, so the installer can -# label its own node ready/unready for the sandbox scheduling gate. No create, -# delete, or list, and no other resources. -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: {{ include "openshell.cniServiceAccountName" . }} - labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni -rules: - apiGroups: - "" resources: @@ -64,28 +38,25 @@ rules: verbs: - get - patch - # get on this DaemonSet only, so the preStop hook can tell an ordinary restart - # from a real teardown. resourceNames keeps the grant scoped to our object. - apiGroups: - apps resources: - daemonsets resourceNames: - - {{ include "openshell.fullname" . }}-cni + - {{ include "openshell.cniName" . }} verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: {{ include "openshell.cniServiceAccountName" . }} + name: {{ include "openshell.cniName" . }} labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni + {{- include "openshell.cniLabels" . | nindent 4 }} roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole - name: {{ include "openshell.cniServiceAccountName" . }} + name: {{ include "openshell.cniName" . }} subjects: - kind: ServiceAccount name: {{ include "openshell.cniServiceAccountName" . }} diff --git a/deploy/helm/openshell/templates/cni-scc.yaml b/deploy/helm/openshell/templates/cni-scc.yaml index 62f7aa8dc5..ef9bd8643e 100644 --- a/deploy/helm/openshell/templates/cni-scc.yaml +++ b/deploy/helm/openshell/templates/cni-scc.yaml @@ -5,10 +5,9 @@ apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: - name: {{ include "openshell.fullname" . }}-cni-scc + name: {{ include "openshell.cniName" . }}-scc labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni + {{- include "openshell.cniLabels" . | nindent 4 }} rules: - apiGroups: - security.openshift.io @@ -22,14 +21,13 @@ rules: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: {{ include "openshell.fullname" . }}-cni-scc + name: {{ include "openshell.cniName" . }}-scc labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni + {{- include "openshell.cniLabels" . | nindent 4 }} roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole - name: {{ include "openshell.fullname" . }}-cni-scc + name: {{ include "openshell.cniName" . }}-scc subjects: - kind: ServiceAccount name: {{ include "openshell.cniServiceAccountName" . }} diff --git a/deploy/helm/openshell/tests/cni_daemonset_test.yaml b/deploy/helm/openshell/tests/cni_daemonset_test.yaml index 62673c23c0..432ae169c5 100644 --- a/deploy/helm/openshell/tests/cni_daemonset_test.yaml +++ b/deploy/helm/openshell/tests/cni_daemonset_test.yaml @@ -41,9 +41,13 @@ tests: content: name: cni-log mountPath: /host/var/log + # Cluster singleton: annotation-driven, empty namespace allowlist. - matchRegex: path: spec.template.spec.containers[0].command[2] - pattern: '"sandboxNamespaces": \["NAMESPACE"\]' + pattern: '"sandboxNamespaces": \[\]' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'owner="openshell"' - matchRegex: path: spec.template.spec.containers[0].command[2] pattern: '"logFile": "\$\{log_file\}"' @@ -105,23 +109,44 @@ tests: path: spec.template.spec.containers[0].command[2] pattern: 'refuses to overwrite' - - it: scopes cni annotation lookup to explicit sandbox namespace + - it: binds readiness to a config version + template: templates/cni-daemonset.yaml + set: + cni.enabled: true + asserts: + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'config_version=' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '"configVersion": "' + + - it: creates the host token atomically at mode 0600 template: templates/cni-daemonset.yaml set: cni.enabled: true - server.sandboxNamespace: agents asserts: - matchRegex: path: spec.template.spec.containers[0].command[2] - pattern: '"sandboxNamespaces": \["agents"\]' + pattern: 'install -m 0600 /var/run/secrets/kubernetes.io/serviceaccount/token' - - it: fails cni-sidecar topology when cni installer is disabled + - it: fails cni-sidecar topology when no CNI installer is available template: templates/cni-daemonset.yaml set: supervisor.topology: cni-sidecar asserts: - failedTemplate: - errorMessage: supervisor.topology=cni-sidecar requires cni.enabled=true so the OpenShell chained CNI plugin is installed on every node + errorMessage: "supervisor.topology=cni-sidecar requires the OpenShell CNI installer: set cni.enabled=true to install the cluster singleton here, or cni.external=true if it is installed by another release" + + - it: allows cni-sidecar with an external singleton and renders no installer + template: templates/cni-daemonset.yaml + set: + supervisor.topology: cni-sidecar + cni.enabled: false + cni.external: true + asserts: + - hasDocuments: + count: 0 - it: renders multus-chain conf and state volumes when mode is multus-chain template: templates/cni-daemonset.yaml diff --git a/deploy/helm/openshell/tests/cni_rbac_test.yaml b/deploy/helm/openshell/tests/cni_rbac_test.yaml index 471a0930f3..3ae2cc4119 100644 --- a/deploy/helm/openshell/tests/cni_rbac_test.yaml +++ b/deploy/helm/openshell/tests/cni_rbac_test.yaml @@ -12,22 +12,29 @@ tests: - hasDocuments: count: 0 - - it: renders service account, role and binding when cni enabled + # The CNI installer is a cluster singleton, so its RBAC is a ServiceAccount + # plus a cluster-scoped ClusterRole/ClusterRoleBinding (no namespaced Role). + - it: renders singleton service account and cluster-scoped rbac when cni enabled set: cni.enabled: true asserts: - hasDocuments: - count: 5 + count: 3 - documentIndex: 0 isKind: of: ServiceAccount - documentIndex: 0 - matchRegex: + equal: path: metadata.name - pattern: -cni$ + value: openshell-cni - documentIndex: 1 isKind: - of: Role + of: ClusterRole + - documentIndex: 1 + equal: + path: metadata.name + value: openshell-cni + # pods get is cluster-wide because one installer serves all sandbox namespaces. - documentIndex: 1 contains: path: rules @@ -38,23 +45,7 @@ tests: - pods verbs: - get - - documentIndex: 2 - isKind: - of: RoleBinding - - documentIndex: 2 - matchRegex: - path: subjects[0].name - pattern: -cni$ - - documentIndex: 2 - matchRegex: - path: roleRef.name - pattern: -cni$ - # ClusterRole/ClusterRoleBinding grant the minimal cluster-scoped node - # get/patch the installer needs to set the readiness label. - - documentIndex: 3 - isKind: - of: ClusterRole - - documentIndex: 3 + - documentIndex: 1 contains: path: rules content: @@ -65,8 +56,8 @@ tests: verbs: - get - patch - # daemonsets get, scoped by resourceNames to this release's DaemonSet. - - documentIndex: 3 + # daemonsets get, scoped by resourceNames to the singleton DaemonSet. + - documentIndex: 1 contains: path: rules content: @@ -78,24 +69,27 @@ tests: - openshell-cni verbs: - get - - documentIndex: 4 + - documentIndex: 2 isKind: of: ClusterRoleBinding - - documentIndex: 4 - matchRegex: + - documentIndex: 2 + equal: path: roleRef.name - pattern: -cni$ + value: openshell-cni + - documentIndex: 2 + equal: + path: subjects[0].name + value: openshell-cni - - it: scopes cni pod-get role to the explicit sandbox namespace + # The singleton name is release-independent (chart name), so a different + # release name still yields the fixed cluster-singleton name. + - it: uses a release-independent singleton name + release: + name: my-gateway set: cni.enabled: true - server.sandboxNamespace: agents asserts: - documentIndex: 1 equal: - path: metadata.namespace - value: agents - - documentIndex: 2 - equal: - path: metadata.namespace - value: agents + path: metadata.name + value: openshell-cni diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index ac3c39c4e2..a0807c2a90 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -66,9 +66,17 @@ supervisor: processBinaryAwareNetworkPolicy: true cni: - # -- Install the OpenShell chained CNI plugin with a privileged node DaemonSet. - # Required when supervisor.topology is "cni-sidecar". + # -- Install the OpenShell chained CNI plugin as a privileged node DaemonSet. + # The installer is a CLUSTER SINGLETON: one installation serves every OpenShell + # release's sandbox pods across all namespaces (enforcement is keyed on pod + # annotations the gateway sets). Enable it in exactly one release per cluster. + # Required for supervisor.topology="cni-sidecar" unless cni.external=true. enabled: false + # -- Use a CNI singleton installed by another release/chart instead of + # installing one here. Set this (with supervisor.topology="cni-sidecar" and + # cni.enabled=false) on additional gateway releases that share the cluster's + # existing OpenShell CNI installation. + external: false # -- CNI installer strategy. "conflist" appends the OpenShell plugin to an # existing CNI .conflist (k3s / vanilla). "multus-chain" writes a standalone # plugin .conf into a Multus vendor-cni-chain subdirectory (OpenShift), which diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 7512eaa65e..d216889868 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -87,6 +87,63 @@ openshell gateway add http://127.0.0.1:8080 --local --name openshift openshell status ``` +## Enable the cni-sidecar topology + +The `cni-sidecar` topology moves pod-network egress enforcement out of the +sandbox pod and onto a node-level CNI plugin, removing the per-pod privileged +network-init container. See [Supervisor Topology](/kubernetes/topology) for the +model. On OpenShift it uses Multus `multus-chain` mode plus purpose-built SCCs. + +The CNI installer is a **cluster singleton**: one installation enforces every +OpenShell release's sandbox pods across all namespaces. Enable it in exactly one +release per cluster. + + + +### Install with the cni-sidecar example values + +```shell +helm install openshell deploy/helm/openshell \ + --namespace openshell \ + -f deploy/helm/openshell/examples/cni-sidecar-openshift.yaml +``` + +The example sets `cni.mode: multus-chain`, the OpenShift CNI paths, and the two +SCC grants (`cni.openshift.privilegedSCC`, `sandboxServiceAccount.openshift.binaryAwareSCC`). +It requires a Multus thick daemon with `auxiliaryCNIChainName` configured +(default on OpenShift 4.x). + +### Verify the CNI installer is Ready on every node + +```shell +oc -n openshell rollout status daemonset/openshell-cni +oc get nodes -L openshell.ai/cni-ready +``` + +Every schedulable node should show `cni-ready=true`. Sandbox pods carry a +required `nodeAffinity` on this label, so they will not schedule onto a node +before its egress enforcement is active. + +### Create a sandbox and confirm enforcement + +```shell +openshell sandbox create --name probe -- sleep infinity +# The pod reaches 2/2 (agent + network sidecar) only on a cni-ready node: +oc get pod default--probe -n openshell +# Default-deny egress: a disallowed destination is blocked. +oc exec default--probe -n openshell -c agent -- \ + sh -c 'curl -sS -m 10 https://example.com; echo "exit=$?"' +``` + +A blocked request (connection/resolution failure) confirms the sandbox's egress +is routed through the OpenShell network sidecar and enforced by policy. + + + +To run **additional** gateway releases on the same cluster, do not install a +second CNI. Use `-f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml` +(`cni.enabled=false`, `cni.external=true`), which reuses the cluster singleton. + ## Next Steps - For TLS-enabled deployments, refer to [Managing Certificates](/kubernetes/managing-certificates). diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index 1e74b77e86..beb793725c 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -233,13 +233,15 @@ runtime must honor the CNI-installed pod-network rules for this topology to provide the expected network enforcement. - -Install only **one** OpenShell release per cluster when using `cni-sidecar`. The -chained CNI plugin and the readiness label are cluster-global host state; the -installer stamps an owner on its plugin entry and refuses to overwrite another -release's entry, so a second release (or a `sandboxNamespace`-changing upgrade) -fails closed rather than silently unrestricting sandboxes. - + +The CNI installer is a **cluster singleton**. Its chained plugin enforces any +pod carrying the OpenShell annotations (set only by a gateway on its own sandbox +pods), so a single installation serves every OpenShell release across all +namespaces. Enable the installer (`cni.enabled=true`) in exactly one release per +cluster; additional gateway releases set `cni.enabled=false` and +`cni.external=true` to share it. The installer refuses to overwrite a chained +plugin entry with a non-OpenShell owner (fail closed). + ## Credential Exposure @@ -357,11 +359,12 @@ injects the OpenShell plugin through the Multus `vendor-cni-chain` auxiliary cha without modifying any operator-managed file. The installer also needs the `privileged` SecurityContextConstraints. -Install with the provided overlay: +Install with the example values (a step-by-step OpenShift walkthrough with +verification is in [OpenShift](/kubernetes/openshift#enable-the-cni-sidecar-topology)): ```shell helm install openshell deploy/helm/openshell \ - -f deploy/helm/openshell/ci/values-cni-sidecar-openshift.yaml + -f deploy/helm/openshell/examples/cni-sidecar-openshift.yaml ``` This sets `cni.binDir: /var/lib/cni/bin`, `cni.chainDir: diff --git a/rfc/cni-sidecar-topology-DRAFT.md b/rfc/cni-sidecar-topology-DRAFT.md index 78699fc5ad..41c0a92522 100644 --- a/rfc/cni-sidecar-topology-DRAFT.md +++ b/rfc/cni-sidecar-topology-DRAFT.md @@ -402,15 +402,18 @@ behavior is behind a default-off value. `cni-ready=true` until the pod is rescheduled and the next reconcile re-checks. Operators diagnose via `kubectl logs daemonset/openshell-cni` and the tailed plugin log. -- **Single release per cluster.** The chained plugin and readiness label are - cluster-global host state keyed to one `sandboxNamespaces` list, so two - OpenShell releases (or a `sandboxNamespace`-changing upgrade) would otherwise - overwrite each other and leave one release's sandboxes unenforced. Mitigation: - the installer stamps an owner (`/`) on its plugin entry and - refuses to overwrite an entry owned by a different release (fail closed); only - one OpenShell release per cluster is supported for `cni-sidecar` today. - Aggregating multiple releases (a cluster-singleton installer, or release-scoped - config and readiness ownership) is future work. +- **Cluster-singleton CNI.** The chained plugin and readiness label are + cluster-global host state. Rather than key them to a per-release namespace list + (which two releases would fight over), the installer is a cluster singleton: the + plugin enforces any pod carrying the OpenShell annotations — set only by a + gateway on its own sandbox pods — so one installation serves every release + across all namespaces (empty namespace allowlist, fixed `openshell` owner, + release-independent resource names). Enable it in one release per cluster; + additional gateway releases set `cni.enabled=false` + `cni.external=true` to + share it. Readiness is bound to a `configVersion` so a stale entry is repaired + before the node is re-marked ready, and the installer refuses to overwrite a + chained plugin entry with a non-OpenShell owner (fail closed). The singleton's + `pods get` RBAC is necessarily cluster-scoped. - **Dependency on Multus aux chain.** `multus-chain` requires the Multus thick daemon to have `auxiliaryCNIChainName` set. It is default on OpenShift 4.x but not guaranteed everywhere. Mitigation: document the requirement; the chart does From 98e2521b9d1d44c08b9500b9bf296bcb31e64e8c Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Tue, 4 Aug 2026 09:37:03 -0400 Subject: [PATCH 15/20] fix(cni-sidecar): drop brittle owner guard, upgrade any openshell-cni entry in place MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The singleton installer refused to overwrite an openshell-cni chained plugin entry whose owner value differed from the fixed "openshell" owner. That guard was redundant — the conflist patch already preserves every non-openshell-cni plugin and replaces only the openshell-cni entry, and the plugin type is itself the ownership signal — and it broke in-place upgrades from an earlier owner format (e.g. "/"), crash-looping the installer. Treat any existing openshell-cni entry as ours to upgrade in place. Signed-off-by: Russell Bryant --- .../openshell/templates/cni-daemonset.yaml | 34 ++++++------------- .../openshell/tests/cni_daemonset_test.yaml | 7 ++-- 2 files changed, 12 insertions(+), 29 deletions(-) diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml index e015ea204d..59ea069faf 100644 --- a/deploy/helm/openshell/templates/cni-daemonset.yaml +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -166,26 +166,18 @@ spec: 'any((.plugins // [])[]; .type == "openshell-cni" and .openshell.owner == $o and .openshell.configVersion == $v)' \ "${_t}" >/dev/null 2>&1 } - # True when the target carries an openshell-cni plugin with a - # non-OpenShell owner. The installer is a cluster singleton owned by - # "openshell"; refuse to overwrite an entry owned by anything else - # (fail closed rather than clobber a foreign plugin of the same type). - conflist_foreign() { - _t="$1" - [ -n "${_t}" ] && [ -f "${_t}" ] && jq -e --arg o "${owner}" \ - 'any((.plugins // [])[]; .type == "openshell-cni" and (.openshell.owner // "") != $o)' \ - "${_t}" >/dev/null 2>&1 - } patch_conflist() { target="$(resolve_conflist_target)" if [ -z "${target}" ] || [ ! -f "${target}" ]; then echo "OpenShell CNI requires an existing CNI .conflist in ${conf_dir}" >&2 return 1 fi - if conflist_foreign "${target}"; then - echo "OpenShell CNI refuses to overwrite a chained plugin with a non-OpenShell owner in ${target}" >&2 - return 1 - fi + # The patch below preserves every non-openshell-cni plugin and + # replaces only the openshell-cni entry, so it never clobbers a + # foreign plugin. The plugin type is itself the ownership signal + # (the cluster singleton always writes owner "openshell"), so any + # existing openshell-cni entry — including one from an older owner + # format — is ours to upgrade in place. if [ ! -f "${target}.openshell-backup" ]; then cp "${target}" "${target}.openshell-backup" fi @@ -230,17 +222,11 @@ spec: and (.openshell.kubeconfig // "") != "" ' "${chain_conf}" >/dev/null 2>&1 } - # True when a chain conf with a non-OpenShell owner is present. - chain_conf_foreign() { - [ -f "${chain_conf}" ] && jq -e --arg o "${owner}" \ - '.type == "openshell-cni" and (.openshell.owner // "") != $o' \ - "${chain_conf}" >/dev/null 2>&1 - } write_chain_conf() { - if chain_conf_foreign; then - echo "OpenShell CNI refuses to overwrite ${chain_conf}: it carries a non-OpenShell owner" >&2 - return 1 - fi + # openshell-cni.conf is an OpenShell-owned file (fixed name, our + # plugin type); the cluster singleton always writes owner + # "openshell", so overwriting an existing one — including from an + # older owner format — is an in-place upgrade, not a clobber. _tmp="$(mktemp "${chain_dir}/openshell-cni.XXXXXX")" # Same fail-closed contract as patch_conflist: never install a # truncated chain config if jq fails. diff --git a/deploy/helm/openshell/tests/cni_daemonset_test.yaml b/deploy/helm/openshell/tests/cni_daemonset_test.yaml index 432ae169c5..4e7d99143b 100644 --- a/deploy/helm/openshell/tests/cni_daemonset_test.yaml +++ b/deploy/helm/openshell/tests/cni_daemonset_test.yaml @@ -101,13 +101,10 @@ tests: - matchRegex: path: spec.template.spec.containers[0].lifecycle.preStop.exec.command[2] pattern: 'daemonset-active' - # Ownership stamp + fence-before-repair reconcile. + # Singleton owner stamp. - matchRegex: path: spec.template.spec.containers[0].command[2] - pattern: '"owner": "' - - matchRegex: - path: spec.template.spec.containers[0].command[2] - pattern: 'refuses to overwrite' + pattern: 'owner="openshell"' - it: binds readiness to a config version template: templates/cni-daemonset.yaml From 5c195606c337f7aaa388b84d12d20196ad33b54e Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Tue, 4 Aug 2026 10:21:20 -0400 Subject: [PATCH 16/20] fix(cni-sidecar): harden sidecar init, scope API lookups, gate reboots MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address the fifth review round of the cni-sidecar topology: - driver: harden the untrusted workspace-init in EVERY sidecar topology, not only the binary-aware case. A non-binary-aware Sidecar/CniSidecar init still needs UID 0 but now drops all capabilities and disables privilege escalation, so it cannot CAP_SETUID to the exempt proxy UID or use CAP_NET_RAW to bypass egress. Only the Combined topology (no separate exempt sidecar) keeps default root capabilities for DAC read access. - cni/helm: add cni.sandboxNamespaces (default: the release's sandbox namespace). The plugin passes through pods in unlisted namespaces WITHOUT a Kubernetes API lookup, so control-plane/RBAC problems no longer block unrelated workloads' pod creation. Multi-release deployments list every sandbox namespace on the singleton. Bound into configVersion. - cni/helm: close the node-reboot fail-open window. openshell-cni node-ready now removes a boot-time NoSchedule taint (openshell.ai/cni-not-ready) once enforcement is ready (optimistic-concurrency JSON patch); the DaemonSet tolerates it. Boot-time application is operator node config — an OpenShift MachineConfig example ships under examples/ with caveats. conflist mode is disk-backed and unaffected by reboot. - helm: make the CNI singleton name a genuinely fixed constant (openshell-cni), not derived from nameOverride, so releases cannot install competing singletons over shared host state. - docs: document cni.external and the singleton in setup.mdx and the debug-openshell-cluster skill; update the RFC and architecture docs. Signed-off-by: Russell Bryant --- .../skills/debug-openshell-cluster/SKILL.md | 15 +++ architecture/compute-runtimes.md | 11 +- crates/openshell-cni/src/lib.rs | 110 ++++++++++++++++-- .../openshell-driver-kubernetes/src/driver.rs | 74 +++++++----- deploy/helm/openshell/README.md | 1 + .../cni-sidecar-openshift-boot-taint.yaml | 63 ++++++++++ deploy/helm/openshell/templates/_helpers.tpl | 17 +-- .../openshell/templates/cni-daemonset.yaml | 35 ++++-- .../openshell/tests/cni_daemonset_test.yaml | 25 +++- deploy/helm/openshell/values.yaml | 7 ++ docs/kubernetes/openshift.mdx | 10 ++ docs/kubernetes/setup.mdx | 3 +- rfc/cni-sidecar-topology-DRAFT.md | 17 ++- 13 files changed, 324 insertions(+), 64 deletions(-) create mode 100644 deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index d6bbf3c9ff..91745bad8b 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -408,6 +408,21 @@ sandbox pod should carry `openshell.ai/cni=enabled`, network setup, inspect the DaemonSet logs, the host CNI config, and whether the cluster actually invokes chained CNI plugins for the sandbox runtime class. +The CNI installer is a **cluster singleton** with a fixed identity +(`openshell-cni`): one installation serves every OpenShell release across all +namespaces. It is not necessarily in the release you are debugging — the owner is +the release with `cni.enabled=true`; other gateway releases set `cni.external=true` +and reuse it. Locate the owner and its resources cluster-wide: + +```bash +kubectl get daemonset -A -l app.kubernetes.io/name=openshell-cni +kubectl get clusterrole,clusterrolebinding openshell-cni +``` + +A gateway release using `cni-sidecar` with neither `cni.enabled` nor +`cni.external` fails to render (template error), so if the gateway installed but +sandboxes are unenforced, confirm the singleton exists and is Ready. + A per-node scheduling gate can also keep cni-sidecar sandbox pods `Pending`. The CNI DaemonSet labels each node `openshell.ai/cni-ready=true` after installing the plugin, and the gateway sets a required `nodeAffinity` on that label. If a diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index b3958329e4..027857cf18 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -280,7 +280,16 @@ node ready once the plugin is healthy again. The gateway sets a required schedule onto a node before that node's egress enforcement is active, and a node whose enforcement later breaks stops accepting new sandbox pods. The label is set through a minimal cluster-scoped grant (`nodes` `get`/`patch`, plus `get` on the -installer's own DaemonSet) bound to the dedicated CNI ServiceAccount. One residual +installer's own DaemonSet) bound to the dedicated CNI ServiceAccount. + +The persistent label cannot by itself cover a node reboot that wipes tmpfs-backed +enforcement (`multus-chain` stores the chain file under `/run`). A boot-time +`NoSchedule` taint (`openshell.ai/cni-not-ready`) closes that window: operator +node config applies it at boot (only node config runs before the scheduler; a +MachineConfig example ships under `deploy/helm/openshell/examples/`), the CNI +DaemonSet tolerates it, and the installer removes it once enforcement is ready +(never re-adding it, so a transient unready does not over-repel). `conflist` mode +keeps the plugin on persistent disk and is unaffected by reboot. One residual limitation: an ungraceful DaemonSet pod deletion (no `preStop`) leaves a stale `cni-ready=true` until the pod is rescheduled and the next reconcile tick re-evaluates it. diff --git a/crates/openshell-cni/src/lib.rs b/crates/openshell-cni/src/lib.rs index 095d9e3128..88b9149543 100644 --- a/crates/openshell-cni/src/lib.rs +++ b/crates/openshell-cni/src/lib.rs @@ -25,6 +25,15 @@ const CNI_SIDECAR_NETWORK_ENFORCEMENT_MODE: &str = "cni-sidecar"; /// constant in `openshell-driver-kubernetes` (used for the sandbox pod /// nodeAffinity) and with the CNI `DaemonSet`'s node-patch RBAC. const NODE_READY_LABEL: &str = "openshell.ai/cni-ready"; +/// Node taint applied at boot (via an operator-provided `MachineConfig` / node +/// config) so a rebooted node repels workloads until egress enforcement is +/// re-established — the persistent `cni-ready` label cannot reflect a reboot that +/// wipes tmpfs-backed enforcement, so a boot-time taint closes that window. The +/// installer only REMOVES this taint (once enforcement is ready); it never adds +/// it, so a transient unready never over-repels a running node. Must stay in sync +/// with the taint key used by the `MachineConfig` and the `DaemonSet` toleration. +const NODE_NOT_READY_TAINT_KEY: &str = "openshell.ai/cni-not-ready"; +const NODE_NOT_READY_TAINT_EFFECT: &str = "NoSchedule"; #[allow(dead_code)] const OPENSHELL_TABLE: &str = "openshell_sidecar_bypass"; #[allow(dead_code)] @@ -218,8 +227,17 @@ pub fn node_ready(args: &[String]) -> Result<()> { let node = std::env::var("NODE_NAME") .map_err(|_| miette::miette!("NODE_NAME env var is required for node-ready"))?; let client = KubeApiClient::from_in_cluster()?; - let value = if ready { Some("true") } else { None }; - client.patch_node_label(&node, NODE_READY_LABEL, value) + if ready { + // Set the label first (still fenced by the boot taint if present), then + // remove the boot taint so scheduling opens only once both agree. + client.patch_node_label(&node, NODE_READY_LABEL, Some("true"))?; + client.remove_node_taint(&node, NODE_NOT_READY_TAINT_KEY, NODE_NOT_READY_TAINT_EFFECT) + } else { + // Clear the label to fence new sandbox pods. Do NOT add the taint here: + // the taint is boot-managed (it would over-repel all workloads on a + // transient unready); the label gate is sandbox-specific. + client.patch_node_label(&node, NODE_READY_LABEL, None) + } } /// Entry point for `openshell-cni daemonset-active`. @@ -408,12 +426,14 @@ fn workload_from_config( let Some(pod) = env.pod_ref() else { return Ok(None); }; - // sandbox_namespaces is an OPTIONAL allowlist. The cluster-singleton - // installer leaves it empty, which means "enforce any pod that carries the - // OpenShell annotations, in any namespace" — enforcement is gated by the - // per-pod annotations below (set only by a gateway on its own sandbox pods), - // so one installation safely serves every release. A non-empty list narrows - // enforcement to those namespaces. + // sandbox_namespaces is the allowlist of namespaces whose pods we inspect. + // A pod outside it is passed through immediately WITHOUT a Kubernetes API + // lookup — a blast-radius guard so control-plane/RBAC problems cannot block + // unrelated workloads' pod creation. Within the allowlist, enforcement is + // gated by the per-pod OpenShell annotations below (set only by a gateway on + // its own sandbox pods). The singleton installer populates this from + // cni.sandboxNamespaces (default: the release's sandbox namespace). An empty + // list means "any namespace" and is intentionally avoided by the chart. if !config.openshell.sandbox_namespaces.is_empty() && !config .openshell @@ -612,6 +632,80 @@ impl KubeApiClient { Ok(()) } + /// Removes the given taint (key + effect) from the node if present, using a + /// JSON Patch whose `test` on `metadata.resourceVersion` provides optimistic + /// concurrency so a concurrent taint change by another controller is not + /// clobbered (retried on conflict). A no-op when the taint is absent. + fn remove_node_taint(&self, node: &str, key: &str, effect: &str) -> Result<()> { + let url = format!("{}/api/v1/nodes/{}", self.server, node); + for _attempt in 0..5 { + let obj = self + .client + .get(&url) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to read node for taint removal")?; + if !obj.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while reading node {}", + obj.status(), + node + )); + } + let node_obj = obj.json::().into_diagnostic()?; + let resource_version = node_obj["metadata"]["resourceVersion"] + .as_str() + .unwrap_or_default() + .to_string(); + let taints = node_obj["spec"]["taints"].as_array(); + let filtered: Vec = taints + .map(|list| { + list.iter() + .filter(|t| { + !(t["key"].as_str() == Some(key) + && t["effect"].as_str() == Some(effect)) + }) + .cloned() + .collect() + }) + .unwrap_or_default(); + // Nothing to remove. + if taints.is_none_or(|list| list.len() == filtered.len()) { + return Ok(()); + } + let patch = serde_json::json!([ + { "op": "test", "path": "/metadata/resourceVersion", "value": resource_version }, + { "op": "replace", "path": "/spec/taints", "value": filtered }, + ]); + let response = self + .client + .patch(&url) + .bearer_auth(&self.token) + .header(reqwest::header::CONTENT_TYPE, "application/json-patch+json") + .body(serde_json::to_vec(&patch).into_diagnostic()?) + .send() + .into_diagnostic() + .wrap_err("failed to patch node taints")?; + let status = response.status().as_u16(); + // 409 (resourceVersion conflict) or 422 (test op failed) → retry. + if status == 409 || status == 422 { + continue; + } + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while removing taint from node {}", + response.status(), + node + )); + } + return Ok(()); + } + Err(miette::miette!( + "failed to remove taint from node {node} after retries (conflict)" + )) + } + /// Returns true when the named `DaemonSet` is gone (404) or has a /// `deletionTimestamp` set (being deleted). A non-404 HTTP error is /// propagated so the caller can treat it as indeterminate. diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index b4324027aa..b689023f43 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -2072,6 +2072,7 @@ fn apply_workspace_persistence( image_pull_policy: &str, sandbox_gid: u32, workspace_init_uid: u32, + harden_init: bool, ) { let Some(spec) = pod_template.get_mut("spec").and_then(|v| v.as_object_mut()) else { return; @@ -2156,20 +2157,24 @@ fn apply_workspace_persistence( // Callers pass the sandbox UID in that mode (non-exempt, still able to // seed the PVC via fsGroup) and 0 otherwise (where UID 0 is not exempt). // - // Hardening applies only when the init runs as the non-root exempt UID - // (the Sidecar/CniSidecar binary-aware case). There, dropping all - // capabilities and disabling privilege escalation stops a setuid binary - // in the untrusted image from regaining the exempt UID 0 and bypassing - // egress, and runAsNonRoot makes the kubelet refuse an image that would - // force UID 0. When the init runs as root (combined topology, where UID 0 - // is not exempt), keep the default root capabilities so it retains the - // broad DAC read access needed to seed the PVC from image files owned by - // other UIDs with restrictive modes. + // Harden the init in every sidecar topology (`harden_init`), not just the + // binary-aware case. A sidecar pod always has a separate network + // supervisor at an exempt UID and shares the pod netns, so a root init + // with default capabilities could switch to the exempt UID (CAP_SETUID) + // or craft packets around the L4 firewall (CAP_NET_RAW) to bypass egress. + // Dropping all capabilities and disabling privilege escalation closes + // both; runAsNonRoot is asserted only when the init already runs non-root + // (binary-aware), since the non-binary-aware sidecar init still needs + // UID 0. Only the Combined topology (no separate exempt sidecar) keeps + // default root capabilities, preserving broad DAC read access to seed the + // PVC from image files owned by other UIDs with restrictive modes. let mut security_context = serde_json::json!({ "runAsUser": workspace_init_uid }); - if workspace_init_uid != 0 { + if harden_init { security_context["allowPrivilegeEscalation"] = serde_json::json!(false); - security_context["runAsNonRoot"] = serde_json::json!(true); security_context["capabilities"] = serde_json::json!({ "drop": ["ALL"] }); + if workspace_init_uid != 0 { + security_context["runAsNonRoot"] = serde_json::json!(true); + } } let mut init_spec = serde_json::json!({ "name": WORKSPACE_INIT_CONTAINER_NAME, @@ -2812,19 +2817,19 @@ fn sandbox_template_to_k8s_with_validated_config( // that /sandbox data survives pod rescheduling. Skipped when the user // provides custom storage through driver_config. if inject_workspace { - // Under binary-aware network policy the CNI firewall exempts UID 0, so - // the untrusted workspace-init must not run as root or it could bypass - // egress policy. This exemption only exists in the sidecar topologies - // (Sidecar / CniSidecar), which run a separate network supervisor at the - // exempt UID; the combined topology has no such exemption, so it keeps - // root to preserve the widest read access to the image's /sandbox - // contents and PVC data. Run as the (non-exempt) sandbox UID only where - // the exemption applies. - let workspace_init_uid = if matches!( + // The sidecar topologies run a separate network supervisor at an exempt + // UID and share the pod netns, so their untrusted workspace-init must be + // hardened against egress bypass; the combined topology has no separate + // exempt sidecar and keeps default root capabilities for read access. + let harden_init = matches!( params.topology, SupervisorTopology::Sidecar | SupervisorTopology::CniSidecar - ) && params.process_binary_aware_network_policy - { + ); + // In binary-aware mode the CNI firewall exempts UID 0 itself, so the init + // must not run as root at all; run it as the non-exempt sandbox UID. + // Otherwise it runs as UID 0 but hardened (no CAP_SETUID/CAP_NET_RAW), so + // it cannot switch to the exempt proxy UID or craft raw packets. + let workspace_init_uid = if harden_init && params.process_binary_aware_network_policy { params.sandbox_uid } else { 0 @@ -2835,6 +2840,7 @@ fn sandbox_template_to_k8s_with_validated_config( params.image_pull_policy, params.sandbox_gid, workspace_init_uid, + harden_init, ); } @@ -5100,8 +5106,10 @@ mod tests { assert!(init["securityContext"]["capabilities"].is_null()); } - /// Companion: without binary-aware policy, UID 0 is not exempt, so the init - /// keeps root for the widest read access to the image's /sandbox contents. + /// Without binary-aware policy the sidecar init still runs as UID 0 (it needs + /// root), but must be hardened: a sidecar pod has a separate exempt network + /// supervisor, so the untrusted init must not keep `CAP_SETUID`/`CAP_NET_RAW` + /// (it could switch to the exempt UID or craft raw packets to bypass egress). #[test] fn non_binary_aware_workspace_init_runs_as_root() { let params = SandboxPodParams { @@ -5133,11 +5141,17 @@ mod tests { .iter() .find(|container| container["name"] == WORKSPACE_INIT_CONTAINER_NAME) .expect("workspace-init container must be present"); + let sc = &init["securityContext"]; assert_eq!( - init["securityContext"]["runAsUser"], + sc["runAsUser"], serde_json::json!(0), - "workspace-init keeps root when UID 0 is not CNI-exempt" + "non-binary-aware sidecar init still needs UID 0" ); + // Hardened even at UID 0: no CAP_SETUID/CAP_NET_RAW, no privilege + // escalation. runAsNonRoot is NOT set (it runs as root). + assert_eq!(sc["allowPrivilegeEscalation"], serde_json::json!(false)); + assert_eq!(sc["capabilities"]["drop"], serde_json::json!(["ALL"])); + assert!(sc["runAsNonRoot"].is_null()); } #[test] @@ -5636,8 +5650,9 @@ mod tests { &mut pod_template, "openshell/sandbox:latest", "IfNotPresent", - 1000, // sandbox_gid - 0, // workspace_init_uid (non-binary-aware: root) + 1000, // sandbox_gid + 0, // workspace_init_uid (root) + false, // harden_init (combined) ); // Init container @@ -5698,6 +5713,7 @@ mod tests { "IfNotPresent", 1000, 0, + false, ); let init_image = pod_template["spec"]["initContainers"][0]["image"] @@ -5720,7 +5736,7 @@ mod tests { } }); - apply_workspace_persistence(&mut pod_template, "img:latest", "Always", 1000, 0); + apply_workspace_persistence(&mut pod_template, "img:latest", "Always", 1000, 0, false); let cmd = pod_template["spec"]["initContainers"][0]["command"] .as_array() diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 00f7a05516..1ae80d950d 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -164,6 +164,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | cni.nodeSelector | object | `{}` | | | cni.openshift.privilegedSCC | bool | `false` | Grant the privileged SecurityContextConstraints to the CNI ServiceAccount. Required on OpenShift for the privileged installer DaemonSet. Creates a ClusterRole + ClusterRoleBinding. | | cni.resources | object | `{}` | | +| cni.sandboxNamespaces | list | `[]` | Namespaces whose pods the plugin inspects at CNI ADD. Pods in other namespaces are passed through WITHOUT a Kubernetes API lookup, so an API/RBAC problem cannot block unrelated workloads. Empty defaults to the release's sandbox namespace. A cluster running multiple OpenShell releases must list EVERY sandbox namespace here on the singleton (a pod in an unlisted namespace is not enforced). | | cni.stateDir | string | `""` | Persistent host directory for plugin credentials (kubeconfig, token, ca.crt) in "multus-chain" mode. Empty falls back to confDir. Must not be a tmpfs path such as one under /run. | | cni.tolerations | list | `[]` | | | fullnameOverride | string | `""` | Override the full generated resource name. | diff --git a/deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml b/deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml new file mode 100644 index 0000000000..7d9101943f --- /dev/null +++ b/deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# OPTIONAL, OpenShift-specific: close the node-reboot fail-open window for the +# cni-sidecar topology. +# +# Why this is needed: on OpenShift (multus-chain) the chained plugin config lives +# under /run (tmpfs) and is wiped on reboot, but the persistent +# `openshell.ai/cni-ready` Node label survives. Between a node coming back up and +# the OpenShell CNI DaemonSet re-installing the plugin, a sandbox pod could +# schedule onto a node with no egress enforcement. A boot-time NoSchedule taint +# repels workloads until enforcement is back; the OpenShell CNI DaemonSet +# tolerates this taint and REMOVES it once enforcement is ready (see +# `openshell-cni node-ready`). It never re-adds the taint, so a transient +# reconcile failure does not over-repel a running node. +# +# This is operator-provided NODE configuration, not part of the Helm chart: the +# taint must be applied before the kubelet marks the node schedulable, which only +# node-level config can do. Apply it to the MachineConfigPool(s) whose nodes run +# sandbox pods. +# +# CAVEATS / prerequisites: +# * Tradeoff: a NoSchedule taint repels ALL non-tolerating pods on the node +# until the OpenShell CNI is ready — not just sandboxes. Scope it to a +# sandbox-dedicated MachineConfigPool if that is undesirable on shared nodes. +# * RBAC: the command below must authenticate as an identity allowed to patch +# node taints. The kubelet's own identity is restricted by the NodeRestriction +# admission plugin, so bind a dedicated ServiceAccount (with a ClusterRole +# granting nodes get/patch) and place its kubeconfig at the path referenced +# below, or adapt the command to your node tooling. +# * `oc`/`kubectl` is assumed on the node image; adjust for your RHCOS tooling. +# +# The taint key/effect MUST match what the OpenShell CNI removes: +# openshell.ai/cni-not-ready=:NoSchedule +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: worker + name: 50-openshell-cni-boot-taint +spec: + config: + ignition: + version: 3.4.0 + systemd: + units: + - name: openshell-cni-boot-taint.service + enabled: true + contents: | + [Unit] + Description=Taint node not-ready for OpenShell CNI until enforcement is up + # Run once the kubelet/API client is available. + After=kubelet.service + Wants=kubelet.service + [Service] + Type=oneshot + RemainAfterExit=yes + # KUBECONFIG must authenticate as an identity allowed to patch node + # taints (see CAVEATS above). Adjust the path/tooling for your nodes. + Environment=KUBECONFIG=/var/lib/openshell/cni-boot-taint.kubeconfig + ExecStart=/usr/bin/oc adm taint node "%H" openshell.ai/cni-not-ready=:NoSchedule --overwrite + [Install] + WantedBy=multi-user.target diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index a487f24bbf..ea67aef7f7 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -57,7 +57,7 @@ avoids requiring an immutable-selector or component-label change on existing gateway releases during upgrade. */}} {{- define "openshell.cniSelectorLabels" -}} -app.kubernetes.io/name: {{ include "openshell.name" . }}-cni +app.kubernetes.io/name: {{ include "openshell.cniName" . }} app.kubernetes.io/instance: {{ .Release.Name }} {{- end }} @@ -88,14 +88,17 @@ Create the name of the service account to use {{/* Cluster-singleton name for the CNI installer and its RBAC/SCC. The CNI installer -is a per-cluster singleton (one chained plugin serves every OpenShell release's -sandbox pods, keyed on pod annotations), so its name is release-independent -(chart name, not fullname). This lets additional gateway releases share one CNI -installation and keeps the host CNI config, ClusterRole, and readiness label -unambiguous across releases. +is a per-cluster singleton: one chained plugin serves every OpenShell release's +sandbox pods (keyed on pod annotations), and it writes fixed host state (the +`/run` chain file / conflist entry and the `openshell.ai/cni-ready` Node label). +The name is therefore a genuinely FIXED cluster identity — a constant, not +derived from nameOverride/fullname — so two releases with different name +overrides cannot install competing "singletons" over the same host state. A +second release that also sets cni.enabled=true collides on these fixed-named +cluster resources (Helm ownership) and must instead set cni.external=true. */}} {{- define "openshell.cniName" -}} -{{- printf "%s-cni" (include "openshell.name" .) | trunc 63 | trimSuffix "-" }} +{{- "openshell-cni" }} {{- end }} {{/* diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml index 59ea069faf..ec7d171d83 100644 --- a/deploy/helm/openshell/templates/cni-daemonset.yaml +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -13,6 +13,7 @@ {{- fail "cni.mode=multus-chain requires cni.chainDir" -}} {{- end }} {{- $sandboxNamespace := .Values.server.sandboxNamespace | default .Release.Namespace }} +{{- $cniSandboxNamespaces := .Values.cni.sandboxNamespaces | default (list $sandboxNamespace) }} {{- $cniLogFile := .Values.cni.logFile | default "/var/log/openshell-cni.log" }} {{- $cniLogDir := dir $cniLogFile }} {{- $stateDir := .Values.cni.confDir }} @@ -98,22 +99,27 @@ spec: cluster: cluster user: openshell-cni EOF - # The CNI installer is a cluster singleton. The chained plugin - # enforces any pod carrying the OpenShell annotations (set only by a - # gateway on its sandbox pods), so a single installation serves every - # release across all namespaces — hence an empty sandboxNamespaces - # allowlist and a fixed "openshell" owner rather than a per-release - # namespace list. `configVersion` binds node readiness to the desired - # config: a reconcile that finds a stale-version entry repairs it - # before re-marking the node ready. + # The CNI installer is a cluster singleton with a fixed "openshell" + # owner. Its plugin enforces any pod carrying the OpenShell + # annotations (set only by a gateway on its sandbox pods), scoped to + # the sandboxNamespaces allowlist. The allowlist is a blast-radius + # guard: pods in namespaces NOT listed are passed through WITHOUT a + # Kubernetes API lookup, so control-plane latency or an RBAC/token + # problem cannot block unrelated workloads' pod creation. It defaults + # to this release's sandbox namespace; a cluster running multiple + # OpenShell releases must list every sandbox namespace here (a pod in + # an unlisted namespace is not enforced). `configVersion` binds node + # readiness to the desired config so a stale entry is repaired before + # the node is re-marked ready. owner="openshell" - config_version="$(printf '%s|%s|%s|v1' "${owner}" "${host_state_dir}/openshell-cni-kubeconfig" "{{ .Values.cni.logLevel }}" | sha256sum | cut -d' ' -f1 | cut -c1-16)" + sandbox_namespaces_json='{{ $cniSandboxNamespaces | toJson }}' + config_version="$(printf '%s|%s|%s|%s|v2' "${owner}" "${host_state_dir}/openshell-cni-kubeconfig" "{{ .Values.cni.logLevel }}" "${sandbox_namespaces_json}" | sha256sum | cut -d' ' -f1 | cut -c1-16)" cat > "${plugin_config}" < +On node reboot the Multus chain file (under `/run`, tmpfs) is wiped while the +`openshell.ai/cni-ready` Node label persists, so a sandbox could briefly schedule +onto a node before enforcement is re-established. To close this window, apply the +boot-time taint MachineConfig at +`deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml` (read its +caveats first — it is operator-provided node config and requires taint RBAC). The +CNI DaemonSet tolerates the taint and removes it once enforcement is ready. + + ## Next Steps - For TLS-enabled deployments, refer to [Managing Certificates](/kubernetes/managing-certificates). diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index e09e759241..8242b81319 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -162,7 +162,8 @@ The most commonly changed values are: | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect based on cluster version: clusters running Kubernetes 1.35 or later use `image-volume` (ImageVolume GA in 1.36); older clusters use `init-container`. Set explicitly to `image-volume` on Kubernetes 1.33 or 1.34 with the ImageVolume feature gate enabled, or to `init-container` to force the legacy path on any version. | | `supervisor.topology` | Sandbox pod topology. Refer to [Topology](/kubernetes/topology). | | `supervisor.sidecar.proxyUid` | Non-root UID used when sidecar process/binary-aware network policy is disabled. The default binary-aware sidecar runs as UID 0 instead. The configured UID must not match the sandbox UID. | -| `cni.enabled` | Install the privileged OpenShell CNI DaemonSet. Required when `supervisor.topology=cni-sidecar`. | +| `cni.enabled` | Install the privileged OpenShell CNI DaemonSet (a per-cluster singleton). Enable in one release per cluster when `supervisor.topology=cni-sidecar`. | +| `cni.external` | Use a CNI singleton installed by another release instead of installing one here. Set on additional `cni-sidecar` gateway releases (with `cni.enabled=false`). | | `cni.configFile` | Existing host CNI conflist to patch. Leave empty to patch the first non-OpenShell `.conflist` in `cni.confDir`. | | `cni.logFile` | Host log file written by the OpenShell CNI plugin and tailed by the installer DaemonSet. Defaults to `/var/log/openshell-cni.log`. | diff --git a/rfc/cni-sidecar-topology-DRAFT.md b/rfc/cni-sidecar-topology-DRAFT.md index 41c0a92522..14e2ca463b 100644 --- a/rfc/cni-sidecar-topology-DRAFT.md +++ b/rfc/cni-sidecar-topology-DRAFT.md @@ -398,10 +398,19 @@ behavior is behind a default-off value. repairs (clears the label the instant enforcement is not verifiably present), and enforcement lives in the host CNI config so ordinary DaemonSet restarts never strip it — `preStop` removes it only on a real teardown, fencing first. - Residual: an ungraceful installer-pod deletion (no `preStop`) can leave a stale - `cni-ready=true` until the pod is rescheduled and the next reconcile re-checks. - Operators diagnose via `kubectl logs daemonset/openshell-cni` and the tailed - plugin log. +- **Node reboot (tmpfs enforcement).** In `multus-chain` mode the chain file + lives under `/run` (tmpfs) and is wiped on reboot, but the persistent + `cni-ready` label survives, so the label gate alone cannot cover a reboot. A + boot-time `NoSchedule` taint (`openshell.ai/cni-not-ready`) closes this: it is + applied by operator-provided node config (only node config runs before the + scheduler; a MachineConfig example ships under `deploy/helm/openshell/examples/`), + the CNI DaemonSet tolerates it, and `openshell-cni node-ready` removes it once + enforcement is ready (never re-adding it, so a transient unready never + over-repels). `conflist` mode stores the plugin on persistent disk and is + unaffected by reboot. Residual: an ungraceful installer-pod deletion (no + `preStop`) can leave a stale `cni-ready=true` until the pod is rescheduled and + the next reconcile re-checks. Operators diagnose via + `kubectl logs daemonset/openshell-cni` and the tailed plugin log. - **Cluster-singleton CNI.** The chained plugin and readiness label are cluster-global host state. Rather than key them to a per-release namespace list (which two releases would fight over), the installer is a cluster singleton: the From 59a936428c977d2c0aac7b215fb75fd0230349a8 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Tue, 4 Aug 2026 10:43:34 -0400 Subject: [PATCH 17/20] docs(cni-sidecar): correct multi-release workflow, boot-taint, and singleton docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address the sixth review round (docs/examples only): - Fix the additional-release workflow: the singleton only enforces namespaces in its cni.sandboxNamespaces allowlist, so an extra cni.external release whose namespace is not added to the owner's allowlist runs unenforced. The extra-release example and the OpenShift docs now require (and show) the two-step flow — add the namespace to the singleton owner and let nodes reconcile, THEN install the additional gateway — with prominent warnings. - Harden the boot-taint MachineConfig example: retry-until-tainted loop, Restart=on-failure, kubelet --register-with-taints for the initial join, and an honest note that a small residual reboot race remains (no chart-shippable mechanism is fully race-free; conflist mode is unaffected). - Correct stale docs to match the implementation: topology and architecture docs described an empty all-namespaces allowlist and a foreign-owner refusal, but the plugin uses an explicit sandboxNamespaces allowlist and upgrades any openshell-cni entry in place. Add cni.sandboxNamespaces and boot-taint diagnosis to the debug-openshell-cluster skill. Signed-off-by: Russell Bryant --- .../skills/debug-openshell-cluster/SKILL.md | 29 +++++++-- architecture/compute-runtimes.md | 31 ++++++---- .../cni-sidecar-openshift-boot-taint.yaml | 60 +++++++++++-------- .../cni-sidecar-openshift-extra-release.yaml | 24 ++++++-- docs/kubernetes/openshift.mdx | 20 ++++++- docs/kubernetes/topology.mdx | 18 +++--- 6 files changed, 127 insertions(+), 55 deletions(-) diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 91745bad8b..c5e88bf0bf 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -409,10 +409,9 @@ network setup, inspect the DaemonSet logs, the host CNI config, and whether the cluster actually invokes chained CNI plugins for the sandbox runtime class. The CNI installer is a **cluster singleton** with a fixed identity -(`openshell-cni`): one installation serves every OpenShell release across all -namespaces. It is not necessarily in the release you are debugging — the owner is -the release with `cni.enabled=true`; other gateway releases set `cni.external=true` -and reuse it. Locate the owner and its resources cluster-wide: +(`openshell-cni`). It is not necessarily in the release you are debugging — the +owner is the release with `cni.enabled=true`; other gateway releases set +`cni.external=true` and reuse it. Locate the owner and its resources cluster-wide: ```bash kubectl get daemonset -A -l app.kubernetes.io/name=openshell-cni @@ -423,6 +422,28 @@ A gateway release using `cni-sidecar` with neither `cni.enabled` nor `cni.external` fails to render (template error), so if the gateway installed but sandboxes are unenforced, confirm the singleton exists and is Ready. +**Namespace allowlist (silent unenforced sandboxes).** The plugin only inspects +pods whose namespace is in its `sandboxNamespaces` allowlist +(`cni.sandboxNamespaces`, default the owner's sandbox namespace); pods elsewhere +are passed through unenforced. If a sandbox reaches 2/2 but its egress is NOT +blocked, confirm its namespace is in the installed allowlist — a common cause is +an additional `cni.external=true` release whose namespace was never added to the +owner's list: + +```bash +# On a CNI installer pod, read the installed plugin config's sandboxNamespaces: +kubectl -n exec ds/openshell-cni -c install-cni -- \ + sh -c 'cat /host/run/multus/cni/net.d/vendor-cni-chain/openshell-cni.conf 2>/dev/null \ + || cat /host/etc/cni/net.d/*.conflist' +``` + +**Node reboot / boot taint.** In `multus-chain` mode the chain file lives under +`/run` (tmpfs) and is wiped on reboot while the `cni-ready` label persists. +Clusters that apply the optional boot-time taint (`openshell.ai/cni-not-ready`, +see `deploy/helm/openshell/examples/`) will show it on a node until the installer +removes it; a node stuck with the taint means the installer has not reached ready +there. Check with `kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints`. + A per-node scheduling gate can also keep cni-sidecar sandbox pods `Pending`. The CNI DaemonSet labels each node `openshell.ai/cni-ready=true` after installing the plugin, and the gateway sets a required `nodeAffinity` on that label. If a diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 027857cf18..040c3363fc 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -294,19 +294,24 @@ limitation: an ungraceful DaemonSet pod deletion (no `preStop`) leaves a stale `cni-ready=true` until the pod is rescheduled and the next reconcile tick re-evaluates it. -The CNI installer is a **cluster singleton**. The chained plugin enforces any pod -carrying the OpenShell annotations (`openshell.ai/cni=enabled` plus the proxy-UID -and enforcement-mode annotations, set only by a gateway on its own sandbox pods), -so a single installation serves every OpenShell release across all namespaces — -enforcement is gated by per-pod annotations, not a namespace list. The installer -resources use release-independent names, the plugin config carries a fixed -`openshell` owner and an empty namespace allowlist, and a `configVersion` binds -readiness to the desired config so a stale-version entry is repaired before the -node is re-marked ready. Install the singleton (`cni.enabled=true`) in one release -per cluster; additional gateway releases set `cni.enabled=false` + -`cni.external=true` to share it, and the installer refuses to overwrite a chained -plugin entry with a non-OpenShell owner. Because the plugin serves all sandbox -namespaces, its `pods get` grant is cluster-scoped. +The CNI installer is a **cluster singleton**. Its chained plugin enforces pods +that carry the OpenShell annotations (`openshell.ai/cni=enabled` plus the +proxy-UID and enforcement-mode annotations, set only by a gateway on its own +sandbox pods) **and** whose namespace is in the plugin's `sandboxNamespaces` +allowlist. Pods in other namespaces are passed through without a Kubernetes API +lookup, so the allowlist bounds the blast radius of the per-pod annotation read. +The allowlist defaults to the owner release's sandbox namespace +(`cni.sandboxNamespaces`); a cluster running multiple OpenShell releases must add +every additional sandbox namespace to the owner's list, or those sandboxes run +unenforced. The installer resources use a fixed release-independent name, the +plugin config carries a fixed `openshell` owner, and a `configVersion` (over the +allowlist and config) binds readiness to the desired config so a stale-version +entry is repaired before the node is re-marked ready. Install the singleton +(`cni.enabled=true`) in one release per cluster; additional gateway releases set +`cni.enabled=false` + `cni.external=true` to share it. The installer treats any +`openshell-cni` chained entry as its own and upgrades it in place (the conflist +patch preserves all other plugins). The `pods get` grant is cluster-scoped so the +one installer can read sandbox pods in any allowlisted namespace. On OpenShift, binary-aware network policy also requires a purpose-built SecurityContextConstraints for sandbox pods: the network sidecar runs as UID 0 diff --git a/deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml b/deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml index 7d9101943f..7335f9ae24 100644 --- a/deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml +++ b/deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml @@ -1,33 +1,40 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# OPTIONAL, OpenShift-specific: close the node-reboot fail-open window for the +# OPTIONAL, OpenShift-specific: reduce the node-reboot fail-open window for the # cni-sidecar topology. # -# Why this is needed: on OpenShift (multus-chain) the chained plugin config lives -# under /run (tmpfs) and is wiped on reboot, but the persistent -# `openshell.ai/cni-ready` Node label survives. Between a node coming back up and -# the OpenShell CNI DaemonSet re-installing the plugin, a sandbox pod could -# schedule onto a node with no egress enforcement. A boot-time NoSchedule taint -# repels workloads until enforcement is back; the OpenShell CNI DaemonSet -# tolerates this taint and REMOVES it once enforcement is ready (see -# `openshell-cni node-ready`). It never re-adds the taint, so a transient -# reconcile failure does not over-repel a running node. +# Why: on OpenShift (multus-chain) the chained plugin config lives under /run +# (tmpfs) and is wiped on reboot, but the persistent `openshell.ai/cni-ready` +# Node label survives. Between a node coming back up and the OpenShell CNI +# DaemonSet re-installing the plugin, a sandbox pod could schedule onto a node +# with no egress enforcement. A NoSchedule taint repels workloads until +# enforcement is back; the OpenShell CNI DaemonSet tolerates it and REMOVES it +# once enforcement is ready (it never re-adds it, so a transient reconcile +# failure does not over-repel a running node). # -# This is operator-provided NODE configuration, not part of the Helm chart: the -# taint must be applied before the kubelet marks the node schedulable, which only -# node-level config can do. Apply it to the MachineConfigPool(s) whose nodes run -# sandbox pods. +# HONEST LIMITATION: there is no chart-shippable mechanism that is fully +# race-free. A taint must exist before the kubelet reports the node schedulable, +# which only node-level config can influence. This example uses two layers: +# 1. kubelet `--register-with-taints` — applied when the kubelet FIRST registers +# the node, so a freshly joined node is tainted with no window. (Configure it +# via a KubeletConfig / MachineConfig for your pool; see OpenShift docs.) +# 2. A boot systemd unit that re-asserts the taint on every boot and RETRIES +# until it succeeds — because kubelet does not re-apply register-with-taints +# when an existing Node object is re-registered after reboot. +# A small residual window remains on reboot (kubelet may mark Ready before the +# unit re-taints). `conflist` mode stores the plugin on persistent disk and is +# unaffected by reboot; prefer it, or cordon/drain before rebooting nodes, if the +# residual window is unacceptable. # # CAVEATS / prerequisites: # * Tradeoff: a NoSchedule taint repels ALL non-tolerating pods on the node -# until the OpenShell CNI is ready — not just sandboxes. Scope it to a -# sandbox-dedicated MachineConfigPool if that is undesirable on shared nodes. -# * RBAC: the command below must authenticate as an identity allowed to patch -# node taints. The kubelet's own identity is restricted by the NodeRestriction -# admission plugin, so bind a dedicated ServiceAccount (with a ClusterRole -# granting nodes get/patch) and place its kubeconfig at the path referenced -# below, or adapt the command to your node tooling. +# until the OpenShell CNI is ready. Scope this to a sandbox-dedicated +# MachineConfigPool if that is undesirable on shared nodes. +# * RBAC: the taint command must authenticate as an identity allowed to patch +# node taints. The kubelet's own identity is restricted by NodeRestriction, +# so provision a dedicated ServiceAccount kubeconfig (bound to a ClusterRole +# granting nodes get/patch) at the path referenced below. # * `oc`/`kubectl` is assumed on the node image; adjust for your RHCOS tooling. # # The taint key/effect MUST match what the OpenShell CNI removes: @@ -49,15 +56,20 @@ spec: contents: | [Unit] Description=Taint node not-ready for OpenShell CNI until enforcement is up - # Run once the kubelet/API client is available. + # Order as early as possible after the kubelet/API client is available. + # A residual race remains (see the file header); the retry loop below + # keeps trying so a transient failure never leaves the node untainted. After=kubelet.service Wants=kubelet.service [Service] Type=oneshot RemainAfterExit=yes # KUBECONFIG must authenticate as an identity allowed to patch node - # taints (see CAVEATS above). Adjust the path/tooling for your nodes. + # taints (see CAVEATS). Adjust the path/tooling for your nodes. Environment=KUBECONFIG=/var/lib/openshell/cni-boot-taint.kubeconfig - ExecStart=/usr/bin/oc adm taint node "%H" openshell.ai/cni-not-ready=:NoSchedule --overwrite + # Retry until the taint is applied; never give up silently. + ExecStart=/bin/sh -c 'until /usr/bin/oc adm taint node "$(hostname)" openshell.ai/cni-not-ready=:NoSchedule --overwrite; do echo "retrying node taint..."; sleep 2; done' + Restart=on-failure + RestartSec=2 [Install] WantedBy=multi-user.target diff --git a/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml b/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml index 9995479cb1..6b23162a2a 100644 --- a/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml +++ b/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml @@ -3,12 +3,26 @@ # Example values for an ADDITIONAL gateway release on a cluster that already has # the OpenShell CNI singleton installed by another release. This release does not -# install its own CNI; it reuses the cluster's existing installation. Its sandbox -# pods are enforced by the shared plugin (enforcement is keyed on pod annotations -# the gateway sets) and gated on the same per-node readiness label. +# install its own CNI; it reuses the cluster's existing installation. # -# helm install openshell-team-b deploy/helm/openshell -n team-b \ -# -f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml +# ┌─ REQUIRED TWO-STEP WORKFLOW ────────────────────────────────────────────────┐ +# │ The CNI singleton only inspects (and enforces) pods in the namespaces listed │ +# │ in its cni.sandboxNamespaces allowlist. Pods in any other namespace are │ +# │ passed through WITHOUT enforcement. So before enabling this gateway you MUST │ +# │ add its sandbox namespace to the SINGLETON OWNER release's allowlist, or its │ +# │ sandboxes will silently run unrestricted. │ +# │ │ +# │ 1. Update the singleton owner to include this release's sandbox namespace: │ +# │ helm upgrade deploy/helm/openshell -n \ │ +# │ --reuse-values \ │ +# │ --set 'cni.sandboxNamespaces={openshell,team-b}' │ +# │ Confirm every schedulable node re-reconciles (configVersion changes) and │ +# │ stays cni-ready before proceeding. │ +# │ │ +# │ 2. THEN install this gateway (its sandbox namespace = team-b here): │ +# │ helm install openshell-team-b deploy/helm/openshell -n team-b \ │ +# │ -f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml +# └──────────────────────────────────────────────────────────────────────────────┘ cni: # Do not install a second CNI installer; use the cluster singleton. diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 8207645c88..f4bd61c69c 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -141,8 +141,24 @@ is routed through the OpenShell network sidecar and enforced by policy. To run **additional** gateway releases on the same cluster, do not install a -second CNI. Use `-f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml` -(`cni.enabled=false`, `cni.external=true`), which reuses the cluster singleton. +second CNI — reuse the singleton with `cni.enabled=false`, `cni.external=true` +(`-f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml`). + + +The singleton only enforces pods in namespaces listed in its +`cni.sandboxNamespaces` allowlist (default: the owner release's sandbox +namespace); pods elsewhere are passed through **unenforced**. Before enabling an +additional release you **must** add its sandbox namespace to the singleton +owner's allowlist and let every node re-reconcile: + +```shell +helm upgrade deploy/helm/openshell -n \ + --reuse-values --set 'cni.sandboxNamespaces={openshell,team-b}' +``` + +Only then install the additional gateway. Skipping this step silently leaves the +new release's sandboxes without egress enforcement. + On node reboot the Multus chain file (under `/run`, tmpfs) is wiped while the diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index beb793725c..d014dc2261 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -234,13 +234,17 @@ provide the expected network enforcement. -The CNI installer is a **cluster singleton**. Its chained plugin enforces any -pod carrying the OpenShell annotations (set only by a gateway on its own sandbox -pods), so a single installation serves every OpenShell release across all -namespaces. Enable the installer (`cni.enabled=true`) in exactly one release per -cluster; additional gateway releases set `cni.enabled=false` and -`cni.external=true` to share it. The installer refuses to overwrite a chained -plugin entry with a non-OpenShell owner (fail closed). +The CNI installer is a **cluster singleton**. Its chained plugin enforces pods +that carry the OpenShell annotations (set only by a gateway on its own sandbox +pods) **and** whose namespace is in the plugin's `cni.sandboxNamespaces` +allowlist (default: the owner release's sandbox namespace). Pods in other +namespaces are passed through without an API lookup, so the allowlist bounds the +blast radius. Enable the installer (`cni.enabled=true`) in exactly one release +per cluster; additional gateway releases set `cni.enabled=false` and +`cni.external=true` to share it — and their sandbox namespace must be added to +the owner's `cni.sandboxNamespaces` first (see +[OpenShift](/kubernetes/openshift#enable-the-cni-sidecar-topology)). The installer +treats any `openshell-cni` chained entry as its own and upgrades it in place. ## Credential Exposure From ee777f8aa986508305f8e2c3dfb8ffaf53b0a011 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Tue, 4 Aug 2026 12:46:59 -0400 Subject: [PATCH 18/20] feat(cni-sidecar): auto-aggregate sandbox namespaces into the CNI allowlist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round seven: make external-release enforcement machine-verifiable, and correct the remaining docs. - cni: add `openshell-cni label-namespace` and `list-sandbox-namespaces` subcommands. Each cni-sidecar gateway release runs a helm hook that labels its sandbox namespace openshell.ai/sandbox=true; the CNI singleton's reconcile aggregates every labeled namespace (unioned with the optional static cni.sandboxNamespaces) into the plugin allowlist and rebinds configVersion. An additional cni.external release is therefore discovered and enforced within one reconcile — no manual owner-allowlist edit, closing the silent-bypass footgun. - helm: add the namespace-label hook (Job + minimal namespaces get/patch RBAC scoped to the release's sandbox namespace) and grant the CNI ClusterRole namespaces list. On discovery failure the reconcile keeps the installed allowlist rather than shrinking it (never fails a running namespace open). - docs: stop claiming the boot taint 'closes' the reboot window (it narrows it; a residual race remains — conflist mode is unaffected). Correct topology, architecture, RFC, and the debug skill to describe the explicit auto-aggregated allowlist and in-place upgrade of any openshell-cni entry, and fix stale RFC claims (cni.enabled-or-external, 30s reconcile). Signed-off-by: Russell Bryant --- architecture/compute-runtimes.md | 45 +++++---- crates/openshell-cni/src/lib.rs | 94 ++++++++++++++++++- crates/openshell-cni/src/main.rs | 2 + deploy/helm/openshell/README.md | 2 +- .../cni-sidecar-openshift-extra-release.yaml | 25 ++--- .../openshell/templates/cni-daemonset.yaml | 55 ++++++++--- .../templates/cni-namespace-label-hook.yaml | 92 ++++++++++++++++++ deploy/helm/openshell/templates/cni-rbac.yaml | 8 ++ .../openshell/tests/cni_daemonset_test.yaml | 8 +- .../tests/cni_namespace_label_hook_test.yaml | 72 ++++++++++++++ .../helm/openshell/tests/cni_rbac_test.yaml | 11 +++ deploy/helm/openshell/values.yaml | 13 +-- docs/kubernetes/openshift.mdx | 32 +++---- docs/kubernetes/topology.mdx | 19 ++-- rfc/cni-sidecar-topology-DRAFT.md | 43 +++++---- 15 files changed, 416 insertions(+), 105 deletions(-) create mode 100644 deploy/helm/openshell/templates/cni-namespace-label-hook.yaml create mode 100644 deploy/helm/openshell/tests/cni_namespace_label_hook_test.yaml diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 040c3363fc..69013c3a56 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -284,12 +284,17 @@ installer's own DaemonSet) bound to the dedicated CNI ServiceAccount. The persistent label cannot by itself cover a node reboot that wipes tmpfs-backed enforcement (`multus-chain` stores the chain file under `/run`). A boot-time -`NoSchedule` taint (`openshell.ai/cni-not-ready`) closes that window: operator -node config applies it at boot (only node config runs before the scheduler; a -MachineConfig example ships under `deploy/helm/openshell/examples/`), the CNI -DaemonSet tolerates it, and the installer removes it once enforcement is ready -(never re-adding it, so a transient unready does not over-repel). `conflist` mode -keeps the plugin on persistent disk and is unaffected by reboot. One residual +`NoSchedule` taint (`openshell.ai/cni-not-ready`) **narrows** that window but does +not fully close it: operator node config applies it at boot (only node config +runs before the scheduler; a MachineConfig example ships under +`deploy/helm/openshell/examples/`), the CNI DaemonSet tolerates it, and the +installer removes it once enforcement is ready (never re-adding it, so a transient +unready does not over-repel). A small residual race remains because the taint is +applied after the kubelet starts, so the kubelet can briefly mark the node +schedulable before the taint lands; `--register-with-taints` covers the initial +join but not reboot re-registration. `conflist` mode keeps the plugin on +persistent disk and is unaffected by reboot, and is the way to avoid the window +entirely. One residual limitation: an ungraceful DaemonSet pod deletion (no `preStop`) leaves a stale `cni-ready=true` until the pod is rescheduled and the next reconcile tick re-evaluates it. @@ -300,18 +305,22 @@ proxy-UID and enforcement-mode annotations, set only by a gateway on its own sandbox pods) **and** whose namespace is in the plugin's `sandboxNamespaces` allowlist. Pods in other namespaces are passed through without a Kubernetes API lookup, so the allowlist bounds the blast radius of the per-pod annotation read. -The allowlist defaults to the owner release's sandbox namespace -(`cni.sandboxNamespaces`); a cluster running multiple OpenShell releases must add -every additional sandbox namespace to the owner's list, or those sandboxes run -unenforced. The installer resources use a fixed release-independent name, the -plugin config carries a fixed `openshell` owner, and a `configVersion` (over the -allowlist and config) binds readiness to the desired config so a stale-version -entry is repaired before the node is re-marked ready. Install the singleton -(`cni.enabled=true`) in one release per cluster; additional gateway releases set -`cni.enabled=false` + `cni.external=true` to share it. The installer treats any -`openshell-cni` chained entry as its own and upgrades it in place (the conflist -patch preserves all other plugins). The `pods get` grant is cluster-scoped so the -one installer can read sandbox pods in any allowlisted namespace. +The allowlist is built **automatically**: each `cni-sidecar` gateway release +labels its sandbox namespace `openshell.ai/sandbox=true` (a helm hook runs +`openshell-cni label-namespace`), and the installer's reconcile aggregates every +labeled namespace (via `list-sandbox-namespaces`) — unioned with the optional +static `cni.sandboxNamespaces` — into the plugin config. An additional +`cni.external` release is therefore discovered and enforced within one reconcile, +with no manual allowlist edit. The installer resources use a fixed +release-independent name, the plugin config carries a fixed `openshell` owner, and +a `configVersion` (over the aggregated allowlist and config) binds readiness so a +stale-version entry is repaired before the node is re-marked ready. Install the +singleton (`cni.enabled=true`) in one release per cluster; additional releases set +`cni.enabled=false` + `cni.external=true`. The installer treats any `openshell-cni` +chained entry as its own and upgrades it in place (the conflist patch preserves +all other plugins). The `pods get` and `namespaces list` grants are cluster-scoped +so the one installer can discover labeled namespaces and read sandbox pods in any +of them. On OpenShift, binary-aware network policy also requires a purpose-built SecurityContextConstraints for sandbox pods: the network sidecar runs as UID 0 diff --git a/crates/openshell-cni/src/lib.rs b/crates/openshell-cni/src/lib.rs index 88b9149543..c2fa87d552 100644 --- a/crates/openshell-cni/src/lib.rs +++ b/crates/openshell-cni/src/lib.rs @@ -28,12 +28,19 @@ const NODE_READY_LABEL: &str = "openshell.ai/cni-ready"; /// Node taint applied at boot (via an operator-provided `MachineConfig` / node /// config) so a rebooted node repels workloads until egress enforcement is /// re-established — the persistent `cni-ready` label cannot reflect a reboot that -/// wipes tmpfs-backed enforcement, so a boot-time taint closes that window. The +/// wipes tmpfs-backed enforcement, so a boot-time taint narrows that window +/// (a small residual reboot race remains — see the `MachineConfig` example). The /// installer only REMOVES this taint (once enforcement is ready); it never adds /// it, so a transient unready never over-repels a running node. Must stay in sync /// with the taint key used by the `MachineConfig` and the `DaemonSet` toleration. const NODE_NOT_READY_TAINT_KEY: &str = "openshell.ai/cni-not-ready"; const NODE_NOT_READY_TAINT_EFFECT: &str = "NoSchedule"; +/// Namespace label a gateway release stamps on its sandbox namespace so the CNI +/// singleton discovers and enforces it automatically — no manual allowlist edit +/// per release. The installer aggregates all namespaces carrying this label into +/// the plugin's `sandboxNamespaces`. Must stay in sync with the label written by +/// the namespace-label hook. +const SANDBOX_NAMESPACE_LABEL: &str = "openshell.ai/sandbox"; #[allow(dead_code)] const OPENSHELL_TABLE: &str = "openshell_sidecar_bypass"; #[allow(dead_code)] @@ -240,6 +247,33 @@ pub fn node_ready(args: &[String]) -> Result<()> { } } +/// Entry point for `openshell-cni label-namespace `. +/// +/// Run by the gateway's namespace-label helm hook so the CNI singleton +/// auto-discovers this release's sandbox namespace. Stamps +/// `openshell.ai/sandbox=true`. +pub fn label_namespace(args: &[String]) -> Result<()> { + let namespace = args + .first() + .ok_or_else(|| miette::miette!("label-namespace requires a namespace argument"))?; + let client = KubeApiClient::from_in_cluster()?; + client.patch_namespace_label(namespace, SANDBOX_NAMESPACE_LABEL, Some("true")) +} + +/// Entry point for `openshell-cni list-sandbox-namespaces`. +/// +/// Run by the installer reconcile to aggregate every namespace carrying +/// `openshell.ai/sandbox=true` into the plugin allowlist. Prints one namespace +/// name per line. +pub fn list_sandbox_namespaces() -> Result<()> { + let client = KubeApiClient::from_in_cluster()?; + let namespaces = client.list_namespaces_by_label(SANDBOX_NAMESPACE_LABEL, "true")?; + for ns in namespaces { + println!("{ns}"); + } + Ok(()) +} + /// Entry point for `openshell-cni daemonset-active`. /// /// Used by the installer `preStop` hook to distinguish an ordinary pod @@ -632,6 +666,64 @@ impl KubeApiClient { Ok(()) } + /// Sets a label on a namespace via a JSON merge patch (null value removes). + fn patch_namespace_label(&self, namespace: &str, key: &str, value: Option<&str>) -> Result<()> { + let url = format!("{}/api/v1/namespaces/{}", self.server, namespace); + let label_value = value.map_or(Value::Null, |v| Value::String(v.to_string())); + let body = serde_json::json!({ "metadata": { "labels": { key: label_value } } }); + let response = self + .client + .patch(url) + .bearer_auth(&self.token) + .header( + reqwest::header::CONTENT_TYPE, + "application/merge-patch+json", + ) + .body(serde_json::to_vec(&body).into_diagnostic()?) + .send() + .into_diagnostic() + .wrap_err("failed to patch namespace label")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while labeling namespace {}", + response.status(), + namespace + )); + } + Ok(()) + } + + /// Returns the names of all namespaces carrying `key=value`. + fn list_namespaces_by_label(&self, key: &str, value: &str) -> Result> { + let selector = format!("{key}={value}"); + let url = format!("{}/api/v1/namespaces", self.server); + let response = self + .client + .get(url) + .query(&[("labelSelector", selector.as_str())]) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to list namespaces by label")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while listing namespaces", + response.status() + )); + } + let list = response.json::().into_diagnostic()?; + let names = list["items"] + .as_array() + .map(|items| { + items + .iter() + .filter_map(|item| item["metadata"]["name"].as_str().map(str::to_string)) + .collect() + }) + .unwrap_or_default(); + Ok(names) + } + /// Removes the given taint (key + effect) from the node if present, using a /// JSON Patch whose `test` on `metadata.resourceVersion` provides optimistic /// concurrency so a concurrent taint change by another controller is not diff --git a/crates/openshell-cni/src/main.rs b/crates/openshell-cni/src/main.rs index f4ded8faeb..8c25f77e78 100644 --- a/crates/openshell-cni/src/main.rs +++ b/crates/openshell-cni/src/main.rs @@ -8,6 +8,8 @@ fn main() { let args: Vec = std::env::args().collect(); let result = match args.get(1).map(String::as_str) { Some("node-ready") => openshell_cni::node_ready(&args[2..]), + Some("label-namespace") => openshell_cni::label_namespace(&args[2..]), + Some("list-sandbox-namespaces") => openshell_cni::list_sandbox_namespaces(), Some("daemonset-active") => openshell_cni::daemonset_active(), // Internal: re-exec'd inside a pod netns to probe for IPv6 (see // netns_requires_ipv6_enforcement). Not part of the public CLI surface. diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 1ae80d950d..1dfea8b54b 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -164,7 +164,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | cni.nodeSelector | object | `{}` | | | cni.openshift.privilegedSCC | bool | `false` | Grant the privileged SecurityContextConstraints to the CNI ServiceAccount. Required on OpenShift for the privileged installer DaemonSet. Creates a ClusterRole + ClusterRoleBinding. | | cni.resources | object | `{}` | | -| cni.sandboxNamespaces | list | `[]` | Namespaces whose pods the plugin inspects at CNI ADD. Pods in other namespaces are passed through WITHOUT a Kubernetes API lookup, so an API/RBAC problem cannot block unrelated workloads. Empty defaults to the release's sandbox namespace. A cluster running multiple OpenShell releases must list EVERY sandbox namespace here on the singleton (a pod in an unlisted namespace is not enforced). | +| cni.sandboxNamespaces | list | `[]` | Optional STATIC additions to the plugin's enforcement allowlist. The allowlist is normally built automatically: each cni-sidecar release labels its sandbox namespace (openshell.ai/sandbox=true) via a helm hook and the singleton aggregates all labeled namespaces. This value is unioned with that discovery for namespaces you want enforced without the label. Pods in namespaces neither labeled nor listed are passed through without a Kubernetes API lookup (blast radius guard). Empty additionally seeds the owner release's sandbox namespace. | | cni.stateDir | string | `""` | Persistent host directory for plugin credentials (kubeconfig, token, ca.crt) in "multus-chain" mode. Empty falls back to confDir. Must not be a tmpfs path such as one under /run. | | cni.tolerations | list | `[]` | | | fullnameOverride | string | `""` | Override the full generated resource name. | diff --git a/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml b/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml index 6b23162a2a..00a2a3af91 100644 --- a/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml +++ b/deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml @@ -5,24 +5,13 @@ # the OpenShell CNI singleton installed by another release. This release does not # install its own CNI; it reuses the cluster's existing installation. # -# ┌─ REQUIRED TWO-STEP WORKFLOW ────────────────────────────────────────────────┐ -# │ The CNI singleton only inspects (and enforces) pods in the namespaces listed │ -# │ in its cni.sandboxNamespaces allowlist. Pods in any other namespace are │ -# │ passed through WITHOUT enforcement. So before enabling this gateway you MUST │ -# │ add its sandbox namespace to the SINGLETON OWNER release's allowlist, or its │ -# │ sandboxes will silently run unrestricted. │ -# │ │ -# │ 1. Update the singleton owner to include this release's sandbox namespace: │ -# │ helm upgrade deploy/helm/openshell -n \ │ -# │ --reuse-values \ │ -# │ --set 'cni.sandboxNamespaces={openshell,team-b}' │ -# │ Confirm every schedulable node re-reconciles (configVersion changes) and │ -# │ stays cni-ready before proceeding. │ -# │ │ -# │ 2. THEN install this gateway (its sandbox namespace = team-b here): │ -# │ helm install openshell-team-b deploy/helm/openshell -n team-b \ │ -# │ -f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml -# └──────────────────────────────────────────────────────────────────────────────┘ +# Registration is AUTOMATIC: this release runs a helm hook that labels its +# sandbox namespace `openshell.ai/sandbox=true`, and the CNI singleton discovers +# every labeled namespace and adds it to the enforcement allowlist within one +# reconcile (~30s). No manual edit of the singleton owner is required. +# +# helm install openshell-team-b deploy/helm/openshell -n team-b \ +# -f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml cni: # Do not install a second CNI installer; use the cluster singleton. diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml index ec7d171d83..4991277c8b 100644 --- a/deploy/helm/openshell/templates/cni-daemonset.yaml +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -105,21 +105,44 @@ spec: # the sandboxNamespaces allowlist. The allowlist is a blast-radius # guard: pods in namespaces NOT listed are passed through WITHOUT a # Kubernetes API lookup, so control-plane latency or an RBAC/token - # problem cannot block unrelated workloads' pod creation. It defaults - # to this release's sandbox namespace; a cluster running multiple - # OpenShell releases must list every sandbox namespace here (a pod in - # an unlisted namespace is not enforced). `configVersion` binds node - # readiness to the desired config so a stale entry is repaired before - # the node is re-marked ready. + # problem cannot block unrelated workloads' pod creation. + # + # The allowlist is computed automatically as the union of the static + # cni.sandboxNamespaces value and every namespace labeled + # openshell.ai/sandbox=true (each gateway release labels its own + # sandbox namespace via a helm hook). So an additional cni.external + # release is discovered and enforced automatically — no manual + # allowlist edit. `configVersion` binds node readiness to the desired + # config, so when the discovered set changes the reconcile repairs the + # entry before re-marking the node ready. owner="openshell" - sandbox_namespaces_json='{{ $cniSandboxNamespaces | toJson }}' - config_version="$(printf '%s|%s|%s|%s|v2' "${owner}" "${host_state_dir}/openshell-cni-kubeconfig" "{{ .Values.cni.logLevel }}" "${sandbox_namespaces_json}" | sha256sum | cut -d' ' -f1 | cut -c1-16)" - cat > "${plugin_config}" </dev/null)"; then + : + elif [ -f "${plugin_config}" ]; then + echo "OpenShell CNI: namespace discovery failed; keeping installed allowlist" >&2 + return 0 + else + discovered="" + fi + allowlist_json="$( + { printf '%s' "${sandbox_static_json}" | jq -r '.[]'; printf '%s\n' "${discovered}"; } \ + | grep -v '^$' | sort -u | jq -R . | jq -s . + )" + config_version="$(printf '%s|%s|%s|%s|v3' "${owner}" "${host_state_dir}/openshell-cni-kubeconfig" "{{ .Values.cni.logLevel }}" "${allowlist_json}" | sha256sum | cut -d' ' -f1 | cut -c1-16)" + _pc_tmp="$(mktemp "${state_dir}/openshell-cni-plugin.XXXXXX")" + cat > "${_pc_tmp}" <&2 @@ -260,6 +283,10 @@ spec: while true; do cp /var/run/secrets/kubernetes.io/serviceaccount/token "${token_file}" chmod 0600 "${token_file}" + # Refresh the aggregated allowlist (picks up newly labeled sandbox + # namespaces); config_version changes when the set does, so the + # health check below repairs the entry and re-enforces. + write_plugin_config # Fence before repair: the instant enforcement is not verifiably in # place, clear readiness so no new sandbox pod schedules here, then # attempt repair and only re-mark ready once it is healthy again. diff --git a/deploy/helm/openshell/templates/cni-namespace-label-hook.yaml b/deploy/helm/openshell/templates/cni-namespace-label-hook.yaml new file mode 100644 index 0000000000..e394e01426 --- /dev/null +++ b/deploy/helm/openshell/templates/cni-namespace-label-hook.yaml @@ -0,0 +1,92 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if eq (.Values.supervisor.topology | default "combined") "cni-sidecar" }} +{{- $sandboxNamespace := .Values.server.sandboxNamespace | default .Release.Namespace }} +{{- $name := printf "%s-cni-ns-label" (include "openshell.fullname" .) | trunc 63 | trimSuffix "-" }} +# Labels this release's sandbox namespace openshell.ai/sandbox=true so the CNI +# singleton auto-discovers and enforces it — no manual allowlist edit, for the +# owner release and any additional cni.external release alike. The label lets the +# installer aggregate every OpenShell sandbox namespace into the plugin allowlist. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ $name }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni +--- +# Minimal: get/patch on this release's sandbox namespace only (namespaces are +# cluster-scoped, so resourceNames-scoped ClusterRole is the tightest grant). +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ $name }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni +rules: + - apiGroups: + - "" + resources: + - namespaces + resourceNames: + - {{ $sandboxNamespace }} + verbs: + - get + - patch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ $name }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ $name }} +subjects: + - kind: ServiceAccount + name: {{ $name }} + namespace: {{ .Release.Namespace }} +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ $name }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni + annotations: + # Run after the release's resources (and their RBAC) are in place, on both + # install and upgrade. Re-runs are idempotent (label set --overwrite). + "helm.sh/hook": post-install,post-upgrade + "helm.sh/hook-weight": "5" + "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded +spec: + backoffLimit: 6 + template: + metadata: + labels: + {{- include "openshell.selectorLabels" . | nindent 8 }} + app.kubernetes.io/component: cni + spec: + restartPolicy: Never + serviceAccountName: {{ $name }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: label-namespace + image: {{ include "openshell.cniImage" . | quote }} + imagePullPolicy: {{ include "openshell.cniImagePullPolicy" . | quote }} + command: + - /openshell-cni + - label-namespace + - {{ $sandboxNamespace | quote }} +{{- end }} diff --git a/deploy/helm/openshell/templates/cni-rbac.yaml b/deploy/helm/openshell/templates/cni-rbac.yaml index 2fa2bd5030..393e5cd202 100644 --- a/deploy/helm/openshell/templates/cni-rbac.yaml +++ b/deploy/helm/openshell/templates/cni-rbac.yaml @@ -31,6 +31,14 @@ rules: - pods verbs: - get + # list namespaces to aggregate those labeled openshell.ai/sandbox=true into the + # plugin allowlist (automatic multi-release discovery). + - apiGroups: + - "" + resources: + - namespaces + verbs: + - list - apiGroups: - "" resources: diff --git a/deploy/helm/openshell/tests/cni_daemonset_test.yaml b/deploy/helm/openshell/tests/cni_daemonset_test.yaml index 7a0124d92e..62dd44f401 100644 --- a/deploy/helm/openshell/tests/cni_daemonset_test.yaml +++ b/deploy/helm/openshell/tests/cni_daemonset_test.yaml @@ -53,7 +53,7 @@ tests: # Singleton owner + namespace allowlist defaulting to the release namespace. - matchRegex: path: spec.template.spec.containers[0].command[2] - pattern: 'sandbox_namespaces_json=.\["openshell"\].' + pattern: 'sandbox_static_json=.\["openshell"\].' - matchRegex: path: spec.template.spec.containers[0].command[2] pattern: 'owner="openshell"' @@ -114,6 +114,10 @@ tests: - matchRegex: path: spec.template.spec.containers[0].command[2] pattern: 'owner="openshell"' + # Auto-discovery: reconcile aggregates labeled namespaces into the allowlist. + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'list-sandbox-namespaces' - it: aggregates multiple sandbox namespaces into the allowlist template: templates/cni-daemonset.yaml @@ -125,7 +129,7 @@ tests: asserts: - matchRegex: path: spec.template.spec.containers[0].command[2] - pattern: 'sandbox_namespaces_json=.\["team-a","team-b"\].' + pattern: 'sandbox_static_json=.\["team-a","team-b"\].' - it: binds readiness to a config version template: templates/cni-daemonset.yaml diff --git a/deploy/helm/openshell/tests/cni_namespace_label_hook_test.yaml b/deploy/helm/openshell/tests/cni_namespace_label_hook_test.yaml new file mode 100644 index 0000000000..32ae1184f8 --- /dev/null +++ b/deploy/helm/openshell/tests/cni_namespace_label_hook_test.yaml @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: cni namespace-label hook +templates: + - templates/cni-namespace-label-hook.yaml +release: + name: openshell + namespace: openshell +tests: + - it: does not render without cni-sidecar topology + asserts: + - hasDocuments: + count: 0 + + # Every cni-sidecar release (owner or cni.external) labels its own sandbox + # namespace so the singleton auto-discovers it. + - it: renders the label hook for a cni-sidecar release + set: + supervisor.topology: cni-sidecar + asserts: + - hasDocuments: + count: 4 + - documentIndex: 3 + isKind: + of: Job + - documentIndex: 3 + equal: + path: metadata.annotations["helm.sh/hook"] + value: post-install,post-upgrade + - documentIndex: 3 + equal: + path: spec.template.spec.containers[0].command + value: + - /openshell-cni + - label-namespace + - openshell + + # An additional external release still runs the label hook (topology drives it, + # not cni.enabled), so its namespace is auto-registered with the singleton. + - it: renders the label hook for an external release + set: + supervisor.topology: cni-sidecar + cni.enabled: false + cni.external: true + asserts: + - hasDocuments: + count: 4 + - documentIndex: 3 + isKind: + of: Job + + # RBAC is scoped to only this release's sandbox namespace. + - it: scopes the label ClusterRole to the sandbox namespace + set: + supervisor.topology: cni-sidecar + server.sandboxNamespace: team-b + asserts: + - template: templates/cni-namespace-label-hook.yaml + documentIndex: 1 + contains: + path: rules + content: + apiGroups: + - "" + resources: + - namespaces + resourceNames: + - team-b + verbs: + - get + - patch diff --git a/deploy/helm/openshell/tests/cni_rbac_test.yaml b/deploy/helm/openshell/tests/cni_rbac_test.yaml index 3ae2cc4119..4df6df0d63 100644 --- a/deploy/helm/openshell/tests/cni_rbac_test.yaml +++ b/deploy/helm/openshell/tests/cni_rbac_test.yaml @@ -45,6 +45,17 @@ tests: - pods verbs: - get + # namespaces list: aggregate labeled sandbox namespaces into the allowlist. + - documentIndex: 1 + contains: + path: rules + content: + apiGroups: + - "" + resources: + - namespaces + verbs: + - list - documentIndex: 1 contains: path: rules diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index fc5d175dfa..d18563ab16 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -77,12 +77,13 @@ cni: # cni.enabled=false) on additional gateway releases that share the cluster's # existing OpenShell CNI installation. external: false - # -- Namespaces whose pods the plugin inspects at CNI ADD. Pods in other - # namespaces are passed through WITHOUT a Kubernetes API lookup, so an API/RBAC - # problem cannot block unrelated workloads. Empty defaults to the release's - # sandbox namespace. A cluster running multiple OpenShell releases must list - # EVERY sandbox namespace here on the singleton (a pod in an unlisted namespace - # is not enforced). + # -- Optional STATIC additions to the plugin's enforcement allowlist. The + # allowlist is normally built automatically: each cni-sidecar release labels its + # sandbox namespace (openshell.ai/sandbox=true) via a helm hook and the singleton + # aggregates all labeled namespaces. This value is unioned with that discovery + # for namespaces you want enforced without the label. Pods in namespaces neither + # labeled nor listed are passed through without a Kubernetes API lookup (blast + # radius guard). Empty additionally seeds the owner release's sandbox namespace. sandboxNamespaces: [] # -- CNI installer strategy. "conflist" appends the OpenShell plugin to an # existing CNI .conflist (k3s / vanilla). "multus-chain" writes a standalone diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index f4bd61c69c..48037942c4 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -143,31 +143,25 @@ is routed through the OpenShell network sidecar and enforced by policy. To run **additional** gateway releases on the same cluster, do not install a second CNI — reuse the singleton with `cni.enabled=false`, `cni.external=true` (`-f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml`). - - -The singleton only enforces pods in namespaces listed in its -`cni.sandboxNamespaces` allowlist (default: the owner release's sandbox -namespace); pods elsewhere are passed through **unenforced**. Before enabling an -additional release you **must** add its sandbox namespace to the singleton -owner's allowlist and let every node re-reconcile: - -```shell -helm upgrade deploy/helm/openshell -n \ - --reuse-values --set 'cni.sandboxNamespaces={openshell,team-b}' -``` - -Only then install the additional gateway. Skipping this step silently leaves the -new release's sandboxes without egress enforcement. - +Registration is automatic: each `cni-sidecar` release runs a helm hook that +labels its sandbox namespace `openshell.ai/sandbox=true`, and the singleton +discovers every labeled namespace and adds it to the enforcement allowlist within +one reconcile (~30s) — no manual edit of the singleton owner is required. +(`cni.sandboxNamespaces` remains available as an optional static addition to the +allowlist for namespaces you want enforced without the label.) On node reboot the Multus chain file (under `/run`, tmpfs) is wiped while the `openshell.ai/cni-ready` Node label persists, so a sandbox could briefly schedule -onto a node before enforcement is re-established. To close this window, apply the -boot-time taint MachineConfig at +onto a node before enforcement is re-established. To **narrow** this window, apply +the boot-time taint MachineConfig at `deploy/helm/openshell/examples/cni-sidecar-openshift-boot-taint.yaml` (read its caveats first — it is operator-provided node config and requires taint RBAC). The -CNI DaemonSet tolerates the taint and removes it once enforcement is ready. +CNI DaemonSet tolerates the taint and removes it once enforcement is ready. This +does not fully close the window: the taint is applied after the kubelet starts, so +a brief reboot race remains. Use `conflist` mode (plugin stored on persistent +disk, unaffected by reboot), or cordon/drain nodes before rebooting, to avoid it +entirely. ## Next Steps diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index d014dc2261..ef29dfa3e9 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -236,15 +236,16 @@ provide the expected network enforcement. The CNI installer is a **cluster singleton**. Its chained plugin enforces pods that carry the OpenShell annotations (set only by a gateway on its own sandbox -pods) **and** whose namespace is in the plugin's `cni.sandboxNamespaces` -allowlist (default: the owner release's sandbox namespace). Pods in other -namespaces are passed through without an API lookup, so the allowlist bounds the -blast radius. Enable the installer (`cni.enabled=true`) in exactly one release -per cluster; additional gateway releases set `cni.enabled=false` and -`cni.external=true` to share it — and their sandbox namespace must be added to -the owner's `cni.sandboxNamespaces` first (see -[OpenShift](/kubernetes/openshift#enable-the-cni-sidecar-topology)). The installer -treats any `openshell-cni` chained entry as its own and upgrades it in place. +pods) **and** whose namespace is in the plugin's enforcement allowlist. Pods in +other namespaces are passed through without an API lookup, so the allowlist +bounds the blast radius. The allowlist is built automatically: each `cni-sidecar` +release labels its sandbox namespace `openshell.ai/sandbox=true` (via a helm +hook) and the singleton aggregates every labeled namespace each reconcile — so +enable the installer (`cni.enabled=true`) in exactly one release per cluster, +and additional releases (`cni.enabled=false`, `cni.external=true`) are discovered +and enforced automatically. `cni.sandboxNamespaces` is an optional static +addition. The installer treats any `openshell-cni` chained entry as its own and +upgrades it in place. ## Credential Exposure diff --git a/rfc/cni-sidecar-topology-DRAFT.md b/rfc/cni-sidecar-topology-DRAFT.md index 14e2ca463b..6b926952e5 100644 --- a/rfc/cni-sidecar-topology-DRAFT.md +++ b/rfc/cni-sidecar-topology-DRAFT.md @@ -214,17 +214,19 @@ settings live in the overlay `ci/values-cni-sidecar-openshift.yaml`. Guardrails enforced at template render time: -- `supervisor.topology=cni-sidecar` requires `cni.enabled=true` (existing - guard). +- `supervisor.topology=cni-sidecar` requires the CNI singleton: `cni.enabled=true` + to install it here, or `cni.external=true` to use one installed by another + release. - `cni.mode` must be `conflist` or `multus-chain`. - `cni.mode=multus-chain` requires `cni.chainDir`. Because `cni.chainDir` is tmpfs (under `/run`), credentials cannot live there. The DaemonSet writes kubeconfig/token/ca.crt to the persistent `cni.stateDir`, -references those paths from the plugin `.conf`, and its periodic loop (already -refreshing the SA token every 300s) re-asserts the chain `.conf` if it goes -missing, so the config survives a Multus restart; a node reboot is covered by -the DaemonSet re-writing on startup. +references those paths from the plugin `.conf`, and its periodic reconcile loop +(every 30s, which also refreshes the SA token) re-asserts the chain `.conf` if it +goes missing, so the config survives a Multus restart. A node reboot wipes the +tmpfs chain file; the DaemonSet re-writes it on startup, and a boot-time taint +(see Risks) narrows the pre-reconcile scheduling window. ### SCC model (OpenShift) @@ -412,17 +414,24 @@ behavior is behind a default-off value. the next reconcile re-checks. Operators diagnose via `kubectl logs daemonset/openshell-cni` and the tailed plugin log. - **Cluster-singleton CNI.** The chained plugin and readiness label are - cluster-global host state. Rather than key them to a per-release namespace list - (which two releases would fight over), the installer is a cluster singleton: the - plugin enforces any pod carrying the OpenShell annotations — set only by a - gateway on its own sandbox pods — so one installation serves every release - across all namespaces (empty namespace allowlist, fixed `openshell` owner, - release-independent resource names). Enable it in one release per cluster; - additional gateway releases set `cni.enabled=false` + `cni.external=true` to - share it. Readiness is bound to a `configVersion` so a stale entry is repaired - before the node is re-marked ready, and the installer refuses to overwrite a - chained plugin entry with a non-OpenShell owner (fail closed). The singleton's - `pods get` RBAC is necessarily cluster-scoped. + cluster-global host state. Rather than key them to a per-release identity (which + two releases would fight over), the installer is a cluster singleton with a + fixed `openshell` owner and release-independent resource names. Its plugin + enforces pods that carry the OpenShell annotations (set only by a gateway on its + own sandbox pods) **and** whose namespace is in the plugin's `sandboxNamespaces` + allowlist; pods elsewhere pass through without an API lookup, bounding blast + radius. The allowlist is built automatically: each `cni-sidecar` release labels + its sandbox namespace `openshell.ai/sandbox=true` (a helm hook), and the + installer's reconcile aggregates every labeled namespace (unioned with the + optional static `cni.sandboxNamespaces`) into the config. So an additional + `cni.external` release is discovered and enforced within one reconcile with no + manual allowlist edit. Enable the installer in one release per cluster. + Readiness is bound to a `configVersion` (over the aggregated allowlist and + config) so a stale entry is repaired before the node is re-marked ready, and the + installer treats any `openshell-cni` chained entry as its own to upgrade in place + (the conflist patch preserves all other plugins). The singleton's `pods get` and + `namespaces list` RBAC are cluster-scoped so it can discover labeled namespaces + and read sandbox pods in any of them. - **Dependency on Multus aux chain.** `multus-chain` requires the Multus thick daemon to have `auxiliaryCNIChainName` set. It is default on OpenShift 4.x but not guaranteed everywhere. Mitigation: document the requirement; the chart does From e70f3d35b97295f15cff113b4c6b7be6236f860c Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Tue, 4 Aug 2026 18:22:38 -0400 Subject: [PATCH 19/20] feat(cni-sidecar): Helm-owned registration marker + wait-for-ack gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round eight: close the registration race and give registration a proper lifecycle. - helm/cni: replace the namespace-label hook with a Helm-owned marker ConfigMap (openshell.ai/cni-registration=true) in each release's sandbox namespace. The installer discovers namespaces by listing marker ConfigMaps (RBAC: configmaps list). Because the marker is a normal Helm resource, uninstalling a release or changing its sandboxNamespace removes the registration automatically — no orphaned allowlist entries. This also drops the hook Job, its namespace-patch RBAC, and the per-release ClusterRole name collision. - cni/helm: close the discovery-window race. The installer publishes on each node the CSV of namespaces it enforces (openshell.ai/cni-sandbox-namespaces annotation, via set-node-coverage); every gateway runs a wait-coverage init container that blocks serving until every cni-ready node acknowledges the gateway's namespace. A newly-registered release therefore cannot create sandboxes before enforcement is confirmed cluster-wide. - docs: update topology, architecture, RFC, values, and the debug skill to the marker + wait-for-ack model; fix the RFC boot-taint 'closes' -> 'narrows' and the design spec 300s -> 30s reconcile interval. Signed-off-by: Russell Bryant --- .../skills/debug-openshell-cluster/SKILL.md | 24 ++- architecture/compute-runtimes.md | 25 +-- crates/openshell-cni/src/lib.rs | 153 +++++++++++++----- crates/openshell-cni/src/main.rs | 3 +- deploy/helm/openshell/README.md | 2 +- .../openshell/templates/_gateway-workload.tpl | 14 ++ .../openshell/templates/cni-daemonset.yaml | 9 ++ .../templates/cni-namespace-label-hook.yaml | 92 ----------- deploy/helm/openshell/templates/cni-rbac.yaml | 7 +- .../openshell/templates/cni-registration.yaml | 24 +++ .../openshell/tests/cni_daemonset_test.yaml | 6 +- .../tests/cni_gateway_wait_test.yaml | 34 ++++ .../tests/cni_namespace_label_hook_test.yaml | 72 --------- .../helm/openshell/tests/cni_rbac_test.yaml | 4 +- .../tests/cni_registration_test.yaml | 54 +++++++ deploy/helm/openshell/values.yaml | 13 +- docs/kubernetes/openshift.mdx | 16 +- docs/kubernetes/topology.mdx | 13 +- ...-30-cni-sidecar-openshift-multus-design.md | 7 +- rfc/cni-sidecar-topology-DRAFT.md | 44 ++--- 20 files changed, 351 insertions(+), 265 deletions(-) delete mode 100644 deploy/helm/openshell/templates/cni-namespace-label-hook.yaml create mode 100644 deploy/helm/openshell/templates/cni-registration.yaml create mode 100644 deploy/helm/openshell/tests/cni_gateway_wait_test.yaml delete mode 100644 deploy/helm/openshell/tests/cni_namespace_label_hook_test.yaml create mode 100644 deploy/helm/openshell/tests/cni_registration_test.yaml diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index c5e88bf0bf..707e1ca674 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -423,20 +423,30 @@ A gateway release using `cni-sidecar` with neither `cni.enabled` nor sandboxes are unenforced, confirm the singleton exists and is Ready. **Namespace allowlist (silent unenforced sandboxes).** The plugin only inspects -pods whose namespace is in its `sandboxNamespaces` allowlist -(`cni.sandboxNamespaces`, default the owner's sandbox namespace); pods elsewhere -are passed through unenforced. If a sandbox reaches 2/2 but its egress is NOT -blocked, confirm its namespace is in the installed allowlist — a common cause is -an additional `cni.external=true` release whose namespace was never added to the -owner's list: +pods whose namespace is in its `sandboxNamespaces` allowlist. That allowlist is +built automatically from Helm-owned marker ConfigMaps +(`openshell.ai/cni-registration=true`) — one per cni-sidecar release, in its +sandbox namespace — unioned with the static `cni.sandboxNamespaces`. If a sandbox +reaches 2/2 but its egress is NOT blocked, confirm its namespace is registered and +in the installed allowlist: ```bash -# On a CNI installer pod, read the installed plugin config's sandboxNamespaces: +# Registration markers (their namespaces are what the singleton enforces): +kubectl get configmaps -A -l openshell.ai/cni-registration=true +# Installed plugin config's aggregated sandboxNamespaces: kubectl -n exec ds/openshell-cni -c install-cni -- \ sh -c 'cat /host/run/multus/cni/net.d/vendor-cni-chain/openshell-cni.conf 2>/dev/null \ || cat /host/etc/cni/net.d/*.conflist' ``` +**Gateway stuck in Init (`wait-cni-coverage`).** A cni-sidecar gateway pod has a +`wait-cni-coverage` init container that blocks until every `cni-ready` node's +`openshell.ai/cni-sandbox-namespaces` annotation includes the gateway's namespace. +If the gateway is stuck initializing, the singleton has not yet acknowledged the +namespace on all nodes — check the per-node coverage: +`kubectl get nodes -o custom-columns=NAME:.metadata.name,COVERAGE:.metadata.annotations.openshell\.ai/cni-sandbox-namespaces` +and confirm a marker ConfigMap exists for that namespace. + **Node reboot / boot taint.** In `multus-chain` mode the chain file lives under `/run` (tmpfs) and is wiped on reboot while the `cni-ready` label persists. Clusters that apply the optional boot-time taint (`openshell.ai/cni-not-ready`, diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 69013c3a56..d757431fbc 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -306,21 +306,28 @@ sandbox pods) **and** whose namespace is in the plugin's `sandboxNamespaces` allowlist. Pods in other namespaces are passed through without a Kubernetes API lookup, so the allowlist bounds the blast radius of the per-pod annotation read. The allowlist is built **automatically**: each `cni-sidecar` gateway release -labels its sandbox namespace `openshell.ai/sandbox=true` (a helm hook runs -`openshell-cni label-namespace`), and the installer's reconcile aggregates every -labeled namespace (via `list-sandbox-namespaces`) — unioned with the optional -static `cni.sandboxNamespaces` — into the plugin config. An additional -`cni.external` release is therefore discovered and enforced within one reconcile, -with no manual allowlist edit. The installer resources use a fixed +ships a Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true`) in its +sandbox namespace, and the installer's reconcile aggregates every marker's +namespace (via `list-sandbox-namespaces`) — unioned with the optional static +`cni.sandboxNamespaces` — into the plugin config. Because the marker is a normal +Helm resource, uninstalling a release or changing its `sandboxNamespace` removes +the registration (no orphaned allowlist entries). An additional `cni.external` +release is discovered within one reconcile, with no manual allowlist edit. To +eliminate the discovery-window race, the installer publishes on each node the CSV +of namespaces it currently enforces (`openshell.ai/cni-sandbox-namespaces` +annotation, via `set-node-coverage`), and every gateway runs a `wait-coverage` +init container that blocks until each enforcement-ready node acknowledges the +gateway's namespace — so a newly-registered release does not serve sandboxes until +enforcement is confirmed cluster-wide. The installer resources use a fixed release-independent name, the plugin config carries a fixed `openshell` owner, and a `configVersion` (over the aggregated allowlist and config) binds readiness so a stale-version entry is repaired before the node is re-marked ready. Install the singleton (`cni.enabled=true`) in one release per cluster; additional releases set `cni.enabled=false` + `cni.external=true`. The installer treats any `openshell-cni` chained entry as its own and upgrades it in place (the conflist patch preserves -all other plugins). The `pods get` and `namespaces list` grants are cluster-scoped -so the one installer can discover labeled namespaces and read sandbox pods in any -of them. +all other plugins). The `pods get` and `configmaps list` grants are cluster-scoped +so the one installer can discover marker ConfigMaps and read sandbox pods in any +allowlisted namespace. On OpenShift, binary-aware network policy also requires a purpose-built SecurityContextConstraints for sandbox pods: the network sidecar runs as UID 0 diff --git a/crates/openshell-cni/src/lib.rs b/crates/openshell-cni/src/lib.rs index c2fa87d552..1a5194b947 100644 --- a/crates/openshell-cni/src/lib.rs +++ b/crates/openshell-cni/src/lib.rs @@ -37,10 +37,15 @@ const NODE_NOT_READY_TAINT_KEY: &str = "openshell.ai/cni-not-ready"; const NODE_NOT_READY_TAINT_EFFECT: &str = "NoSchedule"; /// Namespace label a gateway release stamps on its sandbox namespace so the CNI /// singleton discovers and enforces it automatically — no manual allowlist edit -/// per release. The installer aggregates all namespaces carrying this label into -/// the plugin's `sandboxNamespaces`. Must stay in sync with the label written by -/// the namespace-label hook. -const SANDBOX_NAMESPACE_LABEL: &str = "openshell.ai/sandbox"; +/// per release. The installer aggregates every namespace containing a marker +/// `ConfigMap` with this label into the plugin's `sandboxNamespaces`. The marker is +/// a Helm-owned resource, so uninstalling a release (or changing its sandbox +/// namespace) removes the marker and de-registers the namespace automatically. +const CNI_REGISTRATION_LABEL: &str = "openshell.ai/cni-registration"; +/// Node annotation the installer sets to the sorted CSV of the namespaces it +/// currently enforces on that node, so the gateway can wait for cluster-wide +/// acknowledgement of its namespace before it serves sandboxes. +const NODE_COVERAGE_ANNOTATION: &str = "openshell.ai/cni-sandbox-namespaces"; #[allow(dead_code)] const OPENSHELL_TABLE: &str = "openshell_sidecar_bypass"; #[allow(dead_code)] @@ -247,33 +252,64 @@ pub fn node_ready(args: &[String]) -> Result<()> { } } -/// Entry point for `openshell-cni label-namespace `. -/// -/// Run by the gateway's namespace-label helm hook so the CNI singleton -/// auto-discovers this release's sandbox namespace. Stamps -/// `openshell.ai/sandbox=true`. -pub fn label_namespace(args: &[String]) -> Result<()> { - let namespace = args - .first() - .ok_or_else(|| miette::miette!("label-namespace requires a namespace argument"))?; - let client = KubeApiClient::from_in_cluster()?; - client.patch_namespace_label(namespace, SANDBOX_NAMESPACE_LABEL, Some("true")) -} - /// Entry point for `openshell-cni list-sandbox-namespaces`. /// -/// Run by the installer reconcile to aggregate every namespace carrying -/// `openshell.ai/sandbox=true` into the plugin allowlist. Prints one namespace -/// name per line. +/// Run by the installer reconcile to aggregate every namespace that contains a +/// Helm-owned registration marker `ConfigMap` (label +/// `openshell.ai/cni-registration=true`) into the plugin allowlist. Prints one +/// namespace name per line. pub fn list_sandbox_namespaces() -> Result<()> { let client = KubeApiClient::from_in_cluster()?; - let namespaces = client.list_namespaces_by_label(SANDBOX_NAMESPACE_LABEL, "true")?; + let namespaces = client.list_registration_namespaces(CNI_REGISTRATION_LABEL)?; for ns in namespaces { println!("{ns}"); } Ok(()) } +/// Entry point for `openshell-cni set-node-coverage `. +/// +/// Run by the installer reconcile to publish, on its own node, the sorted CSV of +/// namespaces it currently enforces, so gateways can wait for cluster-wide +/// acknowledgement of their namespace before serving sandboxes. +pub fn set_node_coverage(args: &[String]) -> Result<()> { + let csv = args.first().map_or("", String::as_str); + let node = std::env::var("NODE_NAME") + .map_err(|_| miette::miette!("NODE_NAME env var is required for set-node-coverage"))?; + let client = KubeApiClient::from_in_cluster()?; + client.patch_node_annotation(&node, NODE_COVERAGE_ANNOTATION, csv) +} + +/// Entry point for `openshell-cni wait-coverage `, run as a gateway +/// init container so the gateway does not serve sandboxes until every +/// enforcement-ready node acknowledges the namespace. +/// +/// Blocks until at least one node carries the `cni-ready` label and every such +/// node's coverage annotation includes the namespace. Times out (non-zero exit, +/// failing the init container fail-closed) after a bounded wait. +pub fn wait_coverage(args: &[String]) -> Result<()> { + let namespace = args + .first() + .ok_or_else(|| miette::miette!("wait-coverage requires a namespace argument"))?; + let client = KubeApiClient::from_in_cluster()?; + // ~5 minutes of 5s polls; fail closed if enforcement never converges. + for _ in 0..60 { + match client.namespace_covered_on_all_ready_nodes( + namespace, + NODE_READY_LABEL, + NODE_COVERAGE_ANNOTATION, + ) { + Ok(true) => return Ok(()), + Ok(false) => {} + Err(error) => eprintln!("wait-coverage: transient error: {error:?}"), + } + std::thread::sleep(std::time::Duration::from_secs(5)); + } + Err(miette::miette!( + "namespace {namespace} not enforced on all cni-ready nodes within timeout" + )) +} + /// Entry point for `openshell-cni daemonset-active`. /// /// Used by the installer `preStop` hook to distinguish an ordinary pod @@ -666,11 +702,10 @@ impl KubeApiClient { Ok(()) } - /// Sets a label on a namespace via a JSON merge patch (null value removes). - fn patch_namespace_label(&self, namespace: &str, key: &str, value: Option<&str>) -> Result<()> { - let url = format!("{}/api/v1/namespaces/{}", self.server, namespace); - let label_value = value.map_or(Value::Null, |v| Value::String(v.to_string())); - let body = serde_json::json!({ "metadata": { "labels": { key: label_value } } }); + /// Sets a single node annotation to `value` via a JSON merge patch. + fn patch_node_annotation(&self, node: &str, key: &str, value: &str) -> Result<()> { + let url = format!("{}/api/v1/nodes/{}", self.server, node); + let body = serde_json::json!({ "metadata": { "annotations": { key: Value::String(value.to_string()) } } }); let response = self .client .patch(url) @@ -682,21 +717,22 @@ impl KubeApiClient { .body(serde_json::to_vec(&body).into_diagnostic()?) .send() .into_diagnostic() - .wrap_err("failed to patch namespace label")?; + .wrap_err("failed to patch node annotation")?; if !response.status().is_success() { return Err(miette::miette!( - "Kubernetes API returned {} while labeling namespace {}", + "Kubernetes API returned {} while annotating node {}", response.status(), - namespace + node )); } Ok(()) } - /// Returns the names of all namespaces carrying `key=value`. - fn list_namespaces_by_label(&self, key: &str, value: &str) -> Result> { - let selector = format!("{key}={value}"); - let url = format!("{}/api/v1/namespaces", self.server); + /// Returns the namespaces of all registration marker `ConfigMaps` (those + /// carrying `key=true`) across the cluster. + fn list_registration_namespaces(&self, key: &str) -> Result> { + let selector = format!("{key}=true"); + let url = format!("{}/api/v1/configmaps", self.server); let response = self .client .get(url) @@ -704,24 +740,65 @@ impl KubeApiClient { .bearer_auth(&self.token) .send() .into_diagnostic() - .wrap_err("failed to list namespaces by label")?; + .wrap_err("failed to list registration ConfigMaps")?; if !response.status().is_success() { return Err(miette::miette!( - "Kubernetes API returned {} while listing namespaces", + "Kubernetes API returned {} while listing ConfigMaps", response.status() )); } let list = response.json::().into_diagnostic()?; - let names = list["items"] + let mut namespaces: Vec = list["items"] .as_array() .map(|items| { items .iter() - .filter_map(|item| item["metadata"]["name"].as_str().map(str::to_string)) + .filter_map(|item| item["metadata"]["namespace"].as_str().map(str::to_string)) .collect() }) .unwrap_or_default(); - Ok(names) + namespaces.sort(); + namespaces.dedup(); + Ok(namespaces) + } + + /// Returns true when at least one node carries `ready_label` and every such + /// node's `coverage_annotation` (a comma-separated list) contains `namespace`. + fn namespace_covered_on_all_ready_nodes( + &self, + namespace: &str, + ready_label: &str, + coverage_annotation: &str, + ) -> Result { + let url = format!("{}/api/v1/nodes", self.server); + let response = self + .client + .get(url) + .query(&[("labelSelector", ready_label)]) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to list cni-ready nodes")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while listing nodes", + response.status() + )); + } + let list = response.json::().into_diagnostic()?; + let Some(nodes) = list["items"].as_array() else { + return Ok(false); + }; + if nodes.is_empty() { + return Ok(false); + } + Ok(nodes.iter().all(|node| { + node["metadata"]["annotations"][coverage_annotation] + .as_str() + .unwrap_or_default() + .split(',') + .any(|ns| ns == namespace) + })) } /// Removes the given taint (key + effect) from the node if present, using a diff --git a/crates/openshell-cni/src/main.rs b/crates/openshell-cni/src/main.rs index 8c25f77e78..bf885a9eb2 100644 --- a/crates/openshell-cni/src/main.rs +++ b/crates/openshell-cni/src/main.rs @@ -8,8 +8,9 @@ fn main() { let args: Vec = std::env::args().collect(); let result = match args.get(1).map(String::as_str) { Some("node-ready") => openshell_cni::node_ready(&args[2..]), - Some("label-namespace") => openshell_cni::label_namespace(&args[2..]), Some("list-sandbox-namespaces") => openshell_cni::list_sandbox_namespaces(), + Some("set-node-coverage") => openshell_cni::set_node_coverage(&args[2..]), + Some("wait-coverage") => openshell_cni::wait_coverage(&args[2..]), Some("daemonset-active") => openshell_cni::daemonset_active(), // Internal: re-exec'd inside a pod netns to probe for IPv6 (see // netns_requires_ipv6_enforcement). Not part of the public CLI surface. diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 1dfea8b54b..c8e621ec22 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -164,7 +164,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | cni.nodeSelector | object | `{}` | | | cni.openshift.privilegedSCC | bool | `false` | Grant the privileged SecurityContextConstraints to the CNI ServiceAccount. Required on OpenShift for the privileged installer DaemonSet. Creates a ClusterRole + ClusterRoleBinding. | | cni.resources | object | `{}` | | -| cni.sandboxNamespaces | list | `[]` | Optional STATIC additions to the plugin's enforcement allowlist. The allowlist is normally built automatically: each cni-sidecar release labels its sandbox namespace (openshell.ai/sandbox=true) via a helm hook and the singleton aggregates all labeled namespaces. This value is unioned with that discovery for namespaces you want enforced without the label. Pods in namespaces neither labeled nor listed are passed through without a Kubernetes API lookup (blast radius guard). Empty additionally seeds the owner release's sandbox namespace. | +| cni.sandboxNamespaces | list | `[]` | Optional STATIC additions to the plugin's enforcement allowlist. The allowlist is normally built automatically: each cni-sidecar release ships a Helm-owned marker ConfigMap (openshell.ai/cni-registration=true) in its sandbox namespace and the singleton aggregates all marker namespaces. This value is unioned with that discovery for namespaces you want enforced without a marker. Pods in namespaces neither registered nor listed are passed through without a Kubernetes API lookup (blast-radius guard). Empty additionally seeds the owner release's sandbox namespace. | | cni.stateDir | string | `""` | Persistent host directory for plugin credentials (kubeconfig, token, ca.crt) in "multus-chain" mode. Empty falls back to confDir. Must not be a tmpfs path such as one under /run. | | cni.tolerations | list | `[]` | | | fullnameOverride | string | `""` | Override the full generated resource name. | diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl index e9783a88ce..bb8d5b8d12 100644 --- a/deploy/helm/openshell/templates/_gateway-workload.tpl +++ b/deploy/helm/openshell/templates/_gateway-workload.tpl @@ -37,6 +37,20 @@ spec: {{- end }} securityContext: {{- toYaml .Values.podSecurityContext | nindent 4 }} + {{- if eq (.Values.supervisor.topology | default "combined") "cni-sidecar" }} + # Wait for cluster-wide CNI acknowledgement of this gateway's sandbox namespace + # before the gateway serves. Blocks until every cni-ready node reports the + # namespace in its coverage annotation, so a newly-registered (e.g. additional + # cni.external) release cannot create sandboxes during the discovery window. + initContainers: + - name: wait-cni-coverage + image: {{ include "openshell.cniImage" . | quote }} + imagePullPolicy: {{ include "openshell.cniImagePullPolicy" . | quote }} + command: + - /openshell-cni + - wait-coverage + - {{ .Values.server.sandboxNamespace | default .Release.Namespace | quote }} + {{- end }} containers: - name: openshell-gateway securityContext: diff --git a/deploy/helm/openshell/templates/cni-daemonset.yaml b/deploy/helm/openshell/templates/cni-daemonset.yaml index 4991277c8b..c612a9d6c6 100644 --- a/deploy/helm/openshell/templates/cni-daemonset.yaml +++ b/deploy/helm/openshell/templates/cni-daemonset.yaml @@ -159,6 +159,15 @@ spec: mark_ready() { "${openshell_cni}" node-ready --set \ || echo "OpenShell CNI: failed to mark node ready" >&2 + publish_coverage + } + # Publish, on this node, the CSV of namespaces we currently enforce + # (from the installed plugin config) so gateways can wait for + # cluster-wide acknowledgement of their namespace before serving. + publish_coverage() { + _csv="$(jq -r '(.openshell.sandboxNamespaces // []) | join(",")' "${plugin_config}" 2>/dev/null || echo "")" + "${openshell_cni}" set-node-coverage "${_csv}" \ + || echo "OpenShell CNI: failed to publish coverage" >&2 } # Fencing (clearing readiness) is security-critical: while enforcement # is not in place the node must not advertise readiness. Retry a few diff --git a/deploy/helm/openshell/templates/cni-namespace-label-hook.yaml b/deploy/helm/openshell/templates/cni-namespace-label-hook.yaml deleted file mode 100644 index e394e01426..0000000000 --- a/deploy/helm/openshell/templates/cni-namespace-label-hook.yaml +++ /dev/null @@ -1,92 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -{{- if eq (.Values.supervisor.topology | default "combined") "cni-sidecar" }} -{{- $sandboxNamespace := .Values.server.sandboxNamespace | default .Release.Namespace }} -{{- $name := printf "%s-cni-ns-label" (include "openshell.fullname" .) | trunc 63 | trimSuffix "-" }} -# Labels this release's sandbox namespace openshell.ai/sandbox=true so the CNI -# singleton auto-discovers and enforces it — no manual allowlist edit, for the -# owner release and any additional cni.external release alike. The label lets the -# installer aggregate every OpenShell sandbox namespace into the plugin allowlist. -apiVersion: v1 -kind: ServiceAccount -metadata: - name: {{ $name }} - namespace: {{ .Release.Namespace }} - labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni ---- -# Minimal: get/patch on this release's sandbox namespace only (namespaces are -# cluster-scoped, so resourceNames-scoped ClusterRole is the tightest grant). -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: {{ $name }} - labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni -rules: - - apiGroups: - - "" - resources: - - namespaces - resourceNames: - - {{ $sandboxNamespace }} - verbs: - - get - - patch ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: {{ $name }} - labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: {{ $name }} -subjects: - - kind: ServiceAccount - name: {{ $name }} - namespace: {{ .Release.Namespace }} ---- -apiVersion: batch/v1 -kind: Job -metadata: - name: {{ $name }} - namespace: {{ .Release.Namespace }} - labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: cni - annotations: - # Run after the release's resources (and their RBAC) are in place, on both - # install and upgrade. Re-runs are idempotent (label set --overwrite). - "helm.sh/hook": post-install,post-upgrade - "helm.sh/hook-weight": "5" - "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded -spec: - backoffLimit: 6 - template: - metadata: - labels: - {{- include "openshell.selectorLabels" . | nindent 8 }} - app.kubernetes.io/component: cni - spec: - restartPolicy: Never - serviceAccountName: {{ $name }} - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - containers: - - name: label-namespace - image: {{ include "openshell.cniImage" . | quote }} - imagePullPolicy: {{ include "openshell.cniImagePullPolicy" . | quote }} - command: - - /openshell-cni - - label-namespace - - {{ $sandboxNamespace | quote }} -{{- end }} diff --git a/deploy/helm/openshell/templates/cni-rbac.yaml b/deploy/helm/openshell/templates/cni-rbac.yaml index 393e5cd202..77abd3f35a 100644 --- a/deploy/helm/openshell/templates/cni-rbac.yaml +++ b/deploy/helm/openshell/templates/cni-rbac.yaml @@ -31,12 +31,13 @@ rules: - pods verbs: - get - # list namespaces to aggregate those labeled openshell.ai/sandbox=true into the - # plugin allowlist (automatic multi-release discovery). + # list registration marker ConfigMaps (openshell.ai/cni-registration=true) to + # aggregate their namespaces into the plugin allowlist (automatic multi-release + # discovery with Helm-owned lifecycle). - apiGroups: - "" resources: - - namespaces + - configmaps verbs: - list - apiGroups: diff --git a/deploy/helm/openshell/templates/cni-registration.yaml b/deploy/helm/openshell/templates/cni-registration.yaml new file mode 100644 index 0000000000..e41e2a198a --- /dev/null +++ b/deploy/helm/openshell/templates/cni-registration.yaml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if eq (.Values.supervisor.topology | default "combined") "cni-sidecar" }} +{{- $sandboxNamespace := .Values.server.sandboxNamespace | default .Release.Namespace }} +# Helm-owned registration marker for this release's sandbox namespace. The CNI +# singleton lists ConfigMaps labeled openshell.ai/cni-registration=true and +# aggregates their namespaces into the enforcement allowlist, so every +# cni-sidecar release (owner or cni.external) is discovered automatically. As a +# normal Helm resource it is deleted on uninstall and moved on a sandboxNamespace +# change, so registrations have proper lifecycle — no orphaned state. +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "openshell.fullname" . }}-cni-registration + namespace: {{ $sandboxNamespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: cni + openshell.ai/cni-registration: "true" +data: + release: {{ .Release.Name | quote }} + releaseNamespace: {{ .Release.Namespace | quote }} +{{- end }} diff --git a/deploy/helm/openshell/tests/cni_daemonset_test.yaml b/deploy/helm/openshell/tests/cni_daemonset_test.yaml index 62dd44f401..5a3d3f5ebc 100644 --- a/deploy/helm/openshell/tests/cni_daemonset_test.yaml +++ b/deploy/helm/openshell/tests/cni_daemonset_test.yaml @@ -114,10 +114,14 @@ tests: - matchRegex: path: spec.template.spec.containers[0].command[2] pattern: 'owner="openshell"' - # Auto-discovery: reconcile aggregates labeled namespaces into the allowlist. + # Auto-discovery: reconcile aggregates marker-ConfigMap namespaces, and + # publishes per-node coverage for the gateway wait-for-ack gate. - matchRegex: path: spec.template.spec.containers[0].command[2] pattern: 'list-sandbox-namespaces' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'set-node-coverage' - it: aggregates multiple sandbox namespaces into the allowlist template: templates/cni-daemonset.yaml diff --git a/deploy/helm/openshell/tests/cni_gateway_wait_test.yaml b/deploy/helm/openshell/tests/cni_gateway_wait_test.yaml new file mode 100644 index 0000000000..efdc59dc61 --- /dev/null +++ b/deploy/helm/openshell/tests/cni_gateway_wait_test.yaml @@ -0,0 +1,34 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: gateway cni-coverage wait init +templates: + - templates/gateway-config.yaml + - templates/statefulset.yaml +release: + name: openshell + namespace: openshell +tests: + - it: has no wait init container by default + template: templates/statefulset.yaml + asserts: + - notExists: + path: spec.template.spec.initContainers + + # cni-sidecar gateways must not serve until their namespace is enforced + # cluster-wide, enforced by a wait-coverage init container. + - it: adds the wait-coverage init container in cni-sidecar mode + template: templates/statefulset.yaml + set: + supervisor.topology: cni-sidecar + cni.external: true + asserts: + - equal: + path: spec.template.spec.initContainers[0].name + value: wait-cni-coverage + - equal: + path: spec.template.spec.initContainers[0].command + value: + - /openshell-cni + - wait-coverage + - openshell diff --git a/deploy/helm/openshell/tests/cni_namespace_label_hook_test.yaml b/deploy/helm/openshell/tests/cni_namespace_label_hook_test.yaml deleted file mode 100644 index 32ae1184f8..0000000000 --- a/deploy/helm/openshell/tests/cni_namespace_label_hook_test.yaml +++ /dev/null @@ -1,72 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -suite: cni namespace-label hook -templates: - - templates/cni-namespace-label-hook.yaml -release: - name: openshell - namespace: openshell -tests: - - it: does not render without cni-sidecar topology - asserts: - - hasDocuments: - count: 0 - - # Every cni-sidecar release (owner or cni.external) labels its own sandbox - # namespace so the singleton auto-discovers it. - - it: renders the label hook for a cni-sidecar release - set: - supervisor.topology: cni-sidecar - asserts: - - hasDocuments: - count: 4 - - documentIndex: 3 - isKind: - of: Job - - documentIndex: 3 - equal: - path: metadata.annotations["helm.sh/hook"] - value: post-install,post-upgrade - - documentIndex: 3 - equal: - path: spec.template.spec.containers[0].command - value: - - /openshell-cni - - label-namespace - - openshell - - # An additional external release still runs the label hook (topology drives it, - # not cni.enabled), so its namespace is auto-registered with the singleton. - - it: renders the label hook for an external release - set: - supervisor.topology: cni-sidecar - cni.enabled: false - cni.external: true - asserts: - - hasDocuments: - count: 4 - - documentIndex: 3 - isKind: - of: Job - - # RBAC is scoped to only this release's sandbox namespace. - - it: scopes the label ClusterRole to the sandbox namespace - set: - supervisor.topology: cni-sidecar - server.sandboxNamespace: team-b - asserts: - - template: templates/cni-namespace-label-hook.yaml - documentIndex: 1 - contains: - path: rules - content: - apiGroups: - - "" - resources: - - namespaces - resourceNames: - - team-b - verbs: - - get - - patch diff --git a/deploy/helm/openshell/tests/cni_rbac_test.yaml b/deploy/helm/openshell/tests/cni_rbac_test.yaml index 4df6df0d63..9f16877501 100644 --- a/deploy/helm/openshell/tests/cni_rbac_test.yaml +++ b/deploy/helm/openshell/tests/cni_rbac_test.yaml @@ -45,7 +45,7 @@ tests: - pods verbs: - get - # namespaces list: aggregate labeled sandbox namespaces into the allowlist. + # configmaps list: aggregate registration-marker namespaces into the allowlist. - documentIndex: 1 contains: path: rules @@ -53,7 +53,7 @@ tests: apiGroups: - "" resources: - - namespaces + - configmaps verbs: - list - documentIndex: 1 diff --git a/deploy/helm/openshell/tests/cni_registration_test.yaml b/deploy/helm/openshell/tests/cni_registration_test.yaml new file mode 100644 index 0000000000..ea55a121d9 --- /dev/null +++ b/deploy/helm/openshell/tests/cni_registration_test.yaml @@ -0,0 +1,54 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: cni registration marker +templates: + - templates/cni-registration.yaml +release: + name: openshell + namespace: openshell +tests: + - it: does not render without cni-sidecar topology + asserts: + - hasDocuments: + count: 0 + + # Every cni-sidecar release (owner or cni.external) publishes a Helm-owned + # marker ConfigMap so the singleton auto-discovers its sandbox namespace, with + # lifecycle handled by Helm. + - it: renders a labeled marker ConfigMap for a cni-sidecar release + set: + supervisor.topology: cni-sidecar + asserts: + - hasDocuments: + count: 1 + - isKind: + of: ConfigMap + - equal: + path: metadata.labels["openshell.ai/cni-registration"] + value: "true" + - equal: + path: metadata.namespace + value: openshell + + - it: places the marker in the configured sandbox namespace + set: + supervisor.topology: cni-sidecar + server.sandboxNamespace: team-b + asserts: + - equal: + path: metadata.namespace + value: team-b + + # External releases render the marker too (topology drives it, not cni.enabled). + - it: renders the marker for an external release + set: + supervisor.topology: cni-sidecar + cni.enabled: false + cni.external: true + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.labels["openshell.ai/cni-registration"] + value: "true" diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index d18563ab16..82f0ac7ab7 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -78,12 +78,13 @@ cni: # existing OpenShell CNI installation. external: false # -- Optional STATIC additions to the plugin's enforcement allowlist. The - # allowlist is normally built automatically: each cni-sidecar release labels its - # sandbox namespace (openshell.ai/sandbox=true) via a helm hook and the singleton - # aggregates all labeled namespaces. This value is unioned with that discovery - # for namespaces you want enforced without the label. Pods in namespaces neither - # labeled nor listed are passed through without a Kubernetes API lookup (blast - # radius guard). Empty additionally seeds the owner release's sandbox namespace. + # allowlist is normally built automatically: each cni-sidecar release ships a + # Helm-owned marker ConfigMap (openshell.ai/cni-registration=true) in its sandbox + # namespace and the singleton aggregates all marker namespaces. This value is + # unioned with that discovery for namespaces you want enforced without a marker. + # Pods in namespaces neither registered nor listed are passed through without a + # Kubernetes API lookup (blast-radius guard). Empty additionally seeds the owner + # release's sandbox namespace. sandboxNamespaces: [] # -- CNI installer strategy. "conflist" appends the OpenShell plugin to an # existing CNI .conflist (k3s / vanilla). "multus-chain" writes a standalone diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 48037942c4..a59773b223 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -143,12 +143,16 @@ is routed through the OpenShell network sidecar and enforced by policy. To run **additional** gateway releases on the same cluster, do not install a second CNI — reuse the singleton with `cni.enabled=false`, `cni.external=true` (`-f deploy/helm/openshell/examples/cni-sidecar-openshift-extra-release.yaml`). -Registration is automatic: each `cni-sidecar` release runs a helm hook that -labels its sandbox namespace `openshell.ai/sandbox=true`, and the singleton -discovers every labeled namespace and adds it to the enforcement allowlist within -one reconcile (~30s) — no manual edit of the singleton owner is required. -(`cni.sandboxNamespaces` remains available as an optional static addition to the -allowlist for namespaces you want enforced without the label.) +Registration is automatic and race-free: each `cni-sidecar` release ships a +Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true`) in its sandbox +namespace, and the singleton discovers every marker's namespace and adds it to the +enforcement allowlist — no manual edit of the singleton owner is required. Because +the marker is a normal Helm resource, uninstalling the release (or changing its +`sandboxNamespace`) removes the registration automatically. The gateway does not +serve until enforcement is acknowledged: a `wait-cni-coverage` init container +blocks the gateway pod until every enforcement-ready node reports the namespace, +so an additional release never creates sandboxes during the discovery window. +(`cni.sandboxNamespaces` remains available as an optional static addition.) On node reboot the Multus chain file (under `/run`, tmpfs) is wiped while the diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index ef29dfa3e9..0786a2e05c 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -239,11 +239,14 @@ that carry the OpenShell annotations (set only by a gateway on its own sandbox pods) **and** whose namespace is in the plugin's enforcement allowlist. Pods in other namespaces are passed through without an API lookup, so the allowlist bounds the blast radius. The allowlist is built automatically: each `cni-sidecar` -release labels its sandbox namespace `openshell.ai/sandbox=true` (via a helm -hook) and the singleton aggregates every labeled namespace each reconcile — so -enable the installer (`cni.enabled=true`) in exactly one release per cluster, -and additional releases (`cni.enabled=false`, `cni.external=true`) are discovered -and enforced automatically. `cni.sandboxNamespaces` is an optional static +release ships a Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true`) +in its sandbox namespace and the singleton aggregates every marker's namespace +each reconcile — so enable the installer (`cni.enabled=true`) in exactly one +release per cluster, and additional releases (`cni.enabled=false`, +`cni.external=true`) are discovered automatically, with registration removed by +Helm on uninstall. Each gateway waits (via a `wait-cni-coverage` init container) +until every enforcement-ready node acknowledges its namespace before serving, so +there is no discovery-window bypass. `cni.sandboxNamespaces` is an optional static addition. The installer treats any `openshell-cni` chained entry as its own and upgrades it in place. diff --git a/docs/superpowers/specs/2026-07-30-cni-sidecar-openshift-multus-design.md b/docs/superpowers/specs/2026-07-30-cni-sidecar-openshift-multus-design.md index f0d3fad328..973e7ab58b 100644 --- a/docs/superpowers/specs/2026-07-30-cni-sidecar-openshift-multus-design.md +++ b/docs/superpowers/specs/2026-07-30-cni-sidecar-openshift-multus-design.md @@ -94,9 +94,10 @@ New/updated values (OpenShift defaults): `cni.chainDir` is tmpfs (`/run`), so credentials cannot live there. The DaemonSet writes kubeconfig/token/ca.crt to the persistent `cni.stateDir`, and the plugin `.conf` -references those `stateDir` paths. The DaemonSet's periodic loop (already refreshing the -SA token every 300s) additionally **re-asserts the chain `.conf` if missing**, so it -survives a Multus restart; on a full node reboot the DaemonSet re-writes at startup. +references those `stateDir` paths. The DaemonSet's periodic reconcile loop (every 30s, +which also refreshes the SA token) additionally **re-asserts the chain `.conf` if +missing**, so it survives a Multus restart; on a full node reboot the DaemonSet +re-writes at startup. For back-compat, when `cni.mode == conflist` the `stateDir` default resolves to `cni.confDir` (current behavior). diff --git a/rfc/cni-sidecar-topology-DRAFT.md b/rfc/cni-sidecar-topology-DRAFT.md index 6b926952e5..e05f132ae2 100644 --- a/rfc/cni-sidecar-topology-DRAFT.md +++ b/rfc/cni-sidecar-topology-DRAFT.md @@ -403,13 +403,14 @@ behavior is behind a default-off value. - **Node reboot (tmpfs enforcement).** In `multus-chain` mode the chain file lives under `/run` (tmpfs) and is wiped on reboot, but the persistent `cni-ready` label survives, so the label gate alone cannot cover a reboot. A - boot-time `NoSchedule` taint (`openshell.ai/cni-not-ready`) closes this: it is - applied by operator-provided node config (only node config runs before the - scheduler; a MachineConfig example ships under `deploy/helm/openshell/examples/`), - the CNI DaemonSet tolerates it, and `openshell-cni node-ready` removes it once - enforcement is ready (never re-adding it, so a transient unready never - over-repels). `conflist` mode stores the plugin on persistent disk and is - unaffected by reboot. Residual: an ungraceful installer-pod deletion (no + boot-time `NoSchedule` taint (`openshell.ai/cni-not-ready`) **narrows** this (it + does not fully close it): it is applied by operator-provided node config (only + node config runs before the scheduler; a MachineConfig example ships under + `deploy/helm/openshell/examples/`), the CNI DaemonSet tolerates it, and + `openshell-cni node-ready` removes it once enforcement is ready (never re-adding + it, so a transient unready never over-repels). A residual race remains because + the taint is applied after the kubelet starts. `conflist` mode stores the plugin + on persistent disk and is unaffected by reboot. Residual: an ungraceful installer-pod deletion (no `preStop`) can leave a stale `cni-ready=true` until the pod is rescheduled and the next reconcile re-checks. Operators diagnose via `kubectl logs daemonset/openshell-cni` and the tailed plugin log. @@ -420,18 +421,23 @@ behavior is behind a default-off value. enforces pods that carry the OpenShell annotations (set only by a gateway on its own sandbox pods) **and** whose namespace is in the plugin's `sandboxNamespaces` allowlist; pods elsewhere pass through without an API lookup, bounding blast - radius. The allowlist is built automatically: each `cni-sidecar` release labels - its sandbox namespace `openshell.ai/sandbox=true` (a helm hook), and the - installer's reconcile aggregates every labeled namespace (unioned with the - optional static `cni.sandboxNamespaces`) into the config. So an additional - `cni.external` release is discovered and enforced within one reconcile with no - manual allowlist edit. Enable the installer in one release per cluster. - Readiness is bound to a `configVersion` (over the aggregated allowlist and - config) so a stale entry is repaired before the node is re-marked ready, and the - installer treats any `openshell-cni` chained entry as its own to upgrade in place - (the conflist patch preserves all other plugins). The singleton's `pods get` and - `namespaces list` RBAC are cluster-scoped so it can discover labeled namespaces - and read sandbox pods in any of them. + radius. The allowlist is built automatically: each `cni-sidecar` release ships a + Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true`) in its sandbox + namespace, and the installer's reconcile aggregates every marker's namespace + (unioned with the optional static `cni.sandboxNamespaces`) into the config. As a + Helm resource the marker is removed on uninstall / sandboxNamespace change, so + registrations have proper lifecycle. An additional `cni.external` release is + discovered within one reconcile with no manual allowlist edit. To close the + discovery-window race, the installer publishes per-node coverage + (`openshell.ai/cni-sandbox-namespaces` annotation) and every gateway runs a + `wait-coverage` init container that blocks serving until every enforcement-ready + node acknowledges the gateway's namespace. Enable the installer in one release + per cluster. Readiness is bound to a `configVersion` (over the aggregated + allowlist and config) so a stale entry is repaired before the node is re-marked + ready, and the installer treats any `openshell-cni` chained entry as its own to + upgrade in place (the conflist patch preserves all other plugins). The + singleton's `pods get` and `configmaps list` RBAC are cluster-scoped so it can + discover marker ConfigMaps and read sandbox pods in any allowlisted namespace. - **Dependency on Multus aux chain.** `multus-chain` requires the Multus thick daemon to have `auxiliaryCNIChainName` set. It is default on OpenShift 4.x but not guaranteed everywhere. Mitigation: document the requirement; the chart does From 5ce726ae8b01fb11c4d3741c6c32b1c6d0b13d18 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Wed, 5 Aug 2026 21:55:13 -0400 Subject: [PATCH 20/20] fix(cni): harden cni-sidecar egress enforcement Three hardening changes to the cluster-singleton OpenShell CNI installer and its enforcement allowlist: - Drain-gated allowlist: the reconcile now unions registration-marker namespaces with namespaces that still contain an OpenShell-managed sandbox pod (openshell.ai/managed-by=openshell). Removing a release's marker no longer immediately drops enforcement for a namespace whose sandboxes are still running (which would fail-open their recreated pods); the namespace is pruned only once drained. Adds `pods list` to the CNI ClusterRole. - Read-only CNI CHECK: CHECK now verifies the bypass-prevention rules are present without reinstalling them, so a check never leaves a running pod momentarily unenforced. - Atomic nft install: ensure-exists, delete, and recreate run in a single `nft -f` transaction, so a failed apply never leaves the pod with the table deleted-but-not-recreated (unenforced). - Bounded kube-client timeouts on the CNI ADD path and the installer reconcile/coverage loops, so an API stall cannot wedge pod creation or stall reconciliation. Signed-off-by: Russell Bryant --- architecture/compute-runtimes.md | 7 +- crates/openshell-cni/src/lib.rs | 148 ++++++++++++++++-- deploy/helm/openshell/templates/cni-rbac.yaml | 3 + .../helm/openshell/tests/cni_rbac_test.yaml | 1 + docs/kubernetes/openshift.mdx | 5 +- docs/kubernetes/topology.mdx | 14 +- rfc/cni-sidecar-topology-DRAFT.md | 16 +- 7 files changed, 173 insertions(+), 21 deletions(-) diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index d757431fbc..2cf3b6f6bd 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -311,7 +311,12 @@ sandbox namespace, and the installer's reconcile aggregates every marker's namespace (via `list-sandbox-namespaces`) — unioned with the optional static `cni.sandboxNamespaces` — into the plugin config. Because the marker is a normal Helm resource, uninstalling a release or changing its `sandboxNamespace` removes -the registration (no orphaned allowlist entries). An additional `cni.external` +the marker. Deregistration is **drain-gated and monotonic**: the reconcile also +unions in every namespace that still contains an OpenShell-managed sandbox pod +(`openshell.ai/managed-by=openshell`), so removing a marker does not drop +enforcement while sandboxes are still running (which would fail-open their +recreated pods) — the namespace is pruned only once it is drained (no marker and +no sandbox pods). An additional `cni.external` release is discovered within one reconcile, with no manual allowlist edit. To eliminate the discovery-window race, the installer publishes on each node the CSV of namespaces it currently enforces (`openshell.ai/cni-sandbox-namespaces` diff --git a/crates/openshell-cni/src/lib.rs b/crates/openshell-cni/src/lib.rs index 1a5194b947..2dc371ae97 100644 --- a/crates/openshell-cni/src/lib.rs +++ b/crates/openshell-cni/src/lib.rs @@ -46,6 +46,10 @@ const CNI_REGISTRATION_LABEL: &str = "openshell.ai/cni-registration"; /// currently enforces on that node, so the gateway can wait for cluster-wide /// acknowledgement of its namespace before it serves sandboxes. const NODE_COVERAGE_ANNOTATION: &str = "openshell.ai/cni-sandbox-namespaces"; +/// Label selector identifying OpenShell-managed sandbox pods (set by the gateway +/// driver). Used to keep a namespace enforced while its sandboxes are still +/// running even after its registration marker is removed (drain-gated prune). +const SANDBOX_POD_LABEL_SELECTOR: &str = "openshell.ai/managed-by=openshell"; #[allow(dead_code)] const OPENSHELL_TABLE: &str = "openshell_sidecar_bypass"; #[allow(dead_code)] @@ -188,6 +192,10 @@ trait PodReader { trait RuleInstaller { fn install(&self, netns: &Path, proxy_uid: u32) -> Result; + /// Read-only validation that the bypass-prevention rules are present. Must NOT + /// modify live rules (CNI CHECK runs on a running pod; a destructive reinstall + /// that fails would leave it unenforced). + fn verify(&self, netns: &Path) -> Result<()>; fn cleanup(&self, netns: &Path) -> Result<()>; } @@ -254,13 +262,23 @@ pub fn node_ready(args: &[String]) -> Result<()> { /// Entry point for `openshell-cni list-sandbox-namespaces`. /// -/// Run by the installer reconcile to aggregate every namespace that contains a -/// Helm-owned registration marker `ConfigMap` (label -/// `openshell.ai/cni-registration=true`) into the plugin allowlist. Prints one -/// namespace name per line. +/// Run by the installer reconcile to compute the enforcement allowlist. Prints, +/// one per line, the union of: +/// * namespaces containing a Helm-owned registration marker `ConfigMap` +/// (`openshell.ai/cni-registration=true`) — active registrations; and +/// * namespaces that still contain an OpenShell-managed sandbox pod +/// (`openshell.ai/managed-by=openshell`). +/// +/// The second set makes the allowlist **monotonic while in use**: removing a +/// release's marker does not immediately drop enforcement for a namespace whose +/// sandboxes are still running (which would fail-open their recreated pods). The +/// namespace is pruned only once it is drained — no marker and no sandbox pods. pub fn list_sandbox_namespaces() -> Result<()> { let client = KubeApiClient::from_in_cluster()?; - let namespaces = client.list_registration_namespaces(CNI_REGISTRATION_LABEL)?; + let mut namespaces = client.list_registration_namespaces(CNI_REGISTRATION_LABEL)?; + namespaces.extend(client.list_sandbox_pod_namespaces(SANDBOX_POD_LABEL_SELECTOR)?); + namespaces.sort(); + namespaces.dedup(); for ns in namespaces { println!("{ns}"); } @@ -467,7 +485,12 @@ fn handle_command( let netns = env.netns.as_deref().ok_or_else(|| { miette::miette!("CNI_NETNS is required for OpenShell CNI CHECK") })?; - let report = installer.install(netns, workload.proxy_uid)?; + // CHECK is read-only: verify the rules are present without touching + // them, so a check never leaves a running pod momentarily unenforced. + installer.verify(netns)?; + let report = InstallReport { + backend: "verified", + }; log_cni_info( &config, env, @@ -594,6 +617,16 @@ struct KubeApiClient { client: reqwest::blocking::Client, } +/// Blocking HTTP client builder with bounded connect and request deadlines. The +/// CNI plugin runs synchronously on the CNI ADD path and in the installer's +/// reconcile/coverage loops; without deadlines an API stall would wedge pod +/// creation, invalidate the coverage wait's time bound, and stall reconciliation. +fn kube_client_builder() -> reqwest::blocking::ClientBuilder { + reqwest::blocking::Client::builder() + .connect_timeout(std::time::Duration::from_secs(5)) + .timeout(std::time::Duration::from_secs(15)) +} + impl KubeApiClient { fn from_kubeconfig(path: &Path) -> Result { let contents = std::fs::read_to_string(path) @@ -630,7 +663,7 @@ impl KubeApiClient { )); } }; - let mut builder = reqwest::blocking::Client::builder(); + let mut builder = kube_client_builder(); if let Some(ca) = cluster_certificate_authority(path, &cluster.cluster)? { builder = builder.add_root_certificate(ca); } @@ -663,7 +696,7 @@ impl KubeApiClient { .into_diagnostic() .wrap_err("failed to read in-cluster CA certificate")?; let ca = reqwest::Certificate::from_pem(&ca_pem).into_diagnostic()?; - let client = reqwest::blocking::Client::builder() + let client = kube_client_builder() .add_root_certificate(ca) .build() .into_diagnostic()?; @@ -762,6 +795,39 @@ impl KubeApiClient { Ok(namespaces) } + /// Returns the distinct namespaces of all pods matching `selector` + /// cluster-wide (used to keep enforcement for namespaces with live sandboxes). + fn list_sandbox_pod_namespaces(&self, selector: &str) -> Result> { + let url = format!("{}/api/v1/pods", self.server); + let response = self + .client + .get(url) + .query(&[("labelSelector", selector)]) + .bearer_auth(&self.token) + .send() + .into_diagnostic() + .wrap_err("failed to list sandbox pods")?; + if !response.status().is_success() { + return Err(miette::miette!( + "Kubernetes API returned {} while listing sandbox pods", + response.status() + )); + } + let list = response.json::().into_diagnostic()?; + let mut namespaces: Vec = list["items"] + .as_array() + .map(|items| { + items + .iter() + .filter_map(|item| item["metadata"]["namespace"].as_str().map(str::to_string)) + .collect() + }) + .unwrap_or_default(); + namespaces.sort(); + namespaces.dedup(); + Ok(namespaces) + } + /// Returns true when at least one node carries `ready_label` and every such /// node's `coverage_annotation` (a comma-separated list) contains `namespace`. fn namespace_covered_on_all_ready_nodes( @@ -965,6 +1031,10 @@ impl RuleInstaller for Runtime { install_rules(netns, proxy_uid) } + fn verify(&self, netns: &Path) -> Result<()> { + verify_rules(netns) + } + fn cleanup(&self, netns: &Path) -> Result<()> { cleanup_rules(netns) } @@ -1044,11 +1114,50 @@ fn install_rules(netns: &Path, proxy_uid: u32) -> Result { #[cfg(target_os = "linux")] fn install_nft_rules(netns: &Path, proxy_uid: u32, nft: &str) -> Result<()> { - let _ = run_nft_args_in_netns(netns, &nft, &["delete", "table", "inet", OPENSHELL_TABLE]); - let ruleset = generate_sidecar_bypass_ruleset(proxy_uid, Some("openshell:cni-sidecar:")); + // Atomic replace: ensure-exists, delete, recreate in a SINGLE `nft -f` + // transaction. nft applies the whole file atomically, so a failure never + // leaves the pod with the table deleted-but-not-recreated (unenforced). The + // leading `table {}` makes the subsequent `delete table` safe on first apply. + let body = generate_sidecar_bypass_ruleset(proxy_uid, Some("openshell:cni-sidecar:")); + let ruleset = + format!("table inet {OPENSHELL_TABLE} {{}}\ndelete table inet {OPENSHELL_TABLE}\n{body}"); run_nft_ruleset_in_netns(netns, &nft, &ruleset) } +/// Read-only check that the bypass-prevention rules are present in the netns. +/// Never modifies live rules. Prefers nft (the table), falls back to the iptables +/// chain; errors when neither is present so CNI CHECK surfaces the gap. +#[cfg(target_os = "linux")] +fn verify_rules(netns: &Path) -> Result<()> { + if let Some(nft) = find_nft() { + if run_nft_args_in_netns(netns, &nft, &["list", "table", "inet", OPENSHELL_TABLE]).is_ok() { + return Ok(()); + } + } + if let Some(iptables) = find_iptables() { + if run_command_in_netns( + netns, + &iptables.ipv4, + &["-w", "-t", "filter", "-n", "-L", OPENSHELL_IPTABLES_CHAIN], + ) + .is_ok() + { + return Ok(()); + } + } + Err(miette::miette!( + "OpenShell CNI bypass-prevention rules are not present in the pod network namespace" + )) +} + +#[cfg(not(target_os = "linux"))] +fn verify_rules(netns: &Path) -> Result<()> { + let _ = netns; + Err(miette::miette!( + "OpenShell CNI rule verification is supported only on Linux nodes" + )) +} + #[cfg(not(target_os = "linux"))] fn install_rules(netns: &Path, proxy_uid: u32) -> Result { let _ = (netns, proxy_uid); @@ -1387,6 +1496,7 @@ mod tests { #[derive(Default)] struct TestInstaller { installed: std::sync::Mutex>, + verified: std::sync::Mutex, cleaned: std::sync::Mutex, } @@ -1396,6 +1506,11 @@ mod tests { Ok(InstallReport { backend: "test" }) } + fn verify(&self, _netns: &Path) -> Result<()> { + *self.verified.lock().unwrap() += 1; + Ok(()) + } + fn cleanup(&self, _netns: &Path) -> Result<()> { *self.cleaned.lock().unwrap() += 1; Ok(()) @@ -1502,6 +1617,19 @@ mod tests { assert_eq!(*installer.installed.lock().unwrap(), vec![1337]); } + #[test] + fn check_verifies_without_reinstalling() { + let pods = TestPods { + annotations: openshell_annotations(), + }; + let installer = TestInstaller::default(); + handle_command(&cni_input(), &env("CHECK"), &pods, &installer).unwrap(); + // CHECK must be read-only: verify, never install (a destructive reinstall + // that failed would leave the running pod unenforced). + assert_eq!(*installer.verified.lock().unwrap(), 1); + assert!(installer.installed.lock().unwrap().is_empty()); + } + #[test] fn add_passes_through_non_openshell_pod() { let pods = TestPods { diff --git a/deploy/helm/openshell/templates/cni-rbac.yaml b/deploy/helm/openshell/templates/cni-rbac.yaml index 77abd3f35a..657cd0f270 100644 --- a/deploy/helm/openshell/templates/cni-rbac.yaml +++ b/deploy/helm/openshell/templates/cni-rbac.yaml @@ -31,6 +31,9 @@ rules: - pods verbs: - get + # list sandbox pods (openshell.ai/managed-by=openshell) to keep a namespace + # enforced while its sandboxes are still running (drain-gated prune). + - list # list registration marker ConfigMaps (openshell.ai/cni-registration=true) to # aggregate their namespaces into the plugin allowlist (automatic multi-release # discovery with Helm-owned lifecycle). diff --git a/deploy/helm/openshell/tests/cni_rbac_test.yaml b/deploy/helm/openshell/tests/cni_rbac_test.yaml index 9f16877501..6f2f14a3fe 100644 --- a/deploy/helm/openshell/tests/cni_rbac_test.yaml +++ b/deploy/helm/openshell/tests/cni_rbac_test.yaml @@ -45,6 +45,7 @@ tests: - pods verbs: - get + - list # configmaps list: aggregate registration-marker namespaces into the allowlist. - documentIndex: 1 contains: diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index a59773b223..6cefc6d7b1 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -148,7 +148,10 @@ Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true`) in its sandbo namespace, and the singleton discovers every marker's namespace and adds it to the enforcement allowlist — no manual edit of the singleton owner is required. Because the marker is a normal Helm resource, uninstalling the release (or changing its -`sandboxNamespace`) removes the registration automatically. The gateway does not +`sandboxNamespace`) removes it — but deregistration is drain-gated: the singleton +keeps a namespace enforced while it still has running sandbox pods +(`openshell.ai/managed-by=openshell`) and only prunes it once drained, so a +surviving sandbox is never left unenforced. The gateway does not serve until enforcement is acknowledged: a `wait-cni-coverage` init container blocks the gateway pod until every enforcement-ready node reports the namespace, so an additional release never creates sandboxes during the discovery window. diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index 0786a2e05c..3f02d86457 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -243,12 +243,14 @@ release ships a Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true in its sandbox namespace and the singleton aggregates every marker's namespace each reconcile — so enable the installer (`cni.enabled=true`) in exactly one release per cluster, and additional releases (`cni.enabled=false`, -`cni.external=true`) are discovered automatically, with registration removed by -Helm on uninstall. Each gateway waits (via a `wait-cni-coverage` init container) -until every enforcement-ready node acknowledges its namespace before serving, so -there is no discovery-window bypass. `cni.sandboxNamespaces` is an optional static -addition. The installer treats any `openshell-cni` chained entry as its own and -upgrades it in place. +`cni.external=true`) are discovered automatically. Removing a marker (uninstall / +namespace change) deregisters drain-gated: the singleton keeps a namespace +enforced while it still has running sandbox pods and prunes it only once drained. +Each gateway waits (via a `wait-cni-coverage` init container) until every +enforcement-ready node acknowledges its namespace before serving, so there is no +discovery-window bypass. `cni.sandboxNamespaces` is an optional static addition. +The installer treats any `openshell-cni` chained entry as its own and upgrades it +in place. ## Credential Exposure diff --git a/rfc/cni-sidecar-topology-DRAFT.md b/rfc/cni-sidecar-topology-DRAFT.md index e05f132ae2..3aa6eb5444 100644 --- a/rfc/cni-sidecar-topology-DRAFT.md +++ b/rfc/cni-sidecar-topology-DRAFT.md @@ -425,9 +425,13 @@ behavior is behind a default-off value. Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true`) in its sandbox namespace, and the installer's reconcile aggregates every marker's namespace (unioned with the optional static `cni.sandboxNamespaces`) into the config. As a - Helm resource the marker is removed on uninstall / sandboxNamespace change, so - registrations have proper lifecycle. An additional `cni.external` release is - discovered within one reconcile with no manual allowlist edit. To close the + Helm resource the marker is removed on uninstall / sandboxNamespace change. + Deregistration is drain-gated and monotonic: the reconcile also unions in every + namespace that still has an OpenShell-managed sandbox pod + (`openshell.ai/managed-by=openshell`), so a removed marker never drops + enforcement while sandboxes run; the namespace is pruned only once drained. An + additional `cni.external` release is discovered within one reconcile with no + manual allowlist edit. To close the discovery-window race, the installer publishes per-node coverage (`openshell.ai/cni-sandbox-namespaces` annotation) and every gateway runs a `wait-coverage` init container that blocks serving until every enforcement-ready @@ -456,6 +460,12 @@ behavior is behind a default-off value. - **SCC drift / cluster policy.** A cluster that further restricts SCC or blocks custom SCCs could reject the sandbox pod. Mitigation: the SCC is minimal and documented; the lower-privilege endpoint/L7 mode admits under `restricted-v2`. +- **Non-destructive rule maintenance.** CNI `CHECK` is read-only (verifies the + rules without reinstalling), and `ADD` applies the nft ruleset in a single + atomic transaction (ensure→delete→recreate), so a failed apply/check never + leaves a running pod with the table deleted-but-not-recreated. All Kubernetes + API calls from the plugin/installer use bounded connect/request timeouts so an + API stall cannot wedge CNI `ADD`, the coverage wait, or reconciliation. ## Alternatives