feat(kubernetes): add cni-sidecar supervisor topology

.agents/skills/debug-openshell-cluster/SKILL.md

@@ -395,12 +395,83 @@ The shared state directory should preserve `sandbox_gid` inheritance
 `@openshell-sidecar-ssh`; the network sidecar verifies its peer PID before
 bridging gateway relay requests. No `ssh.sock` file should appear in the shared
 state directory.
+
+If `topology = "cni-sidecar"` is rendered, the gateway should render
+the same process container and long-running network sidecar as sidecar mode, but
+there should be no `openshell-network-init` init container in sandbox pods.
+Instead, the chart must install the privileged `openshell-cni` DaemonSet and the
+sandbox pod should carry `openshell.ai/cni=enabled`,
+`openshell.ai/network-enforcement-mode=cni-sidecar`, and
+`openshell.ai/proxy-uid=<uid>` annotations. The CNI DaemonSet copies
+`/openshell-cni` into the host CNI binary directory and patches an existing CNI
+`.conflist`; if sandbox pods bypass network enforcement or fail during pod
+network setup, inspect the DaemonSet logs, the host CNI config, and whether the
+cluster actually invokes chained CNI plugins for the sandbox runtime class.
+
+The CNI installer is a **cluster singleton** with a fixed identity
+(`openshell-cni`). It is not necessarily in the release you are debugging — the
+owner is the release with `cni.enabled=true`; other gateway releases set
+`cni.external=true` and reuse it. Locate the owner and its resources cluster-wide:
+
+```bash
+kubectl get daemonset -A -l app.kubernetes.io/name=openshell-cni
+kubectl get clusterrole,clusterrolebinding openshell-cni
+```
+
+A gateway release using `cni-sidecar` with neither `cni.enabled` nor
+`cni.external` fails to render (template error), so if the gateway installed but
+sandboxes are unenforced, confirm the singleton exists and is Ready.
+
+**Namespace allowlist (silent unenforced sandboxes).** The plugin only inspects
+pods whose namespace is in its `sandboxNamespaces` allowlist. That allowlist is
+built automatically from Helm-owned marker ConfigMaps
+(`openshell.ai/cni-registration=true`) — one per cni-sidecar release, in its
+sandbox namespace — unioned with the static `cni.sandboxNamespaces`. If a sandbox
+reaches 2/2 but its egress is NOT blocked, confirm its namespace is registered and
+in the installed allowlist:
+
+```bash
+# Registration markers (their namespaces are what the singleton enforces):
+kubectl get configmaps -A -l openshell.ai/cni-registration=true
+# Installed plugin config's aggregated sandboxNamespaces:
+kubectl -n <owner-ns> exec ds/openshell-cni -c install-cni -- \
+  sh -c 'cat /host/run/multus/cni/net.d/vendor-cni-chain/openshell-cni.conf 2>/dev/null \
+         || cat /host/etc/cni/net.d/*.conflist'
+```
+
+**Gateway stuck in Init (`wait-cni-coverage`).** A cni-sidecar gateway pod has a
+`wait-cni-coverage` init container that blocks until every `cni-ready` node's
+`openshell.ai/cni-sandbox-namespaces` annotation includes the gateway's namespace.
+If the gateway is stuck initializing, the singleton has not yet acknowledged the
+namespace on all nodes — check the per-node coverage:
+`kubectl get nodes -o custom-columns=NAME:.metadata.name,COVERAGE:.metadata.annotations.openshell\.ai/cni-sandbox-namespaces`
+and confirm a marker ConfigMap exists for that namespace.
+
+**Node reboot / boot taint.** In `multus-chain` mode the chain file lives under
+`/run` (tmpfs) and is wiped on reboot while the `cni-ready` label persists.
+Clusters that apply the optional boot-time taint (`openshell.ai/cni-not-ready`,
+see `deploy/helm/openshell/examples/`) will show it on a node until the installer
+removes it; a node stuck with the taint means the installer has not reached ready
+there. Check with `kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints`.
+
+A per-node scheduling gate can also keep cni-sidecar sandbox pods `Pending`. The
+CNI DaemonSet labels each node `openshell.ai/cni-ready=true` after installing the
+plugin, and the gateway sets a required `nodeAffinity` on that label. If a
+sandbox pod stays `Pending` with an "unmatched nodeAffinity" event, check that
+the CNI DaemonSet is Ready on schedulable nodes and that the label is present
+(`kubectl get nodes -L openshell.ai/cni-ready`). A missing label means the
+installer has not completed its first patch or a reconcile tick cleared it after
+a plugin-restore failure.
+
 Inspect all three when sandbox registration or egress enforcement fails:
 
 ```bash
 kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^\[openshell\.drivers\.kubernetes\]|^topology\s*='
 kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.initContainers[*]}{.name}{" "}{.command}{"\n"}{end}'
 kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}'
+kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{.metadata.annotations}'
+kubectl -n openshell get daemonset,pod -l app.kubernetes.io/component=cni
+kubectl -n openshell logs daemonset/openshell-cni -c install-cni --tail=200
 kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-network-init --tail=200
 kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-network --tail=200
 kubectl -n <sandbox-namespace> logs <sandbox-pod> -c agent --tail=200
@@ -446,6 +517,7 @@ openshell logs <sandbox-name>
 | HTTP request returns `middleware_failed` or `middleware_denied` | Selected stage failed or explicitly denied the admitted request | Sandbox OCSF logs; policy-local middleware config; service availability; `on_error` |
 | Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or configured with a reserved/mismatched name | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs |
 | Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials |
+| CNI-sidecar sandbox pods fail network setup | OpenShell CNI DaemonSet did not patch the node CNI conflist, cannot read pods, or the runtime class does not invoke the chained plugin | `kubectl -n openshell logs daemonset/openshell-cni -c install-cni`, chart `cni.*` values, host CNI config |
 | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled |
 | HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://<host>` |
 | HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https://<host> --oidc-issuer <url>`, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster |

.agents/skills/helm-dev-environment/SKILL.md

@@ -70,13 +70,20 @@ mise run helm:skaffold:run:sidecar
 mise run helm:skaffold:run:sidecar-mtls
 ```
 
-Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
+**Supervisor CNI-sidecar topology** (build once and leave running):
+```bash
+mise run helm:skaffold:run:cni-sidecar
+```
+
+These commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
 chart. The sidecar profile renders an `openshell-network-init` init container for
 nftables setup and an `openshell-supervisor-network` runtime sidecar for proxying.
 Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and
 `DAC_READ_SEARCH`; relaxed mode can run it as the configured proxy UID. The
 sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores
-`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install
+`server.disableTls=false` inline for Skaffold. The cni-sidecar profile enables
+the privileged OpenShell CNI DaemonSet and uses the sidecar runtime model
+without the pod-local network init container. The `pkiInitJob` hook (a pre-install
 Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first
 install. Envoy Gateway opt-in; see the Optional Add-ons section below.
 
@@ -87,6 +94,30 @@ The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or
 create the Secret named `openshell-ha-pg` with a `uri` key, then run
 `mise run helm:skaffold:run` or `mise run helm:skaffold:dev`.
 
+### Kubernetes e2e profiles
+
+Run the default Kubernetes e2e environment:
+
+```bash
+mise run e2e:kubernetes
+```
+
+Run the sidecar topology e2e environment:
+
+```bash
+mise run e2e:kubernetes:sidecar
+```
+
+Run the CNI-sidecar topology e2e environment:
+
+```bash
+mise run e2e:kubernetes:cni-sidecar
+```
+
+The cni-sidecar e2e task applies `ci/values-cni-sidecar.yaml` through
+`OPENSHELL_E2E_KUBE_EXTRA_VALUES` and requires OpenShell CNI installer
+permissions on the target cluster's nodes.
+
 ### TLS behaviour
 
 `ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
@@ -149,6 +180,12 @@ For a sidecar-profile deployment:
 mise run helm:skaffold:delete:sidecar
 ```
 
+For a cni-sidecar-profile deployment:
+
+```bash
+mise run helm:skaffold:delete:cni-sidecar
+```
+
 ### Delete the cluster entirely
 
 ```bash
@@ -274,6 +311,7 @@ for dependencies still declared in `Chart.yaml`.
 | `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
 | `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
 | `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev |
+| `deploy/helm/openshell/ci/values-cni-sidecar.yaml` | Supervisor CNI-sidecar topology overlay for Kubernetes e2e/dev; enables the OpenShell CNI DaemonSet |
 | `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
 | `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
 | `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |

.github/workflows/docker-build.yml

@@ -246,6 +246,9 @@ jobs:
           fi
           mkdir -p "$stage"
           install -m 0755 "$found" "$stage/$binary"
+          if [[ "${{ inputs.component }}" == "supervisor" ]]; then
+            PREBUILT_ARCH="${{ matrix.arch }}" tasks/scripts/stage-prebuilt-binaries.sh cni
+          fi
           ls -lh "$stage/"
 
       - name: Build ${{ inputs.component }} image

AGENTS.md

@@ -33,6 +33,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
 | `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary |
 | `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing |
 | `crates/openshell-policy/` | Policy engine | Filesystem, network, process, and inference constraints |
+| `crates/openshell-cni/` | Chained CNI plugin | Node-level egress bypass-prevention for the cni-sidecar topology; also sets the per-node readiness label |
 | `crates/openshell-router/` | Privacy router | Privacy-aware LLM routing |
 | `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage |
 | `crates/openshell-gateway-interceptors/` | Gateway interceptors | Intercepts and transforms configured gRPC requests at the gateway routing boundary |

architecture/compute-runtimes.md

@@ -241,6 +241,108 @@ already unprivileged. Sidecar pods use a shared process namespace so the
 network sidecar can resolve workload process and binary identity through
 `/proc/<entrypoint-pid>`.
 
+The cni-sidecar topology keeps the sidecar runtime model and its shared-state
+boundary, but removes the privileged `openshell-network-init` init container and
+its `NET_ADMIN`. Instead, the privileged OpenShell CNI DaemonSet installs the
+pod-network bypass-prevention rules during CNI `ADD` using nftables or iptables.
+The driver annotates sandbox pods so the chained CNI plugin can read the proxy
+UID and enforcement mode. The network sidecar keeps the same privilege profile
+as the other sidecar topologies: in the default binary-aware mode it runs as UID
+0 with `SYS_PTRACE` and `DAC_READ_SEARCH` (but no `NET_ADMIN`) to resolve
+cross-UID `/proc`, and only stays non-root with no added capabilities when
+`process_binary_aware_network_policy` is disabled. The agent container stays
+non-root with no added Linux capabilities in either mode. Because the node CNI —
+not an in-pod container — programs the firewall, no container in the sandbox pod
+holds `NET_ADMIN`.
+
+The CNI installer supports two modes. `conflist` (default) appends the
+`openshell-cni` plugin to an existing CNI `.conflist` (k3s / vanilla). On
+OpenShift (Multus / OVN-Kubernetes) there is no appendable `.conflist`, so
+`multus-chain` writes a standalone plugin `.conf` into the Multus
+`vendor-cni-chain` auxiliary-chain directory and stores plugin credentials in a
+persistent `stateDir`. Neither mode modifies a CNO-managed file.
+
+The installer patches the chained plugin at startup and then re-verifies it on a
+reconcile tick, re-patching when the plugin is missing. This keeps enforcement in
+place across CNI config rewrites (for example a CNO reconcile) and DaemonSet
+restarts, bounding any such gap to one reconcile interval. The chained plugin
+lives in the host CNI config and survives installer pod restarts, so an ordinary
+DaemonSet restart or rolling update never strips enforcement: the `preStop` hook
+removes it only when the owning DaemonSet is actually being deleted (helm
+uninstall), and fences the node before doing so.
+
+A per-node scheduling gate closes the cold-start race. Once the chained plugin is
+installed, the installer labels its node `openshell.ai/cni-ready=true`. Each
+reconcile tick fences before it repairs: the instant enforcement is not
+verifiably in place it clears the label, attempts repair, and only re-marks the
+node ready once the plugin is healthy again. The gateway sets a required
+`nodeAffinity` on that label for every cni-sidecar sandbox pod, so a pod cannot
+schedule onto a node before that node's egress enforcement is active, and a node
+whose enforcement later breaks stops accepting new sandbox pods. The label is set
+through a minimal cluster-scoped grant (`nodes` `get`/`patch`, plus `get` on the
+installer's own DaemonSet) bound to the dedicated CNI ServiceAccount.
+
+The persistent label cannot by itself cover a node reboot that wipes tmpfs-backed
+enforcement (`multus-chain` stores the chain file under `/run`). A boot-time
+`NoSchedule` taint (`openshell.ai/cni-not-ready`) **narrows** that window but does
+not fully close it: operator node config applies it at boot (only node config
+runs before the scheduler; a MachineConfig example ships under
+`deploy/helm/openshell/examples/`), the CNI DaemonSet tolerates it, and the
+installer removes it once enforcement is ready (never re-adding it, so a transient
+unready does not over-repel). A small residual race remains because the taint is
+applied after the kubelet starts, so the kubelet can briefly mark the node
+schedulable before the taint lands; `--register-with-taints` covers the initial
+join but not reboot re-registration. `conflist` mode keeps the plugin on
+persistent disk and is unaffected by reboot, and is the way to avoid the window
+entirely. One residual
+limitation: an ungraceful DaemonSet pod deletion (no `preStop`) leaves a stale
+`cni-ready=true` until the pod is rescheduled and the next reconcile tick
+re-evaluates it.
+
+The CNI installer is a **cluster singleton**. Its chained plugin enforces pods
+that carry the OpenShell annotations (`openshell.ai/cni=enabled` plus the
+proxy-UID and enforcement-mode annotations, set only by a gateway on its own
+sandbox pods) **and** whose namespace is in the plugin's `sandboxNamespaces`
+allowlist. Pods in other namespaces are passed through without a Kubernetes API
+lookup, so the allowlist bounds the blast radius of the per-pod annotation read.
+The allowlist is built **automatically**: each `cni-sidecar` gateway release
+ships a Helm-owned marker ConfigMap (`openshell.ai/cni-registration=true`) in its
+sandbox namespace, and the installer's reconcile aggregates every marker's
+namespace (via `list-sandbox-namespaces`) — unioned with the optional static
+`cni.sandboxNamespaces` — into the plugin config. Because the marker is a normal
+Helm resource, uninstalling a release or changing its `sandboxNamespace` removes
+the marker. Deregistration is **drain-gated and monotonic**: the reconcile also
+unions in every namespace that still contains an OpenShell-managed sandbox pod
+(`openshell.ai/managed-by=openshell`), so removing a marker does not drop
+enforcement while sandboxes are still running (which would fail-open their
+recreated pods) — the namespace is pruned only once it is drained (no marker and
+no sandbox pods). An additional `cni.external`
+release is discovered within one reconcile, with no manual allowlist edit. To
+eliminate the discovery-window race, the installer publishes on each node the CSV
+of namespaces it currently enforces (`openshell.ai/cni-sandbox-namespaces`
+annotation, via `set-node-coverage`), and every gateway runs a `wait-coverage`
+init container that blocks until each enforcement-ready node acknowledges the
+gateway's namespace — so a newly-registered release does not serve sandboxes until
+enforcement is confirmed cluster-wide. The installer resources use a fixed
+release-independent name, the plugin config carries a fixed `openshell` owner, and
+a `configVersion` (over the aggregated allowlist and config) binds readiness so a
+stale-version entry is repaired before the node is re-marked ready. Install the
+singleton (`cni.enabled=true`) in one release per cluster; additional releases set
+`cni.enabled=false` + `cni.external=true`. The installer treats any `openshell-cni`
+chained entry as its own and upgrades it in place (the conflist patch preserves
+all other plugins). The `pods get` and `configmaps list` grants are cluster-scoped
+so the one installer can discover marker ConfigMaps and read sandbox pods in any
+allowlisted namespace.
+
+On OpenShift, binary-aware network policy also requires a purpose-built
+SecurityContextConstraints for sandbox pods: the network sidecar runs as UID 0
+with `SYS_PTRACE` and `DAC_READ_SEARCH` to inspect cross-UID `/proc`, which
+`restricted-v2` forbids. `sandboxServiceAccount.openshift.binaryAwareSCC` creates
+a minimal SCC (the `restricted-v2` baseline plus only those two capabilities, UID
+0, and the `image` volume type) and binds it to the sandbox ServiceAccount.
+Disabling `processBinaryAwareNetworkPolicy` drops the capability requirement and
+lets the stock `restricted-v2` SCC apply.
+
 ## Images
 
 The gateway image and Helm chart are built from this repository. Sandbox images