K8s fleet onboarding — console UX for provider selection ('+ New fleet')

[brettchien]

Context

Follow-up to #97 (K8s driver — ADR #63 slice 3, closed 2026-08-27, all 6 sub-slices merged). Driver backend for k8s (K8sDriver, K8sFleetBinding/fleets-k8s.toml, k8s identity/context observation, k8s secret refs, bundle restore via pre_seed) is complete and on main. This issue tracks the next phase: console-side onboarding — today "+ New fleet" only supports AWS (Region/Credential profile/Principal); there is no UI path to onboard a k8s-driven fleet, only manual fleets-k8s.toml editing.

Design discussion: Discord thread 1537474496406233130, 2026-08-26.

Scope

Full deployment onboarding UX for k8s fleets in the desktop console, on par with the existing AWS "+ New fleet" wizard.

Design decisions (confirmed)

1. Provider picker — identity form gets a provider choice (AWS / k8s) before the rest of the fields; switching provider resets the form (field semantics don't map across providers).
2. k8s field group: Context, Namespace, Service account (optional). All three are <select> (reusing the existing fillOptions pattern from the compose-library select in deploy.ts), not free-text — with a manual-entry fallback when enumeration is empty/fails.
   • Namespace: new list_namespaces tool (Api<Namespace>::list()), + free-text fallback for a namespace that doesn't exist yet.
   • Service account: new list_service_accounts(context, namespace) tool (Api<ServiceAccount>::list(namespace)) — depends on context+namespace being chosen first. If listing fails (including RBAC 403), silently fall back to leaving it blank (k8s uses the namespace's default ServiceAccount) — no error UI needed for this one.
   • Service account value is stored in a new K8sFleetBinding.expected_principal: Option<String> field — deliberately named to match the AWS side's expected_principal, since observe_k8s_identity (studio-cp/lib.rs:281) + k8s_principal_kind (:259) already provide the same verify semantics as AWS's observe_identity/identity_matches — reuse, don't reinvent.
3. deploy_provision gets a provider parameter and branches internally to dispatch to K8sDriver — not a separate MCP tool.
4. fleets-k8s.toml write path: new k8s_fleet_config_write MCP tool wired to the already-existing save_k8s_bindings_text (studio-cp/lib.rs:677 — has unit test coverage, just never wired to a tool). Client-side: new fleetsK8sToml.ts mirroring fleetToml.ts's pure text-mutation helpers (appendMember/appendFleetBlock equivalents for the k8s file).
5. list_aws_profiles / list_k8s_contexts failure UX — three tiers, not a single generic error:
   • Config file missing / empty → actionable guidance (AWS: "run aws configure or aws sso login"; k8s: "install OrbStack/kind/minikube for local, or merge your cloud vendor's kubeconfig into ~/.kube/config"). This is actionable because oab-mcp is a local Tauri sidecar child process (src-tauri/src/mcp.rs:1-16) reading the operator's own machine.
   • Tool call itself errors (parse failure, permissions) → show the raw error, don't guess.
   • Both cases: field falls back to free-text input, doesn't block the wizard.
6. Admin/management agent access — no extra wiring needed. servers_for(management, conn_id) (src-tauri/src/remote.rs:345) already declares the oab reverse-MCP server for management-bound connections; new tools are automatically reachable once built. Non-management agent consoles stay without access (least-privilege, unchanged).

New MCP tools needed

• list_aws_profiles — read ~/.aws/config, return profile names (+ region if set)
• list_k8s_contexts — kube::config::Kubeconfig::read(), return context names + which is current-context
• list_namespaces(context) — Api<Namespace>::list()
• list_service_accounts(context, namespace) — Api<ServiceAccount>::list(namespace)
• k8s_fleet_config_write — wire to existing save_k8s_bindings_text
• deploy_provision — extend with a provider parameter, dispatch to K8sDriver for k8s

Schema changes

• K8sFleetBinding (studio-cp/lib.rs:562) — add expected_principal: Option<String>

Client changes

• console/src/render.ts / deploy.ts — provider selector step in the identity form
• console/src/fleetsK8sToml.ts (new) — mirrors fleetToml.ts

Explicit non-goals for this phase

• Bundle carrier — already solved (fix(bundle): actually restore a deployed agent's bundle at boot, via pre_seed (slice 3c) #103, via pre_seed), no work needed here.
• Extending non-management agent consoles' tool access — out of scope, would need a separate least-privilege decision.

[brettchien]

Progress (usercron-k8s-onboarding, first tick): opened #105 — list_aws_profiles + list_k8s_contexts MCP tools (the two dependency-free discovery tools; list_namespaces/list_service_accounts need context+namespace already picked, so they're a separate next sub-item).

Local build/test hit the known aws-sdk-ec2/s3 OOM constraint on this machine (see PR description) — couldn't get a green local test run, hand-verified the kube-client API surface against vendored crate source instead and am relying on CI as the actual gate. Flagging so a reviewer knows to wait for CI rather than assume local-verified.

Next tick: list_namespaces/list_service_accounts, or console-side wiring, depending on how #105's CI/review lands.

[brettchien]

Progress (usercron-k8s-onboarding, second tick): opened #106 (stacked on #105, please merge #105 first) — list_namespaces + list_service_accounts MCP tools. Also factored the from_kubeconfig+Client::try_from boilerplate into a shared k8s_client_for() helper and refactored observe_k8s_identity to use it (no behavior change).

#105 still open/unreviewed, CI was pending as of this tick's start (not red — just hasn't run to completion yet).

Remaining backend items: k8s_fleet_config_write (wire the already-existing save_k8s_bindings_text), deploy_provision provider param (dispatch to K8sDriver), K8sFleetBinding.expected_principal schema field. Console-side wiring (provider picker, <select> population) is separate again. Next tick: probably K8sFleetBinding.expected_principal + k8s_fleet_config_write together (same file, related), unless #105/#106 need review-comment fixes by then.

[brettchien]

Progress (usercron-k8s-onboarding, third tick): opened #107 (stacked on #106 → #105, merge in order) — K8sFleetBinding.expected_principal schema field + k8s_fleet_config_write MCP tool. save_k8s_bindings_text needed zero changes (already generic).

Status: #105 CI-green, #106/#107 not yet checked this tick's end (opened just now). None merged/reviewed yet.

Remaining: deploy_provision provider param (dispatch to K8sDriver — the one item that needs real design thought, not just following the AWS-side pattern), plus all console-side wiring (provider picker, fleetsK8sToml.ts, population against the 4 list_* tools). Backend/tool surface for #104 is now feature-complete pending those two. Next tick: probably deploy_provision provider param, since console wiring depends on having something for it to call.

[brettchien]

Stopping before writing code on the deploy_provision provider param — found something that changes the shape of the problem, want to flag it rather than build on a wrong premise (same call as the bundle-carrier discovery during #97).

Traced the actual call chain for deploy_provision → t_provision → studio_cp::provision_from_library → oabctl::studio_api::redeploy:

1. redeploy() requires a pre-existing stored manifest (load_manifest() → S3 GetObject at manifests/{namespace}/{name}.yaml) — if none exists it errors "no stored manifest ... create the agent before redeploying". It patches that existing manifest's image/bundle_from, then calls provision() → EcsDriver::apply().
2. EcsDriver::apply() wraps apply.rs::apply_manifests — which is where the S3 PUT of the manifest actually happens (apply.rs:~606, "1. Upload manifest to S3 (record of desired state)").
3. K8sDriver::apply() (3b) is a fully independent implementation — direct kube Api<Deployment>::patch (server-side apply), doesn't touch S3 at all, doesn't store a manifest anywhere. It implements the same ProvisionDriver trait, so swapping it in for EcsDriver in provision() is mechanically easy — but doing so would mean k8s agents never get a stored manifest, so redeploy()'s "patch the existing one" pattern has nothing to patch.

The deeper finding: deploy_provision/redeploy has never supported creating a brand-new agent from scratch, for any provider. It only patches an already-stored manifest. The only place that writes a manifest's first version is apply.rs::apply_manifests (ECS-only), which — per create.rs's own AWS-coupled interactive wizard (the CLI tool, not Studio) — is how agents get created today. I don't see any code path where Studio's console (deploy_provision) creates an agent's first manifest. If that's right, the "+ New fleet" wizard's [+ Add instance]/[+ New fleet] flow may only work today for agents someone already created via the CLI — which would be a pre-existing gap independent of k8s, not something k8s support needs to fix, but it does mean "add a k8s branch to deploy_provision" can't just mean "swap the driver" — there's no manifest-creation entry point in the MCP surface at all to swap a driver into.

Not proceeding on this sub-item without your read on it — is my trace right, or is there a manifest-creation path I'm missing (maybe deploy_apply is meant to be called first by the console before deploy_provision, and I haven't found where that's documented/wired)? Pivoting this tick to console-side work that doesn't depend on this (the provider picker UI skeleton, wiring the four list_* tools already merged/PR'd) since that's independently useful regardless of how this resolves.

[brettchien]

Progress (usercron-k8s-onboarding, fourth tick): no response yet on the deploy_provision architecture question from last tick, so pivoted to independent console-side work as planned. Opened #108 (stacked on #107→#106→#105) — provider picker (AWS/k8s) in the "+ New fleet" identity form, context/namespace fields wired live to #105/#106's list_k8s_contexts/list_namespaces tools.

k8s submission is intentionally blocked in this PR (clear "not available yet, tracked in #104" message) rather than wired through to a deploy_provision call that can't actually work yet per the open architecture question — see #108's description for the reasoning. AWS path is unchanged.

This one's fully locally verified (TS/vite, no OOM constraint) — typecheck clean, 102/102 tests pass, build succeeds. Unlike #105-107 this isn't relying on CI-as-the-only-gate.

Remaining: list_service_accounts console wiring (small), fleetsK8sToml.ts (client-side write helper, needed once k8s identity form actually submits), and the deploy_provision/K8sDriver dispatch design question (still open, blocking full end-to-end k8s onboarding). Next tick: check for a response on the architecture question first; if none, list_service_accounts wiring is the next independent piece.

[brettchien]

Progress (usercron-k8s-onboarding, fifth tick): still no response on the deploy_provision architecture question. Opened #109 (stacked on #108→#107→#106→#105) — list_service_accounts wired into the k8s field group (select, silent-fallback-on-failure per the design). Also locally verified (typecheck/tests/build all pass).

Remaining open items in #104: fleetsK8sToml.ts (client-side write helper — lower priority until identity-form-for-k8s can actually submit anything), and the deploy_provision/K8sDriver dispatch design question (still the one blocking item, unresolved). All 5 PRs open (#105-#109), none merged/reviewed yet, all CI-green or pending-not-failed.

Next tick: check for a response on the architecture question again; if still none, will likely start fleetsK8sToml.ts (it's independently useful even before deploy_provision dispatch is resolved — the write path can be built now, submission gated same as today) since that's the last clearly-scoped, unblocked item.

[brettchien]

Progress (usercron-k8s-onboarding, sixth tick — final for this job run): opened #110 (stacked on #109→#108→#107→#106→#105) — fleetsK8sToml.ts, the client-side write helper (mirrors fleetToml.ts, reuses appendMember/findFleetBlock rather than duplicating).

This closes out every independently-buildable sub-item in #104's scope: all 5 new MCP tools (list_aws_profiles, list_k8s_contexts, list_namespaces, list_service_accounts, k8s_fleet_config_write), K8sFleetBinding.expected_principal, and the full console UI (provider picker, context/namespace/service-account fields, write-path helper) are now open as PRs #105-110, stacked in dependency order, all CI-green or passing-so-far, none merged/reviewed yet.

The one item still blocked: deploy_provision's provider-param dispatch to K8sDriver. Traced this two ticks ago (comment above) — redeploy() requires an already-stored manifest to patch, and K8sDriver::apply doesn't persist one, so this isn't a simple driver swap. Posted the finding, no response yet. This is the one piece that needs your design input before it can move, not something further autonomous ticks can resolve — stopping the usercron job here rather than keep polling every 10 min for a response with nothing else to do in the meantime. Whenever you have a read on it, happy to pick this straight back up (either via a new usercron job or synchronously) — the write-path/UI/tools are all ready and waiting for it.

[brettchien]

Brett's decision (2026-08-27) on the deploy_provision architecture question: go with the "wire creation through deploy_apply" option (not "make K8sDriver mirror the missing behavior" and not "cram creation into redeploy"). Split into a dedicated issue since it's not k8s-specific — #111.

#104's deploy_provision provider-param item now depends on #111. Once #111 lands (AWS-side "+ New fleet" actually able to create a new agent, via deploy_apply's create-or-update path rather than redeploy's patch-only path), k8s's provider dispatch happens at that same entry point. Resuming usercron-k8s-onboarding to work #111 first.