compute: launch a second, interactive compute runtime

[antiguru]

Sixth of eight PRs splitting #37770. Stacks on #38390. Tracked by CPU-216.

With --interactive-compute-timely-config, clusterd runs two compute runtimes in one process: the first takes Maintenance, the second Interactive, and both share the one sharing registry so a reader on worker i of either finds the slot a publisher on worker i of the other filled. They must span an equal number of Timely peers, which the config preparation asserts, because the registry pairs workers by ordinal. Without the flag the process runs a single Solo runtime, unchanged from a deployment with no second runtime.

One controller endpoint still fronts the replica: with two runtimes a Multiplexer serves it, with one the maintenance client builder serves it directly.

Shared fate is the read-hold mechanism. Both runtimes' threads are covered by the process-global panic hook installed at the top of main, so a panic on either aborts the process, bounding an interactive import's read hold to the life of the replica without a lease. A subprocess test asserts the abort, which cannot be observed from inside the panicking process.

enable_compute_interactive_runtime is replica-scoped but resolved in environmentd, since the controller decides ServiceConfig::ports before the replica exists. replica_scoped_bool parses through the dyncfg rather than str::parse, because a stored bool formats as on/off.

Off in production and in tests. Flipping it changes how a replica is provisioned, so it is not a live toggle. The interactive runtime cannot serve index peeks until the next PR in the stack, which is safe only because the flag is off here. That is also why the flag is registered in UNINTERESTING_SYSTEM_PARAMETERS and pinned to FALSE in parallel-workload's FlipFlagsAction: check-test-flags requires a new flag to be registered in the PR that introduces it, but nothing may turn this one on yet. The last PR of the stack promotes it to a variable system parameter defaulting on.

[antiguru]

Missing newline above.

[antiguru]

Fixed. (Posted by Claude Code.)

[linear-code]

CPU-215

[antiguru]

Could this be generic over the dyncfg and type?

[antiguru]

Yes, and it is a strict improvement. Now replica_scoped<D: ConfigDefault>(&self, replica_id, config: &Config<D>) -> D::ConfigType.

The dyncfgs.entry(name) lookup and the ConfigVal::Bool match both disappear, because D::ConfigType::parse is directly callable. That also keeps the on/off behavior structurally rather than by comment: <bool as ConfigType>::parse handles those spellings itself, which is exactly the hazard the old doc paragraph was about.

Both call sites already passed NAME.name() and NAME.get(dyncfgs), so three arguments collapse to one and a name/default mismatch stops being expressible.

One detail worth flagging: the fallback reads config.get(dyncfgs), not config.default(). The former is the environment value including a production flag flip; the latter is the compile-time constant, and using it would silently ignore such a flip. There is a comment on that line saying so.

Posted by Claude Code.

[def-]

QA LLM Review

1. MEDIUM -- turning enable_compute_interactive_runtime on aborts the replica process on any DROP INDEX/DROP MATERIALIZED VIEW

src/clusterd/src/lib.rs:575

This PR makes Multiplexer live, and the multiplexer broadcasts every non-transient AllowCompaction to the interactive runtime, which at this commit hosts none of those collections. The empty-frontier form of that command is the collection drop, and it lands on .expect("dropped untracked collection"); under this PR's own shared-fate model that panic aborts the whole clusterd process, taking the maintenance runtime and the replica down with it. The PR body warns only that index peeks do not work yet, so the blast radius of flipping the flag is larger than documented.

Details

src/compute-client/src/multiplex.rs:240 forwards AllowCompaction { id, frontier } to the interactive client whenever the owner is maintenance and !id.is_transient(). On the interactive runtime handle_allow_compaction (src/compute/src/compute_state.rs:925) has no ownership check at this commit: an empty frontier goes straight to drop_collection, whose collections.remove(&id) returns None for a collection whose CreateDataflow was routed to maintenance only, and the .expect at src/compute/src/compute_state.rs:1003 fires. Interactive's collections map holds only the logging-index copies installed by initialize_logging; every user index and materialized view is maintenance-only. The controller sends the empty frontier through the ordinary read-capability path (src/compute-client/src/controller/instance.rs:1995), so plain DROP INDEX, DROP MATERIALIZED VIEW, or a DROP TABLE that cascades into an index is enough.

The multiplexer's comment immediately above the broadcast already names this hazard for maintenance's transient collections ("the drop in that sequence would ask it to drop what it does not have") and excludes them; the non-transient ones need the same exclusion at this point in the stack. The non-empty frontier case is harmless only by luck: TraceManager::allow_compaction silently ignores unknown ids.

A second manifestation, corroborating rather than separate: every Peek is routed to interactive, and handle_peek does self.compute_state.traces.get(id).unwrap() (src/compute/src/compute_state.rs:941), which is None for a maintenance-owned index. So a fast-path index peek is not merely unserved, it aborts the process too.

Reachability: enable_compute_interactive_runtime is a plain dyncfg in all_dyncfgs, and SystemVars::new auto-registers every dyncfg as a settable system parameter (src/sql/src/session/vars.rs:1334), so ALTER SYSTEM SET enable_compute_interactive_runtime = on (or a parameter-sync push) is all it takes; the next replica provisioning picks it up.

Suggested fix, if the flag is to stay settable at this point in the stack, is the ownership check that the follow-up commit in the stack adds anyway:

fn handle_allow_compaction(&mut self, id: GlobalId, frontier: Antichain<Timestamp>) {
    // The multiplexer broadcasts the peer's compaction so this runtime keeps the create and the
    // compactions after it on one ordered stream. Nothing here hosts those collections yet, and
    // `drop_collection` panics on one that was never installed.
    if self.compute_state.role() == ComputeRuntimeRole::Interactive
        && !self.compute_state.collections.contains_key(&id)
    {
        return;
    }
    ...
}

The alternative, if the intent is that the flag simply must not be turned on before the stack lands, is to make it unsettable rather than merely default-off, since nothing today stops an operator from flipping it.

[def-]

QA LLM Review

1. MEDIUM -- a published index never compacts: nothing advances the standing hold

src/compute/src/sharing.rs:408

With enable_compute_interactive_runtime on, the maintenance runtime publishes every exported index into the sharing registry, and the publication point's standing hold bounds that arrangement's logical compaction. ArrangementSharingRegistry::note_standing_hold is the only thing that advances that hold, and it has no production caller at this commit, so every index arrangement on the replica stays pinned at the frontier it was seeded with when its dataflow was rendered and never compacts, retaining full history until the replica is OOM-killed.

Details

The chain, all of it new with the flag (Solo::publishes() is false, Maintenance::publishes() is true):

• src/compute/src/render.rs:901 and :1015 publish every exported index; src/compute/src/logging/initialize.rs:407 publishes the logging indexes too, so even an idle replica is affected.
• src/compute/src/shared_trace/publish.rs:194 seeds the standing hold from the arrangement's own get_logical_compaction(), which at render time is the minimum: no AllowCompaction has arrived yet and TraceManager::set requests none. SharedReader::set_logical_compaction (src/timely-util/src/shared_trace.rs:606) joins rather than assigns, so the hold moves only when something calls it again.
• src/timely-util/src/shared_trace.rs:403 applies local_logical.meet(&remote_logical) to the inner trace. The controller's AllowCompaction does reach traces.allow_compaction on maintenance, and is then meet-ed straight back down to the seeded hold.

Nothing calls the advancing entry point. A grep of the tree finds note_standing_hold only at the adopt seed, inside sharing.rs's own alias handover, and in sharing/tests.rs, render/tests.rs, shared_trace/tests.rs. (rust-analyzer's index predates this stack, so grep is the check here.) The alias path at sharing.rs:299 reads Aliases::holds, which only that same uncalled entry point populates, so meet_over returns None and a re-export inherits the frozen hold as well.

Both ends of the mechanism document a call that does not exist. multiplex.rs's module doc says the broadcast means interactive "applies it as a standing hold on the shared arrangement, and the publisher compacts only as far as the slower of the two runtimes has applied"; sharing.rs:400 says note_standing_hold is "Called from handle_allow_compaction on the runtime that may import id but does not host it". handle_allow_compaction (src/compute/src/compute_state.rs:925) has only the drop / traces.allow_compaction branches. src/compute/src/render/tests.rs:294 calls registry.note_standing_hold directly with the comment "exactly as handle_allow_compaction does on each side", so the tests assert the intended behaviour through a call production never makes and pass while the wiring is absent.

Fix: have handle_allow_compaction call self.compute_state.sharing_registry.note_standing_hold(id, worker_index, &frontier) on the runtime that does not host id, as both docs describe. Note that merely skipping the broadcast on the non-hosting runtime does not address this: the hold then stays at its seed forever and the arrangement still never compacts. Alternatively, do not publish until something can advance the hold.