compute: turn the interactive runtime on in test configurations

[antiguru]

Last of eight PRs splitting #37770. Stacks on #38392. Tracked by CPU-216.

enable_compute_interactive_runtime moves out of UNINTERESTING_SYSTEM_PARAMETERS, where #38391 parked it, and becomes a variable system parameter defaulting to on. That is what makes every preceding piece reachable: sqllogictest, testdrive, and the mzcompose suites provision replicas with two runtimes. Parallel-workload's FlipFlagsAction is unpinned to both values at the same time. Production keeps the dyncfg's own default, off.

The clusterd mzcompose service grows the second runtime's port and --interactive-compute-timely-config, and the local driver runner takes INTERACTIVE_COMPUTE=1 for the same. Four clusterd-test-driver specs cover the runtime boundary: an index read, a query dataflow that binds before its index is published, a read through an index that re-exports another's arrangement and so aliases its publication point, and a query dataflow that binds to such a re-export before it renders, the one case where the re-export publishes through an import instead. The driver gains two things the binding-first specs need. create-dataflow takes until=, and the specs set until = as_of + 1, without which the dataflow is not single-time and the multiplexer routes it to maintenance. create-dataflow ... defer registers a dataflow's exports without submitting it and submit-dataflow submits it later, so a spec can create and schedule the interactive dataflow before the index it imports exists. Its import then binds to an unbacked publication point that the later maintenance render adopts, which is the ordering the registry exists for and which a race could otherwise only reach by accident.

read_at_max_timestamp.slt covers reads at Timestamp::MAX, the one single-time shape whose until is empty and so the one that is_peek_dataflow admits while !until.is_empty() would have rejected. Such a read is answerable only against a collection that has sealed, since no finite upper passes MAX, so the file reads off constant-backed indexes, whose dataflows have no inputs and drop their capability after emitting once. Both shapes are covered: a fast-path index peek, and a join that cannot take the fast path and so builds a peek dataflow importing two shared indexes. The reads double as the assertion that those indexes sealed, since a peek finding a live frontier would never return. Passes with the flag defaulted either way.

relations.slt gains the publisher operators, and pins the flag itself rather than inheriting the environment default. A variable system parameter can be flipped by CI_SYSTEM_PARAMETERS=random, which would otherwise make the golden mismatch about half the time in randomized runs. The flag is read when a replica is provisioned, and the file creates its own cluster after reset-server, so setting it beforehand is enough.

introspection-sources.td raises a coarse arrangement-size bound from 16 KiB to 32 KiB, since publication doubles the reported size of a one-record index.

singlereplica_attribution_sources.slt asserted a materialized view's timely dataflow id, which counted the peek dataflows the replica rendered before it. Those now render on the interactive runtime, so the assertion is reduced to the id-to-collection pairing.

The ReadIsolationUnderHydration parallel-benchmark scenario from #38542 A/Bs the flag, measuring peek latency with the feature on against off while hydration saturates the maintenance workers.

[linear-code]

CPU-215

[antiguru]

Confirmed and fixed in 7e51333. create-dataflow takes an until= argument, threaded through to DataflowBuilder::until, and the spec sets until=1, so is_peek_dataflow holds and the multiplexer routes the dataflow to interactive.

The same commit fixes the relations.slt golden that failed SLT 1 on the previous push: the publisher operators are named PublishShared(<id> oks), not PublishShared.

An assertion that distinguishes the two runtimes is still missing; the interactive runtime's introspection is unreachable (CPU-222), so there is nothing to read yet.

Posted by Claude Code.

[def-]

QA LLM Review

1. MEDIUM -- two-runtime query-dataflow spec cannot reach the unbacked publication point it documents

test/clusterd-test-driver/scripts/two_runtime_query_dataflow.spec:44

The scenario the spec is built around, the interactive import binding to an unbacked publication point that schedule id=2001 later adopts, does not happen as written: create-dataflow name=maint-index renders the maintenance dataflow immediately and publishes and adopts the point right there, before the interactive dataflow is even submitted. The reader-first branch of ArrangementSharingRegistry::get_or_create is therefore reached only by an unsynchronized race between the two runtimes' worker threads, and normally not at all, so a regression that made publish install a fresh slot instead of adopting a reader-created one in place would leave this spec green.

Details

handle_create_dataflow calls build_compute_dataflow unconditionally (src/compute/src/compute_state.rs:964); its own comment at compute_state.rs:851 says "Every dataflow builds immediately, in command arrival order". render::export_index publishes during that build (src/compute/src/render.rs:917), and publish is get_or_create + adopt + notify (src/compute/src/sharing.rs:143). Schedule only drops the suspension token (compute_state.rs:985); it releases the publisher operator's first activation, it neither creates nor adopts the point.

The driver sends commands without waiting for an ack (src/clusterd-test-driver/src/driver.rs:136) and the two runtimes drain their streams on independent worker threads, so which side calls get_or_create first is a race. Since create-dataflow name=maint-index goes out first and maintenance is idle, maintenance normally wins. The comments at lines 17-21, 41-43, 59-61, 71-72 and 77-78 all assert the opposite ordering.

The script cannot simply be reordered: import index=2001 requires the index to be registered already, which only a successful create-dataflow submit does (src/clusterd-test-driver/src/script.rs:847). Reaching reader-first deterministically needs a driver command that registers an index's shape without submitting its dataflow. Short of that, the comments should describe what actually runs: the point is created and adopted at create-dataflow time with an empty chain, and what schedule id=2001 releases is the publisher's first activation, which fills the importer queue and wakes the import.

[antiguru]

Confirmed: create-dataflow rendered and published at submit time, so the reader-first order the spec described was a race maintenance normally won. Fixed in 872ae4f by giving the driver the command the finding says it needs.

create-dataflow ... defer registers the dataflow's exports without submitting it, and submit-dataflow name=<name> submits the recorded spec later. The spec now registers the maintenance index with defer, creates and schedules the interactive dataflow, and only then submits and schedules the index. The import therefore binds to an unbacked point deterministically, and publish adopting that point in place is what the peek's answer depends on. Comments rewritten to describe that sequence.

Posted by Claude Code.