benchmarks: guard the interactive runtime's read paths

[antiguru]

Stacks on #38393. Tracked by CPU-216.

Benchmarks and regression tests for the read paths the interactive runtime changes, so a regression in them shows up in the nightly comparison against the previous build rather than in production.

Feature benchmark. scenarios/interactive_runtime.py adds two shapes the main scenario set lacks: a peek dataflow joining two indexes (the temporary-dataflow floor) and a per-replica introspection read served from the maintenance runtime's published logging index. The point lookup and CREATE INDEX plus first read are already FastPathFilterIndex and CreateIndex. The benchmark's replica is an unmanaged one backed by the composition's clusterd container, which enable_compute_interactive_runtime never reaches, so the container is configured with the second runtime directly. An image without the option ignores the variable, so every scenario runs against a two-runtime replica on this build and against whatever the other build's image provides, with no flag pinning and no version gate. The nightly's this-vs-other comparison is the regression signal, and the first build carrying the runtime shows the on-vs-off delta there.

Parallel benchmark. Three scenarios for what the second runtime is meant to buy. TemporaryDataflowFloor is a closed loop of a join over two small indexed tables on a quiet replica, so the reported latency is the dataflow's fixed cost. IntrospectionUnderHydration reads mz_dataflow_arrangement_sizes at a fixed rate on the replica that hydration churn saturates. FreshnessUnderPeekWalks alternates a write and a strict serializable read on one connection while full index walks run on the same replica, so the read's latency is how far the walks hold the written index's frontier back. The two existing isolation scenarios now report regressions on their measured loops, with CONTENDED_THRESHOLDS looser than the defaults on the stats the suite gates, including the closed-loop qps that is the mean latency inverted. p99 stays reported and ungated, as everywhere else.

Regression tests. interactive_runtime.slt pins the flag on a two-worker replica and covers same-key duplicate indexes across drops, error results on the fast path and through a peek dataflow, strict serializable reads over shared arrangements, and cluster re-provisioning. The clusterd-test-driver two-runtime workflow runs its specs with one and two workers, so the registry's per-worker pairing is exercised rather than assumed.

Not covered here, because the frameworks cannot express them: a resource benchmark for the idle cost of the second runtime (RSS and CPU with many indexes and no reads), an assertion that a query ran on the interactive runtime (blocked on CPU-222), and cancellation of a parked shared peek.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDm7opomJLxbNUEP3r9BLk

[def-]

QA LLM Review

1. HIGH -- feature-benchmark scenarios never run on a two-runtime replica

misc/python/materialize/feature_benchmark/scenarios/interactive_runtime.py:12

The four new scenarios run on a replica that can never have an interactive runtime, so they measure a single-runtime clusterd on both sides of the nightly comparison and give no signal for the feature they exist to guard. The module docstring and the PR body both assert the opposite, so a green result here will be read as evidence the interactive read paths did not regress.

Details

test/feature-benchmark/mzcompose.py:298-300 creates cluster_default as an unmanaged replica (COMPUTECTL ADDRESSES ['clusterd:2101'] … WORKERS 1) and makes it the system default cluster, so every benchmark query runs on the clusterd container. create_replica ignores interactive_runtime on the ReplicaLocation::Unmanaged arm (src/controller/src/clusters.rs:467-478); only provision_replica acts on it. So enable_compute_interactive_runtime=true, which get_default_system_parameters() does put in environmentd's MZ_SYSTEM_PARAMETER_DEFAULT, has no effect at all in this composition.

The container is built by create_clusterd_service (test/feature-benchmark/mzcompose.py:393-398) as a bare Clusterd(image=…), i.e. interactive_compute=False, so no CLUSTERD_INTERACTIVE_COMPUTE_TIMELY_CONFIG is set and clusterd starts one runtime (src/clusterd/src/lib.rs:516-557, where the second runtime is gated purely on that argument).

Note the parallel-benchmark half of the PR is unaffected: test/parallel-benchmark/mzcompose.py uses orchestrated replicas, so the flag does reach them.

Fix — configure the second runtime on the container, since environmentd is not in a position to:

 def create_clusterd_service(
     clusterd_image: str | None,
     default_size: int,
     additional_system_parameter_defaults: dict[str, str] | None,
 ) -> Clusterd:
-    return Clusterd(image=clusterd_image)
+    # Benchmark queries run on `cluster_default`, an unmanaged replica backed by
+    # this container. environmentd never provisions it, so
+    # `enable_compute_interactive_runtime` never reaches it and the runtime has
+    # to be configured here. An older image ignores the unknown env var and
+    # stays single-runtime, which is the intended this-vs-other comparison.
+    return Clusterd(image=clusterd_image, interactive_compute=True)

Clusterd derives the interactive timely config from the same workers value as the maintenance one (default 1), which matches the WORKERS 1 in the replica definition.

2. MEDIUM -- qps is not in CONTENDED_THRESHOLDS, so the loosening does nothing for FreshnessUnderPeekWalks

misc/python/materialize/parallel_benchmark/scenarios.py:1278

CONTENDED_THRESHOLDS covers avg/p50/p95/p99 but not qps, which falls through to the default 1.2. For a closed loop qps is just 1/mean latency, so FreshnessUnderPeekWalks still fails on a 20% mean-latency increase even though its avg threshold was deliberately set to 1.5, and the scenario will report regressions at exactly the run-to-run variance the constant exists to tolerate.

Details

check_regressions resolves the threshold per stat as scenario.regression_thresholds.get(query, {}).get(stat) or REGRESSION_THRESHOLDS[stat] (test/parallel-benchmark/mzcompose.py:627-633), and less_than_is_regression("qps") is true, so the test is this_qps < other_qps / 1.2, i.e. mean_this > 1.2 * mean_other. The measured action at scenarios.py:1786-1792 is a ClosedLoop, whose Statistics.qps = queries / elapsed, so qps and avg are the same signal inverted and the stricter of the two binds.

The two existing scenarios that gained report_regressions=True are unaffected: their measured loops are OpenLoop at a fixed Periodic rate, where qps is the offered rate and does not move. TemporaryDataflowFloor is also a closed loop but uses the defaults throughout, so it is self-consistent.

Adding "qps": 1.5 to CONTENDED_THRESHOLDS fixes it without affecting the open-loop scenarios.

[antiguru]

Both confirmed and fixed in 7d7df7c.

1. create_clusterd_service now passes interactive_compute=True, with a comment on why the system parameter cannot reach the unmanaged replica. The module doc and PR body now describe the container configuration instead of the flag.
2. CONTENDED_THRESHOLDS gains "qps": 1.5.

Posted by Claude Code.

[def-]

QA LLM Review

1. MEDIUM -- CONTENDED_THRESHOLDS turns the p99 gate on for four scenarios rather than loosening it

misc/python/materialize/parallel_benchmark/scenarios.py:1282

REGRESSION_THRESHOLDS["p99"] is None, i.e. p99 is deliberately not checked for any query in the suite. Putting "p99": 2.0 in CONTENDED_THRESHOLDS therefore does not loosen anything: it newly subjects the four scenarios that use the constant to a p99 gate that no other query in the benchmark has, and it does so on exactly the loops whose tails are the noisiest in the suite. The likely outcome is red nightlies from run-to-run tail variance rather than from a real regression.

Details

check_regressions resolves a threshold as scenario.regression_thresholds.get(query, {}).get(stat) or REGRESSION_THRESHOLDS[stat] (test/parallel-benchmark/mzcompose.py:627-633), and Statistics.__dir__ (test/parallel-benchmark/mzcompose.py:260-272) includes p99, so a per-query 2.0 wins over the global None and the comparison runs.

The four other entries do loosen: qps 1.5 vs 1.2, avg 1.5 vs 1.2, p50 1.5 vs 1.2, p95 1.5 vs 1.3. p99 is the only one that goes from unchecked to checked, which also contradicts the constant's own rationale ("contended tails vary more between runs than a quiet query's") -- a quiet query's p99 is not checked at all.

This bites hardest on the two scenarios this PR flips from report_regressions=False to True. ReadIsolationUnderHydration and PeekIsolationUnderExpensivePeeks previously had no gate on any stat; they now get a p99 gate whose sensitivity their own docstrings warn about ("Check queries on both sides before comparing percentiles. A query that raises is logged and dropped rather than recorded, so a run that lost samples reports percentiles over the ones that survived, and the ones that fail are the ones taken when the replica was worst.").

Dropping the "p99" key restores the intended behaviour: the four remaining entries loosen the checked stats, and p99 stays reportable-but-not-gating like everywhere else. Keeping it is fine too, but then it is a new gate and worth saying so rather than filing it under "looser thresholds for contended tails".

[def-]

QA LLM Review

1. MEDIUM -- PointLookup and CreateIndexPublish are exact duplicates of existing feature-benchmark scenarios

misc/python/materialize/feature_benchmark/scenarios/interactive_runtime.py:69

Two of the four new scenarios render byte-identical testdrive scripts to FastPathFilterIndex and CreateIndex, at the same SCALE, on the same replica. Now that create_clusterd_service turns the second runtime on for the whole composition, those two existing scenarios already measure these shapes on a two-runtime clusterd, so the new pair adds no signal and doubles the nightly cost of two of the more expensive workloads.

Details

PointLookup (interactive_runtime.py:69) vs FastPathFilterIndex (benchmark_main.py:120): both init build v1 AS SELECT <unique_values> AS f1 FROM <join> plus CREATE DEFAULT INDEX ON v1, and both benchmark bodies are SET auto_route_introspection_queries TO false / BEGIN / SELECT 1 /* A */ / 1000 x SELECT * FROM v1 WHERE f1 = 1 / SELECT 1 /* B */. REPEAT = 1000 equals the existing range(0, 1000), and both inherit SCALE = 6.

CreateIndexPublish (interactive_runtime.py:109) vs CreateIndex (benchmark_main.py:840): identical init (t1 filled from <unique_values>, then SELECT 1 FROM t1 WHERE f1 = 0) and identical benchmark (DROP INDEX IF EXISTS i1 /* A */, CREATE INDEX i1 ON t1(f1), the two-way self-join on f1 = 0 at /* B */). Normalising for ;, case and whitespace, the only textual difference between the two classes is that the original carries a # Make sure the dataflow is fully hydrated comment.

Both duplicates also run under --root-scenario Scenario (the default), so a regression in the fast path or in CREATE INDEX now shows up as two report rows under different names, which reads as two independent signals during triage. PeekDataflowJoin and IntrospectionRead are genuinely new shapes and are unaffected.

Dropping the two duplicated classes keeps the coverage: their twins now run against the two-runtime container. If the intent was a distinct shape, they need to differ from it -- for example a lookup outside an explicit transaction, so per-read timestamp selection is in the measurement, or a CREATE INDEX whose first reader is a fast-path peek rather than a join.

[antiguru]

Both remaining review items confirmed and fixed in 18ac9dc.

• PointLookup and CreateIndexPublish are gone. They rendered the same testdrive as FastPathFilterIndex and CreateIndex, which run on the same two-runtime container, so they were two report rows for one signal. The module doc names the originals so nobody re-adds them.
• "p99" is out of CONTENDED_THRESHOLDS. The global threshold for p99 is None, so the entry created a gate on the noisiest loops rather than loosening one. The four remaining entries loosen stats the suite already gates.

Posted by Claude Code.