benchmarks: price oversubscription and publication at scale

[antiguru]

Stacks on #38676. Tracked by CPU-216.

The first nightly with the interactive runtime on showed reads isolated from hydration (point lookups 37x better at p99 under churn, introspection reads 100 ms instead of minutes) and nothing else moving. It left three questions the benchmarks did not ask.

What a saturated interactive runtime does to maintenance. Two runtimes are twice the worker threads on the same cores. MaintenanceUnderPeekSaturation loads an eight-worker replica, half the cores of the nightly agents, with one join peek in flight per worker and two hydration churn loops, while a writer advances a small materialized view. The measured query reads that view strict serializable from a separate one-worker cluster, so it cannot answer before the view's write frontier passes the write and nothing else competes with it: its latency is the view's maintenance lag under the peek load. Read on the loaded replica it would queue behind the joins on the runtime that serves peeks instead, which is what the join loops' own latency reports.

What publishing costs at scale with nobody reading. ManyIndexesIdle and ManyIndexesRestart publish two hundred indexed views over one table, each a distinct arrangement. The first measures one lookup and, through the clusterd memory column, the resident cost of two hundred publications. The second restarts the replica and waits for a read that imports every index, so every publication is rendered and bound again on the way up. ManyReexportsIdle is the other shape: two hundred indexes on one relation and key share one arrangement and its publication point. Before #38389 aliased re-exports it measured 125 to 185 KB per re-export, the operators of a re-import.

Whether PeekIsolationUnderExpensivePeeks guards anything. Its two sides came out identical: at 100k rows and 2/s the walks used under 10% of the worker and the lookups never noticed them. It now walks 500k rows at 3/s, about 60%. Its docstring says what it guards: both queries are peeks, so it measures how the serving runtime interleaves them, not the split between peeks and maintenance, which is why two runtimes cannot move it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDm7opomJLxbNUEP3r9BLk

[def-]

QA LLM Review

1. MEDIUM -- ManyReexportsIdle samples clusterd memory without waiting for the 200 re-exports to hydrate

misc/python/materialize/feature_benchmark/scenarios/interactive_runtime.py:202

The scenario's stated purpose is that "the clusterd memory measurement is the resident cost of INDEXES publications of the same arrangement", but nothing in init() waits for those 200 index dataflows to become resident, so the reported figure can be a snapshot taken while they are still building. Its sibling ManyIndexesIdle closes exactly this gap and this one does not, which makes the number understate precisely on the builds where re-exports are expensive.

Details

CREATE INDEX returns once the catalog transaction commits; hydration proceeds in the background. The only gate at the end of init() is SELECT count(*) FROM t (line 202). import_into_dataflow imports every index on t, but prune_and_annotate_dataflow_index_imports then drops the ones the optimized plan does not need, so that read binds a single index and waits for a single frontier. The measured SELECT f1 FROM t WHERE f1 = 1 in benchmark() is a fast-path peek on one index and does not gate either. Contrast ManyIndexes.init(), which ends on SELECT c FROM all_v: that read imports all 200 indexes, so under strict serializable it cannot answer until every one has hydrated.

What that costs: Benchmark.run() terminates after at least 6 iterations, each roughly a testdrive docker exec plus two docker stats calls, and _memory_clusterd_filter = FilterFirst(3) discards the first three samples, so the first recorded clusterd sample lands ~10s after init(). MinAggregation then reports the minimum over the recorded samples, i.e. deliberately the earliest, least-hydrated one. When re-exports share the arrangement the 200 dataflows hydrate ~instantly and this is harmless. When they do not — 200 re-imports each arranging a 100k-row collection on the one-worker cluster_default, tens of millions of record insertions on a single worker — hydration outlasts that window, and the min sample is taken mid-build. That is the regression the scenario exists to price, and it is the case in which the number is attenuated.

A gate in the same shape as the sibling's, appended to init():

> SELECT bool_and(hydrated) FROM mz_internal.mz_hydration_statuses
  WHERE object_id IN (SELECT id FROM mz_catalog.mz_indexes WHERE on_id = (SELECT id FROM mz_catalog.mz_tables WHERE name = 't'))
true

mz_internal.mz_hydration_statuses covers indexes via mz_catalog.mz_indexes LEFT JOIN mz_internal.mz_compute_hydration_statuses, with COALESCE(hydrated, false), so a not-yet-reporting index reads as false rather than dropping out of the aggregate. Worth confirming an aliased re-export does report hydrated there before relying on it; if it does not, the same effect can be had by making the last statement a read that the optimizer cannot satisfy from a single index.