Skip to content

fix(pipeline): give the delivery tier a log-halt latch at the claim gate (ADR 0189) - #1178

Open
wshallwshall wants to merge 7 commits into
mainfrom
worktree-agent-a050617fac2865a9d
Open

wshallwshall wants to merge 7 commits into
mainfrom
worktree-agent-a050617fac2865a9d

Conversation

@wshallwshall

@wshallwshall wshallwshall commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

UPDATE FROM THE MANAGER: THE ALTITUDE PASS DID RETURN, AND IT FOUND SOMETHING.
This body says below that it never returned and its output file was zero bytes. That was
true for the author, whose only instrument was that file. The report reached the parent
session instead, and its finding is posted as a comment on this PR. ITS HEADLINE WAS LATER
FALSIFIED -- see "Door six" at the foot of this body: a halt PAUSES every owned lane and a paused
lane is never claimed, so zero lanes spin on the ordinary halt. One reachable spin path remained
and is now gated.
What it said was: the pooled retry path
parks for a flat 1.0s and never reaches the dispatcher's existing exponential backoff, so
a log halt spins every pooled lane once a second for its whole duration with no
lane_stuck alert. Three of its four areas came back clean, including a check that a
lane_provider-based gate would have been leaky.

Read the comment before concluding the altitude question is unanswered. The author's
pointer to ADR 0189 Options considered 5 is still correct and still worth reading; it
simply is not the whole answer any more.

What this changes

The engine halts delivery when it cannot write its application log, because CLAUDE.md section 1
says every message a connection puts out is counted and logged. The inbound tier enforces that
with a latch (_log_halted) read at each worker's loop top. The delivery tier had no state of
its own: the halt took a lane down by pausing it, and a pause lives in _outbound_paused — which
_teardown_body deliberately clears, and which no consumer can tell apart from an operator pause.

So the rule could only be re-asserted at each door into resuming delivery, and four were found
one at a time, each only after the previous fix shipped: start_outbound, restart_outbound, a
stop()+start() teardown, and a reload's _unpark_outbound_lane. Four gates are a completeness
claim nobody can verify (CLAUDE.md section 11, SDS-3.6).

This adds _delivery_halted, read at the claim gate in both claim modes. Doors are unbounded;
claim modes are two and closed.

Claim mode Gate site Shape
per_lane _delivery_worker loop top, above the pause gate and before the claim return — no row left INFLIGHT, the same terminal state _router_worker's gate leaves
pooled _dispatch_delivery, the first runner-owned code a claimed row reaches reschedule_claimed the head + RETRY — parks the lane

The four door gates are KEPT. They fail fast and page with the reason, which the claim gate
cannot. The latch is defence in depth behind them, never the recovery path. Removing working gates
in the same change would make a regression unattributable.

ADR: 0189,
allocated with alloc.ps1 and indexed in the same commit. No BACKLOG number was allocated for this
change; #122 is cited as the existing item this subsystem already belongs to.

The fifth-door test, measured red first

tests/test_log_write_guard.py::test_an_unguarded_start_cannot_deliver_while_the_halt_is_latched
deliberately uses none of the four doors. It drives _start_outbound_unsafe directly as a
stand-in for the fifth door — whatever it turns out to be — and asserts that with the halt latched
and both sinks genuinely dead, no bytes reach the output directory.

RED on main, both claim modes:

AssertionError: a queued row shipped with no application log behind it
assert [WindowsPath('.../out/M1.hl7')] == []
  Left contains one more item: WindowsPath('.../test_an_unguarded_start_cannot1/out/M1.hl7')

FAILED tests/test_log_write_guard.py::test_an_unguarded_start_cannot_deliver_while_the_halt_is_latched[pooled]
FAILED tests/test_log_write_guard.py::test_an_unguarded_start_cannot_deliver_while_the_halt_is_latched[per_lane]
2 failed, 57 deselected in 15.19s

GREEN with the latch: 2 passed, 57 deselected in 4.51s.

The red was measured with the three _delivery_halted assertions temporarily stripped, so the
failure is the bytes assertion rather than an AttributeError on a symbol main does not have.

Which assertions are real measurements, and which are harness properties

Real measurements (the engine, through a real File connector draining a real store):

  • list(outdir.iterdir()) == [] while halted — the load-bearing one. This is the assertion that
    went red on main.
  • len(await store.outbox_for(message_id)) == 1 — the row is retained PENDING, not dead-lettered
    and not stranded INFLIGHT by the refusal.
  • await _until(lambda: any(outdir.iterdir())) and _until_processed(...) in the control arm — the
    same rig, the same queued row, delivered once the disk is repaired.

Harness properties (they check the rig reached the state under test, and would not by themselves
have caught the defect):

  • await _until(lambda: runner._log_write_stopped) — the halt fired.
  • runner._delivery_halted before and after the unguarded start — a read-back of the flag this
    change adds. Kept only to attribute the refusal to the latch rather than to some other reason the
    lane might be down.
  • assert list(outdir.iterdir()) == [] before the sinks are killed — proves the lane is
    genuinely loaded, so "no file was written" later cannot pass for the wrong reason (an empty
    outbound stage).

Operator-surface assertions (real reads of the shipped API, but state and strings rather than
bytes): outbound_status(...) == "log_halted", not outbound_running(...),
outbound_status(...) == "running" after recovery, and the resend-409 wording — that last one with
its own negative control, since a merely operator-paused lane must still get the start-it
instruction or the new branch would be a rewrite rather than a discrimination.

Claim modes

Both, via the file's existing CLAIM_MODES = ["pooled", "per_lane"]. Neither is vacuous here,
and the reason is structural rather than inherited: the gate lives in a different function in
each mode (_delivery_worker vs _dispatch_delivery) with a different refusal shape (pre-claim
return vs post-claim reschedule-and-park), so a single-mode test would leave one of the two
implementations entirely unexercised. Both were measured red on main and green with the latch.

What an operator now sees that differs

  1. A halted lane reads log_halted, not stopped. The halt takes a lane down by pausing it, and
    a pause is what an operator does — so /connections was telling an operator the lane was waiting
    for them to press start, when what it was waiting for was a writable disk. It renders in the BAD
    colour, not the muted grey of "stopped".
  2. A lane an unguarded path brought up reads log_halted rather than running. Such a lane is
    not in _outbound_paused, so it read as actively delivering while the claim gate refused every
    one of its rows.
  3. outbound_running is false for every lane while the latch holds, so /stats' running/stopped
    split counts halted lanes as not running rather than as running.
  4. A resend's 409 names the log instead of a start button. Consequence of 3, and it needed its
    own fix: both resend guards read outbound_running and 409 with "start it before resending"
    correct for an operator-paused lane, wrong for a halted engine, where the very door the message
    points at is refused until the log works. That is the compensating-control-on-a-false-premise
    shape (SDS-3.7), so the two identical guards now route through one helper that names the cause.
  5. Halted lanes with no metrics edge still get a row. log_halted joins stopping/stopped in
    the standalone-row loop, because the halt takes down every lane at once and a console listing
    only the trafficked ones would silently omit the rest.
  6. What deliberately did NOT change. not_deployed, failed and filtered still outrank
    log_halted on the display ladder — each is a fact about that one connection, fixable on that
    row, while the halt is process-wide and already has its own log_write_failed page.
    outbound_quiesced still answers off the pause set, so purge-eligibility is untouched and a
    halted-and-quiesced lane stays purgeable.

Surfaces updated: the ConnectionRow.status and GraphNode.status vocabularies, the /connections
standalone-row loop, both resend guards, .status-log_halted / .gnode-log_halted in the web
console CSS (the console derives both its class and its label from the raw string, matching how
not_deployed is handled), docs/SERVICE.md and docs/CONNECTIONS.md.

Design note: the latch is DERIVED, not a second flag

_delivery_halted is a property returning _log_write_stopped rather than a parallel boolean.
"This process cannot log and has fail-closed" is one load-bearing fact, and _log_write_stopped
already states it: both halt sites set it, only _log_recovery_ok clears it, and that path re-tests
the sinks by writing to them. A second bool set and cleared at the same moments would be state that
must agree with this one, with nothing checking that it does — SDS-3.5. It inherits the property
that matters: _teardown_body does not clear it, so the latch survives a teardown, which is the
mechanism behind doors 3 and 4.

A set[str] of lane names — mirroring _log_halted's per-inbound shape — was rejected on
measurement, not taste. _stop_all_for_log_failure pauses only the lanes not already paused, so
a set built from it would omit exactly the engine-parked lane door 4 is about; and no set covers a
lane built after the halt. Per-lane recovery is meaningless here anyway: the broken sink is
process-global, so the moment one lane's door re-validates it, no lane's halt reason survives.
The full list of alternatives considered and rejected is in ADR 0189, Options considered.

Checks run

Check Result
ruff check . All checks passed
ruff format --check . 1299 files already formatted
mypy messagefoundry (strict) Success: no issues found in 274 source files
pytest tests/test_log_write_guard.py 59 passed
pytest tests/ -k "outbound or log_write or connections" 460 passed, 37 skipped
pytest tests/test_api.py tests/test_auto_start.py tests/test_not_deployed.py tests/test_wiring_engine.py 145 passed
pytest packaging/messagefoundry-webconsole/tests 500 passed, 3 skipped
messagefoundry adr-analyze 178 ADRs analyzed; no finding against 0189
glyph scan of the ADDED lines, with a positive control 4 occurrences across the whole branch, all U+2192 in the ADR's Acceptance Criteria — the character the ADR template mandates and adr-analyze parses. Positive control on docs/FEATURE-MAP.md through the same instrument: 179 occurrences found, so the 4 is not a false zero
/simplify run before the first commit; findings and skips below

Legs a reader still has to check after this process exits. The full pytest suite was not run in
one pass — the selections above cover the changed surfaces and the web console, but not the tree. And
every leg that only exists on a hosted runner (windows-service-smoke and the other NSSM/service
legs) is invisible from here.

/simplify findings

Run on the first commit's diff. Reuse, simplification and efficiency all came back clean on the
questions that matter here; the altitude pass had not returned when this turn ended, and its question
— whether the two-site gate is the right depth, or whether anything belongs inside StageDispatcher
— is answered explicitly in ADR 0189, Options considered 5, so a reviewer can check that reasoning
directly rather than take it on trust.

  • Simplification: clean on both questions asked. The derived property is justified (four readers,
    no new stored state, nothing to fall out of sync), and two sites is the structural minimum: pooled's
    claim happens inside stage_dispatcher.py, a module RegistryRunner does not own and which has no
    generic pre-claim hook, while pushing the check down into _process_delivery_item /
    _process_delivery_batch would still be two sites and would stop preventing the per_lane claim.
    • Applied: the stale "tri-state" wording in the standalone-row comment block
      (commit 91c18a94b).
  • Reuse: clean. The per_lane gate matches the _log_halted idiom already in the three internal
    workers; the pooled gate uses reschedule_claimed, the documented primitive for exactly this
    machinery-fault-head case.
    • Skipped: a nit that .status-log_halted / .gnode-log_halted should join the existing
      var(--bad) comma-lists rather than stand alone. .status-not_deployed is already standalone
      with its own explanatory comment, and the new rules follow that precedent so they can carry the
      reason they read BAD rather than grey.
  • Efficiency: clean. One property read per claim-loop iteration, before the DB claim; in the
    halted case it is a net win, since it skips the claim I/O entirely. reschedule_claimed is a
    single guarded UPDATE on an indexed predicate, awaited on an async cursor, bounded to a 1 s cadence
    by _WORKER_ERROR_BACKOFF_SECONDS.

The third commit (the resend 409) landed after that pass; it is itself a de-duplication of two
byte-identical guards into one helper, so it moves in the direction /simplify asks for rather than
against it.


Door six, and two sentences that understated what they measured

Added 2026-09-15, on top of everything above. Three tasks and one optional extra.

The earlier headline was WRONG, and the banner at the top of this body still repeats it

The update banner says "a log halt spins every pooled lane once a second for its whole duration".
That is falsified. _stop_all_for_log_failure pauses every OWNED outbound through
_stop_outbound_unsafe -> dispatcher.pause_lane, and a PAUSED lane is never claimed. On the
ordinary halt zero lanes spin. Every structural link of that finding held; only its headline did
not. Read the banner with this paragraph beside it.

But one spin path was reachable, and it is door six.

1. Door six is gated

_reconcile_outbounds brought a newly ADDED outbound (deployed=True, auto_start=True) past both
config gates with no halt check. Such a lane is in neither _gate_parked nor _outbound_paused, so
the un-park gate never asked about it, the branches below built its connector and armed its lane, and
the reload's own notify_work seeded it READY. No bytes shipped -- the claim gate refuses every row,
which is what the latch is for -- but the lane reached that gate once per
_WORKER_ERROR_BACKOFF_SECONDS for the halt's whole duration. ADR 0189 option 4's rejection names
this case in so many words
("a lane BUILT AFTER the halt (a reload adding an outbound), which is
door six"
) and did not close it.

It is closed by widening door 4's existing gate, not by adding a seventh:

if self._delivery_halted or name in self._gate_parked:
    if unpark_permitted is None:
        unpark_permitted = self._outbound_start_permitted(name)
    if not unpark_permitted:
        if name not in self._outbound_paused:
            self._pause_delivery_lanes([name])
        continue

The first commit used a raw _delivery_halted read beside the existing gate. The /simplify altitude
pass found why that was wrong, and three independent reasons agree:

  • _outbound_start_permitted is a PROBE, not a predicate -- it re-validates the sinks by WRITING
    to them, it can CLEAR the latch, and a refusal PAGES. Its docstring requires every caller to ask at
    most once per operator action and to memoise across a loop. The raw read was documented as
    deliberately un-memoised, which argued against the helper's own contract.
  • With the raw read, whether a reload probed the log depended on whether some unrelated lane
    happened to be _gate_parked
    . Fix the disk and reload a graph that only ADDS an outbound: with a
    parked lane present the probe runs, the latch clears and the lane comes up; with none, the latch
    reads stale-True and the lane is stopped. Same repair, same reload, two outcomes.
    _stop_all_for_log_failure's own docstring says "Fix the disk, reload, and the backlog drains."
  • The raw read refused door six silently, when this ADR's stated reason for keeping door gates at
    all is that a refusal there pages with a cause.

_stop_outbound_unsafe, never _park_outbound_lane, and that is the load-bearing detail: a park
writes _gate_parked, which this very gate LIFTS the moment a probe succeeds, so a park would re-open
the hole one reload later off a marker the method wrote itself. It goes down through
_pause_delivery_lanes -- the documented single owner of how the halt takes a delivery lane down --
which routes through _stop_outbound_unsafe and adds a per-lane except, so a lane that refuses to
pause is logged and the rest of the reload still reconciles.

The caller's name not in self._outbound_paused filter is what keeps the two doors apart, and it is
the same filter _stop_all_for_log_failure uses. Every _gate_parked lane is already paused, so it
takes the no-write path and its marker survives for a later reload -- byte-identical to before.
Re-stopping an already-paused lane would CLEAR its quiescence Event, flipping a drained lane's status
back from stopped to stopping and withdrawing its purge-eligibility for a reload that changed
nothing.

Measured RED, both claim modes

tests/test_log_write_guard.py::test_a_reload_that_adds_an_outbound_into_a_dead_log_lands_it_paused,
on ADR 0189's own rig. Phase 1 seeds a genuine PENDING row for a connection through an engine-parked
lane; phase 2 brings a second runner up on the same store without that connection, halts it, then
reloads the connection in. A fresh runner over the same store is the only way to get a real queued row
for a lane this process has never heard of, which is the definition of the door.

RED with the gate removed, both modes:

AssertionError: a reload brought an outbound up while the delivery halt was latched
assert 'OB_TEST_ADDED' in {'OB_TEST'}
 +  where {'OB_TEST'} = <RegistryRunner object>._outbound_paused

FAILED tests/test_log_write_guard.py::test_a_reload_that_adds_an_outbound_into_a_dead_log_lands_it_paused[pooled]
FAILED tests/test_log_write_guard.py::test_a_reload_that_adds_an_outbound_into_a_dead_log_lands_it_paused[per_lane]
2 failed, 59 deselected

GREEN with it: 2 passed, 59 deselected.

WHAT IS RED IS THE LANE STATE, NOT THE BYTES, and the test says so. The claim gate this PR already
adds refuses every row, so the output directory stays empty with or without the new gate -- the byte
assertion is a regression guard on the latch, not the discriminator for this door. The spin is the
other half, and it is measured separately: with the lane-state assertions neutralised, the counter and
row-untouched assertions fail in pooled only (assert 9 == 0), and pass in per_lane, where the
worker returns above the claim and there is no spin to find.

9, not 1, because the window patches _WORKER_ERROR_BACKOFF_SECONDS down and waits ten claim
cycles. A window merely longer than one backoff buys a margin of one event, and a test that
discriminates by one event is a scheduling hiccup from proving nothing. Ten cycles is a real margin
and finishes in half the wall clock a single un-patched cycle needed.

2. ADR 0189's risk sentence understated the cost

It called a spin cycle "a bounded store round-trip per lane per second". Verified against
store.py, it is two write transactions and a payload decrypt:

Step Under self._lock? Commit
claim_fifo_heads: SELECT, UPDATE to inflight with attempts+1, re-SELECT, delivered_keys probe yes one, amortized across the lane chunk
_outbox_item_from_row: decrypts the payload where at-rest encryption is on (plus a shared_body read for a body_ref row) no, off the lock --
reschedule_claimed: UPDATE yes, lock retaken its own, amortized across nothing (the gate passes one id, for one lane)

That lock is the one enqueue_ingress and every stage handoff serialize behind -- both reach it
through _writer_txn, group-committer enabled or not -- so the cost is not confined to a tier already
refusing to work.

3. LaneItemResult's docstring promised the wrong producer

It said RETRY means the body "has already mark_failed'd the head and returns that additive
next_attempt_at"
. _dispatch_delivery's halt gate returns a reschedule_claimed deadline instead.

The sentence is fixed, not the type, because the widening is in the safe direction: mark_failed
spends a retry and can eventually dead-letter, reschedule_claimed does neither, and the dispatcher
reads retry_until only as a park deadline. It now names both producers and says which one spends a
retry. The LaneResultKind.RETRY enum comment repeated the same wrong half and now links to the
docstring instead of restating it.

4. (optional extra) /stats halted_claim_gate_hits

OUTBOUND rows the pooled claim gate refused while halted. This is the one signal the latch
structurally cannot get from an enumeration:
its own argument -- that counting doors is a claim
nobody can verify -- also means nobody can verify that every door is now gated. A lane can only reach
that gate if some path armed it unpaused while the halt held.

Two qualifiers travel with the number, and the runner property owns them rather than the API field:

  • a small count at the halt is expected, not a defect. The latch is set before
    _stop_all_for_log_failure takes the reload lock and pauses the lanes, so a claim already in flight
    across that window lands here legitimately. What names a door is a count that keeps climbing.
  • zero is not a clean bill. POOLED only -- same shape as claim_lock_timeouts. The per_lane gate
    is at the worker's loop TOP, above the claim. It deliberately does not tick there: every running
    per_lane worker passes that gate once at halt time, and the resulting floor would bury the signal.

No lane_stuck alert was added. The operator is already paged log_write_failed plus
connection_stopped per connection, and a second page restates a load-bearing fact (CLAUDE.md
section 11, SDS-3.5).

A flake the wide selection found, and its cause

Worth recording because it failed three assertions away from its cause, and because it is the
shape the file's own _until_processed docstring already warns about.

Phase 1 of the door-six test stopped its seeding runner as soon as the first lane's file appeared.
A written file and a resolved row are different instants: the connector writes, and the store write
marking the row DONE commits after it. Stopping inside that gap left the row INFLIGHT, phase 2's
reset_stale_inflight reverted it to PENDING, and the message could then never reach PROCESSED --
that lane is paused by the halt for the rest of the test, so nothing was ever going to deliver it
again. It surfaced as the control arm's delivered but never finalized.

Measured 1 failure in 6 before the fix, 8/8 green after it. The wait is now on the store, through
a _until_delivery_status helper, and the test asserts its own premise after the stop: exactly one
row, PENDING, for the lane phase 2 has never heard of. The door-six assertions are unchanged and
still RED in both claim modes with the gate removed.

It was found by running the wide selection, not by review, and not by the file's own suite --
which passed 61/61 every time. That is the argument for the wider -k rather than the narrow one.

Explicitly NOT done

  • is_infra_fault is not decoupled from eventual STOP. _apply_retry's streak semantics are ADR
    0070's own declared sharp edge, its blast radius is ingress/routed/response/outbound, and the lane
    population that would benefit is zero on the ordinary path. _apply_retry, _infra_backoff and
    infra_fault_policy are untouched, and ADR 0070 is not amended -- its scope is the T17 machinery
    path and it says so.
  • A REVIEWER-RAISED ALTERNATIVE I DID NOT ACT ON, for the ADR's owner to rule. The altitude pass
    argued the spin belongs in _pooled_lane_provider -- filter its OUTBOUND branch on
    _delivery_halted, and the three arming sites (start's seed loop, notify_work's union,
    _run_sweep_once's owned set) all close at once, where this gate closes one path. Its case is that
    ADR 0189 option 5 rejected that as "a partial gate that LOOKS total", which is a claim about a
    correctness gate misleading a reader, and a cost reducer sitting behind an acknowledged
    correctness backstop cannot mislead that way. I did not do it: it reverses an accepted ADR's
    recorded decision, it needs an amendment rather than a Builder's judgment, and it touches the
    dispatcher's arming for every stage. Note the reviewer agrees the reconcile gate is still needed
    either way -- a provider filter would not stop build_destination warming an MLLP socket, DB pool
    or SMART token for a lane that cannot deliver.
  • The two /simplify passes disagreed with each other about the merged gate, and the
    disagreement is recorded rather than hidden. Simplification derived the same merge and rejected it,
    on the grounds that a reload-while-halted would then probe and page where it is silent today, and
    that it trades a live read for a memo. Altitude argued the silence and the asymmetry are the
    defect. I went with altitude, because the memo is _outbound_start_permitted's own documented
    contract for a looping caller and the silence contradicts the ADR's stated reason for keeping door
    gates -- but a reviewer who weighs the extra probe higher has a real argument and it is above.

Checks run on this change

Check Result
ruff check messagefoundry tests All checks passed
ruff format --check messagefoundry tests 1083 files already formatted
mypy messagefoundry (strict) Success: no issues found in 274 source files
pytest tests/test_log_write_guard.py 61 passed
pytest tests/test_log_write_guard.py x8 (the new test) 8/8 passed, after reproducing a 1-in-6 flake and fixing its cause
pytest tests -k "outbound or log_write or reconcile or connections or stats or dispatcher or api" 1366 passed, 131 skipped, zero failures -- this wide selection found BOTH the flake and a /stats stub break that the narrow one missed
glyph scan of MY added lines, two positive controls 435 added lines, zero hits. Controls through the same instrument: docs/FEATURE-MAP.md 164, docs/CONNECTIONS.md 236 -- so the zero is not a false one. The 4 U+00B7 in the ADR's Related line predate this change
/simplify 4 agents; findings applied or recorded above

Legs a reader still has to check after this process exits. The full tree was not run in one pass,
and every leg that only exists on a hosted runner (windows-service-smoke, the other NSSM/service
legs) is invisible from here. The web console suite was not re-run for this change; it touches no
console surface, but halted_claim_gate_hits is a new StatsResponse field.

wshallwshall added 3 commits September 15, 2026 08:53
…ate (ADR 0189)

The engine halts delivery when it cannot write its application log, because
CLAUDE.md section 1 says every message a connection puts out is counted and
logged. The inbound tier enforces that with a latch read at each worker loop
top. The delivery tier had no state of its own: the halt paused each lane, and
a pause lives in _outbound_paused, which _teardown_body deliberately clears and
which no consumer can tell apart from an operator pause.

So the rule could only be re-asserted at each DOOR into resuming delivery, and
four were found one at a time, each only after the previous fix shipped. Four
gates are a completeness claim nobody can verify (SDS-3.6).

Add _delivery_halted, read at the CLAIM GATE in both claim modes: the per_lane
_delivery_worker loop top (before the claim, so no row is left INFLIGHT) and
the pooled _dispatch_delivery adapter (post-claim, so it reschedules the head
and parks the lane). Doors are unbounded; claim modes are two and closed.

The latch is DERIVED from _log_write_stopped rather than a second flag: that
fact is already stated once, and _teardown_body does not clear it, which is the
teardown survival doors 3 and 4 needed. The four door gates are KEPT as defence
in depth; they fail fast and page with the reason.

Operator-visible: outbound_status gains log_halted ahead of the running/
stopping/stopped triple and outbound_running is false while it holds, so a
halted lane stops reading as an operator pause it never had.

The acceptance test uses none of the four doors. It drives
_start_outbound_unsafe directly as a stand-in for the fifth door and asserts on
bytes over a real File connector draining a real store, with a repaired-disk
control arm. Measured red on main in both claim modes.
The block above the standalone-row loop described the live outbound state as a
tri-state, and the loop it explains now matches on three names plus log_halted.
Found by /simplify on this change. Comment only; no behaviour.
outbound_running is now false for every lane while the delivery tier is
log-halted, so the two resend guards 409 with "start it before resending" -
the right instruction for an operator-paused lane and the wrong one here,
because the door it names is refused until the application log is writable
again. A 409 that sends an operator to a control that cannot help is the
compensating-control-on-a-false-premise shape (SDS-3.7).

Both guards ask the identical question, so route them through one helper that
names the cause when the lane is halted. Pinned by an assertion in the ADR 0189
acceptance test, with a negative control: a merely operator-paused lane must
still get the start-it instruction, so the branch is a discrimination rather
than a rewrite.
@wshallwshall

Copy link
Copy Markdown
Collaborator Author

ALTITUDE REVIEW RESULT, from the Manager who dispatched this. The PR body records that /simplify's altitude pass never returned and its output file was zero bytes. It has since returned, and it found something. Posting it because the PR is otherwise complete and a reviewer would have no reason to look for this.

THREE OF FOUR AREAS ARE CLEAN, and the review checked rather than assumed:

  • The two-gate split is right. _delivery_halted is a single derived property with zero new mutable state, read at two sites, which is already this codebase idiom for the sibling inbound latch. It also tested the obvious alternative: gating earlier via lane_provider is leaky, because notify_work unions the provider with set(self._states) and producer wakes call mark_ready directly, bypassing the provider entirely. Two choke points is the right count.
  • StageDispatcher was correctly left alone. Teaching a deliberately-minimal 4-stage state machine about one stage runner-owned concept would be the real altitude violation.
  • The log_halted status string sits at the right layer. Both API call sites fall through to outbound_status() with zero new logic, and no new special-casing was needed anywhere -- which is the sign it was added once, at the source.

THE ONE REAL FINDING, and it is worth a decision before or shortly after this lands.

The POOLED gate is post-claim, which is forced by StageDispatcher owning the claim. That positioning is fine. The problem is what the compensating action reinvents.

_dispatch_delivery returns LaneItemResult(RETRY, park_until) via reschedule_claimed. That path always carries is_infra_fault=False, so _apply_retry resets infra_error_streak = 0 and parks for exactly park_until -- which here is always a flat 1.0s. It never reaches _infra_backoff(streak), the exponential curve capped at 60s that the dispatcher ALREADY implements for exactly this fault class, under ADR 0070 fix A/B, described there as a "machinery fault, not the message's" -- the same phrase this diff own comment invokes.

CONCRETE COST: for the whole duration of a log-write halt, which could be hours, every pooled outbound lane re-enters claim-and-reschedule once per second, forever, with no growth and NO lane_stuck ALERT -- that alert is only reachable through the is_infra_fault branch. That is the identical escalation-less spin ADR 0070 was built to close, reproduced one layer up.

WHY IT HAPPENED, and this is the part that makes it a design question rather than a mistake: is_infra_fault=True is entangled with eventual STOP under infra_fault_policy="stop", and this ADR explicitly rejects STOP for a log halt. So avoiding that path was NECESSARY. The right fix is to generalise the ladder so "escalating backoff" and "eventual STOP" are separable, not to bypass the ladder and hand-roll a flatter substitute beside it.

SECONDARY, SAME ROOT CAUSE: LaneItemResult documented contract says "RETRY: it has already mark_failed"d the head." Every other producer goes through _to_lane_result, backed by _mark_failed_and_arm, honouring that. This is the ONLY call site constructing LaneItemResult(RETRY, ...) directly against reschedule_claimed, which quietly widens a shared type behaviour past what its own docstring -- in a file this diff does not touch -- tells callers they may assume.

MY RECOMMENDATION: land this PR as it stands. It closes a real hole and the fifth-door test is the structural guarantee that was missing. Do NOT widen it to generalise the dispatcher ladder -- that is a change to core machinery that wants its own change and probably an ADR 0070 amendment. But file it, because an operator whose logging has failed currently gets a lane spinning once a second for hours with no alert naming it.

No number is allocated for that follow-up and I have not cited one.

@github-actions github-actions Bot added the ci-red A required check went red. Attribute it before retrying. label Sep 15, 2026
wshallwshall added 4 commits September 15, 2026 10:50
…lt is latched

ADR 0189 replaced four door gates with a claim-gate latch and named, in option
4's rejection, one case a lane-name set could not cover: "a lane BUILT AFTER the
halt (a reload adding an outbound), which is door six". It named it and did not
close it.

_reconcile_outbounds brings a newly added outbound past the deployed and
auto_start gates with no halt check. Such a lane is in neither _gate_parked nor
_outbound_paused, so the un-park gate above it never asks, the branches below
build its connector and arm its lane, and the reload's own notify_work seeds it
READY. No bytes ship -- the claim gate refuses every row, which is what the latch
is for -- but the lane reaches that gate once per _WORKER_ERROR_BACKOFF_SECONDS
for the halt's whole duration. On the ordinary halt zero lanes do that, because
_stop_all_for_log_failure pauses every owned lane and a PAUSED lane is never
claimed; this was the one reachable exception.

The gate routes such a lane through _stop_outbound_unsafe, NOT
_park_outbound_lane. A park writes _gate_parked, which the un-park gate directly
above LIFTS the moment a probe succeeds, so a park would re-open the door one
reload later off a marker the method wrote itself. Stopping leaves the lane where
the halt left every other lane, so recovery is the same operator start through a
gated door. A lane already paused is left untouched: _stop_outbound_unsafe clears
the quiescence Event, which would flip a drained lane from 'stopped' back to
'stopping' and withdraw its purge-eligibility for a reload that changed nothing.

Two prose corrections, both understatements that would be cited later as
measurements.

ADR 0189's risk sentence called a spin cycle "a bounded store round-trip per lane
per second". Measured against store.py it is two write transactions and a payload
decrypt: claim_fifo_heads SELECTs, UPDATEs to inflight with attempts+1,
re-SELECTs and probes delivered_keys under the process-wide self._lock with one
commit amortized across the lane chunk; off the lock _outbox_item_from_row
decrypts the payload where at-rest encryption is on; then reschedule_claimed
takes self._lock again for its own UPDATE and its own commit, amortized across
nothing. That lock is the one enqueue_ingress and every stage handoff serialize
behind, via _writer_txn, so the cost is not confined to a tier already refusing
to work.

LaneItemResult's docstring promised RETRY means the body "has already
mark_failed'd the head". _dispatch_delivery's halt gate returns a
reschedule_claimed deadline instead. The widening is in the safe direction --
reschedule_claimed spends no retry and cannot dead-letter, and the dispatcher
reads retry_until only as a park deadline -- so the sentence is fixed to name
both producers rather than the type narrowed.

Test: a reload that adds an outbound into a dead log lands it paused, on ADR
0189's own rig, parametrized over both claim modes. Measured RED in BOTH modes
without the gate: "assert 'OB_TEST_ADDED' in {'OB_TEST'}" on _outbound_paused.
With the lane-state assertions neutralized, the row-untouched assertion fails in
POOLED only, which is the spin itself.
…counter

Follow-up to the previous commit, applying a /simplify pass. Three changes, one
of which is behavioural.

BEHAVIOURAL. Door six was gated by a raw `_delivery_halted` read beside the
existing `_gate_parked` probe. Two adjacent gates answering the same question by
two different mechanisms, and the raw read was the wrong one:

- `_outbound_start_permitted` is a PROBE, not a predicate. It re-validates the
  sinks by WRITING to them, it can CLEAR the latch, and a refusal PAGES. Its
  docstring requires every caller to ask at most once per operator action and to
  memoise across a loop. The raw read was documented as deliberately un-memoised,
  which argued against the helper's own stated contract.
- With the raw read, whether a reload probes the log at all depended on whether
  some UNRELATED lane happened to be `_gate_parked`. Fix the disk and reload a
  graph that only ADDS an outbound: with a parked lane present the probe runs,
  the latch clears and the lane comes up; with none, the latch reads stale-True
  and the lane is stopped. Same repair, same reload, two outcomes.
  `_stop_all_for_log_failure`'s own docstring says "Fix the disk, reload, and the
  backlog drains".
- The raw read refused door six SILENTLY. ADR 0189's stated reason for keeping
  the door gates is that a refusal there pages with a cause.

So the two collapse into `if self._delivery_halted or name in self._gate_parked`,
asking the one memoised probe. The `_delivery_halted` half keeps a HEALTHY reload
from probing at all, which is the pre-existing behaviour. The doors still differ
in what they leave behind, and the caller's `name not in self._outbound_paused`
filter is what separates them: every `_gate_parked` lane is already paused, so it
takes the no-write path and its marker survives for a later reload, exactly as
before; a lane this reload would have brought up is not, so it goes down through
`_pause_delivery_lanes`. No new alert type; it reuses the page the door-4 gate
already produces.

Also down through `_pause_delivery_lanes` rather than calling
`_stop_outbound_unsafe` directly. That helper's docstring says it is the one place
that knows how the halt takes a delivery lane down and that a third copy of the
loop is how the two halts drift. Reusing it inherits its per-lane except, so a
lane that refuses to pause is logged and the rest of the reload's outbounds still
reconcile, instead of the raise aborting the loop partway down the list.

NEW INSTRUMENT. `/stats` grows `halted_claim_gate_hits`: OUTBOUND rows the pooled
claim gate refused while the delivery tier was halted. This is the one signal the
latch structurally cannot get from an enumeration -- its own argument, that
counting doors is a claim nobody can verify, also means nobody can verify that
every door is gated. A lane can only reach that gate if some path armed it
unpaused while the halt held. Two qualifiers travel with the number and the runner
property owns them: a small count at the moment of the halt is the documented
window between setting the latch and pausing the lanes, and zero is not a clean
bill because the counter is POOLED-only (the per_lane gate sits above the claim,
and making it tick at the loop top would floor the signal with every running
worker at halt time).

TEST. The absence-of-a-spin window now patches `_WORKER_ERROR_BACKOFF_SECONDS`
down and waits ten claim cycles rather than one. A window merely longer than one
backoff bought a margin of one event. Measured with the gate removed: 9 claim-gate
hits inside the window, in half the wall clock the un-patched single cycle needed.

The /stats field is read through getattr-with-default, like the counters beside
it, and test_api's stub-the-runner test now carries it with its own distinct
value. That test exists because every field there was once replaceable with a
literal 0 without a reader noticing; direct attribute access also 500'd the whole
stats read against its SimpleNamespace stub.

`_until_outbound_rows` folded into `_until_outbound_row` as a `count` kwarg, and
the second outbound moved from a `_two_outbound_registry` builder that wrote
`reg.handlers["h"]` directly -- bypassing `add_handler`'s duplicate guard and its
`handler_accepts` bookkeeping -- to `added_outdir` / `added_auto_start` knobs on
`_e2e_registry`. Two same-file restatements removed from the reconcile memo and
the ADR.
…committed row

Found by running the wide selection, not by review: one failure in six, and it
surfaced three assertions away from its cause as the control arm's "delivered but
never finalized".

Phase 1 of the door-six test stopped its seeding runner as soon as the first
lane's file appeared. The file existing and the row being DONE are different
instants -- the connector writes, and the store write marking the row resolved
commits after it. Stopping the runner inside that gap left the row INFLIGHT,
phase 2's reset_stale_inflight reverted it to PENDING, and the message could then
never reach PROCESSED: that lane is paused by the halt for the rest of the test,
so nothing was ever going to deliver it again.

The wait is now on the STORE, through a new `_until_delivery_status` helper, and
the test also asserts its own premise after the stop -- exactly one row, PENDING,
for the lane phase 2 has never heard of.

Measured 8/8 green after the fix, having reproduced the failure at 1/6 before it.
The door-six assertions are unchanged and still RED in both claim modes with the
gate removed: "assert 'OB_TEST_ADDED' in {'OB_TEST'}" on _outbound_paused.
@wshallwshall

Copy link
Copy Markdown
Collaborator Author

RESOLVING THE ONE OPEN QUESTION IN THIS PR BODY, from the Manager who dispatched it. The body records that the two /simplify passes disagreed about whether the spin gate belongs in _pooled_lane_provider, and leaves both arguments standing rather than picking. The disagreement is already settled by a measurement taken before this commit existed.

THE lane_provider PLACEMENT IS LEAKY, AND IT WAS CHECKED AGAINST THE TREE. An adversarial review of this PR examined exactly that alternative and confirmed ADR 0189 option 5 rejection on mechanism rather than on preference:

notify_work unions self._lane_provider() with set(self._states)
producer wakes call mark_ready DIRECTLY -- _wake_lane -> d.mark_ready
(wiring_runner.py:1258, invoked from 5864, 6163, 6361 among others)

So a lane can be armed without ever passing through lane_provider. A gate placed there would be bypassed by every producer wake, which is precisely the leak option 5 names. Two choke points is the right count.

SO NO ADR AMENDMENT IS NEEDED and the Builder was right not to take it on its own judgment. The altitude pass argument -- that one placement closes three arming sites at once -- is true about the sites it can see and wrong about the ones it cannot. Recording it here so the next reader does not reopen it from the PR body alone.

TWO THINGS THIS BUILDER DID BETTER THAN THE BRIEF I GAVE IT, worth naming because both are the kind of deviation a reviewer should want:

  1. I BRIEFED A SEPARATE RAW-LATCH CHECK IN _reconcile_outbounds. It refused, and gave three reasons it verified against the tree: _outbound_start_permitted is a probe whose own docstring REQUIRES a looping caller to memoise; a raw read would make "fix the disk and reload" work only when some unrelated lane happened to be _gate_parked, contradicting what _stop_all_for_log_failure documents; and a raw read would refuse door six SILENTLY, when a refusal that pages is the stated reason the door gates exist at all. It widened door 4 gate instead of adding a seventh. That is the better change and it is the opposite of what I asked for.

  2. I BRIEFED "ASSERT NO FILE REACHES THE OUTPUT DIRECTORY". That assertion would have PASSED FOR THE WRONG REASON -- the claim gate already refuses every row, so the directory is empty with or without the door-six fix. The Builder said so in the test itself and moved the discriminator onto lane state, measuring the spin separately (pooled-only, assert 9 == 0). A test that cannot distinguish the fix from its absence is worse than no test, and my brief would have shipped one.

IT ALSO FOUND A 1-IN-6 FLAKE IN ITS OWN NEW TEST, by wide selection rather than by review: phase 1 waited for a FILE where it needed a committed ROW, and stopping in that gap left the row INFLIGHT for reset_stale_inflight to revert, surfacing three assertions later as the control arm "delivered but never finalized". Fixed at the source, 8/8 after. Worth knowing that the failure presented far from its cause.

Final state: 1366 passed, 131 skipped, zero failures; ruff, ruff format and mypy strict clean; glyph scan zero over 435 added lines against controls of 164 and 236.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-red A required check went red. Attribute it before retrying.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant