fix(pipeline): give the delivery tier a log-halt latch at the claim gate (ADR 0189) - #1178
wshallwshall wants to merge 7 commits into
Conversation
…ate (ADR 0189) The engine halts delivery when it cannot write its application log, because CLAUDE.md section 1 says every message a connection puts out is counted and logged. The inbound tier enforces that with a latch read at each worker loop top. The delivery tier had no state of its own: the halt paused each lane, and a pause lives in _outbound_paused, which _teardown_body deliberately clears and which no consumer can tell apart from an operator pause. So the rule could only be re-asserted at each DOOR into resuming delivery, and four were found one at a time, each only after the previous fix shipped. Four gates are a completeness claim nobody can verify (SDS-3.6). Add _delivery_halted, read at the CLAIM GATE in both claim modes: the per_lane _delivery_worker loop top (before the claim, so no row is left INFLIGHT) and the pooled _dispatch_delivery adapter (post-claim, so it reschedules the head and parks the lane). Doors are unbounded; claim modes are two and closed. The latch is DERIVED from _log_write_stopped rather than a second flag: that fact is already stated once, and _teardown_body does not clear it, which is the teardown survival doors 3 and 4 needed. The four door gates are KEPT as defence in depth; they fail fast and page with the reason. Operator-visible: outbound_status gains log_halted ahead of the running/ stopping/stopped triple and outbound_running is false while it holds, so a halted lane stops reading as an operator pause it never had. The acceptance test uses none of the four doors. It drives _start_outbound_unsafe directly as a stand-in for the fifth door and asserts on bytes over a real File connector draining a real store, with a repaired-disk control arm. Measured red on main in both claim modes.
The block above the standalone-row loop described the live outbound state as a tri-state, and the loop it explains now matches on three names plus log_halted. Found by /simplify on this change. Comment only; no behaviour.
outbound_running is now false for every lane while the delivery tier is log-halted, so the two resend guards 409 with "start it before resending" - the right instruction for an operator-paused lane and the wrong one here, because the door it names is refused until the application log is writable again. A 409 that sends an operator to a control that cannot help is the compensating-control-on-a-false-premise shape (SDS-3.7). Both guards ask the identical question, so route them through one helper that names the cause when the lane is halted. Pinned by an assertion in the ADR 0189 acceptance test, with a negative control: a merely operator-paused lane must still get the start-it instruction, so the branch is a discrimination rather than a rewrite.
|
ALTITUDE REVIEW RESULT, from the Manager who dispatched this. The PR body records that THREE OF FOUR AREAS ARE CLEAN, and the review checked rather than assumed:
THE ONE REAL FINDING, and it is worth a decision before or shortly after this lands. The POOLED gate is post-claim, which is forced by StageDispatcher owning the claim. That positioning is fine. The problem is what the compensating action reinvents.
CONCRETE COST: for the whole duration of a log-write halt, which could be hours, every pooled outbound lane re-enters claim-and-reschedule once per second, forever, with no growth and NO WHY IT HAPPENED, and this is the part that makes it a design question rather than a mistake: SECONDARY, SAME ROOT CAUSE: MY RECOMMENDATION: land this PR as it stands. It closes a real hole and the fifth-door test is the structural guarantee that was missing. Do NOT widen it to generalise the dispatcher ladder -- that is a change to core machinery that wants its own change and probably an ADR 0070 amendment. But file it, because an operator whose logging has failed currently gets a lane spinning once a second for hours with no alert naming it. No number is allocated for that follow-up and I have not cited one. |
…lt is latched
ADR 0189 replaced four door gates with a claim-gate latch and named, in option
4's rejection, one case a lane-name set could not cover: "a lane BUILT AFTER the
halt (a reload adding an outbound), which is door six". It named it and did not
close it.
_reconcile_outbounds brings a newly added outbound past the deployed and
auto_start gates with no halt check. Such a lane is in neither _gate_parked nor
_outbound_paused, so the un-park gate above it never asks, the branches below
build its connector and arm its lane, and the reload's own notify_work seeds it
READY. No bytes ship -- the claim gate refuses every row, which is what the latch
is for -- but the lane reaches that gate once per _WORKER_ERROR_BACKOFF_SECONDS
for the halt's whole duration. On the ordinary halt zero lanes do that, because
_stop_all_for_log_failure pauses every owned lane and a PAUSED lane is never
claimed; this was the one reachable exception.
The gate routes such a lane through _stop_outbound_unsafe, NOT
_park_outbound_lane. A park writes _gate_parked, which the un-park gate directly
above LIFTS the moment a probe succeeds, so a park would re-open the door one
reload later off a marker the method wrote itself. Stopping leaves the lane where
the halt left every other lane, so recovery is the same operator start through a
gated door. A lane already paused is left untouched: _stop_outbound_unsafe clears
the quiescence Event, which would flip a drained lane from 'stopped' back to
'stopping' and withdraw its purge-eligibility for a reload that changed nothing.
Two prose corrections, both understatements that would be cited later as
measurements.
ADR 0189's risk sentence called a spin cycle "a bounded store round-trip per lane
per second". Measured against store.py it is two write transactions and a payload
decrypt: claim_fifo_heads SELECTs, UPDATEs to inflight with attempts+1,
re-SELECTs and probes delivered_keys under the process-wide self._lock with one
commit amortized across the lane chunk; off the lock _outbox_item_from_row
decrypts the payload where at-rest encryption is on; then reschedule_claimed
takes self._lock again for its own UPDATE and its own commit, amortized across
nothing. That lock is the one enqueue_ingress and every stage handoff serialize
behind, via _writer_txn, so the cost is not confined to a tier already refusing
to work.
LaneItemResult's docstring promised RETRY means the body "has already
mark_failed'd the head". _dispatch_delivery's halt gate returns a
reschedule_claimed deadline instead. The widening is in the safe direction --
reschedule_claimed spends no retry and cannot dead-letter, and the dispatcher
reads retry_until only as a park deadline -- so the sentence is fixed to name
both producers rather than the type narrowed.
Test: a reload that adds an outbound into a dead log lands it paused, on ADR
0189's own rig, parametrized over both claim modes. Measured RED in BOTH modes
without the gate: "assert 'OB_TEST_ADDED' in {'OB_TEST'}" on _outbound_paused.
With the lane-state assertions neutralized, the row-untouched assertion fails in
POOLED only, which is the spin itself.
…counter Follow-up to the previous commit, applying a /simplify pass. Three changes, one of which is behavioural. BEHAVIOURAL. Door six was gated by a raw `_delivery_halted` read beside the existing `_gate_parked` probe. Two adjacent gates answering the same question by two different mechanisms, and the raw read was the wrong one: - `_outbound_start_permitted` is a PROBE, not a predicate. It re-validates the sinks by WRITING to them, it can CLEAR the latch, and a refusal PAGES. Its docstring requires every caller to ask at most once per operator action and to memoise across a loop. The raw read was documented as deliberately un-memoised, which argued against the helper's own stated contract. - With the raw read, whether a reload probes the log at all depended on whether some UNRELATED lane happened to be `_gate_parked`. Fix the disk and reload a graph that only ADDS an outbound: with a parked lane present the probe runs, the latch clears and the lane comes up; with none, the latch reads stale-True and the lane is stopped. Same repair, same reload, two outcomes. `_stop_all_for_log_failure`'s own docstring says "Fix the disk, reload, and the backlog drains". - The raw read refused door six SILENTLY. ADR 0189's stated reason for keeping the door gates is that a refusal there pages with a cause. So the two collapse into `if self._delivery_halted or name in self._gate_parked`, asking the one memoised probe. The `_delivery_halted` half keeps a HEALTHY reload from probing at all, which is the pre-existing behaviour. The doors still differ in what they leave behind, and the caller's `name not in self._outbound_paused` filter is what separates them: every `_gate_parked` lane is already paused, so it takes the no-write path and its marker survives for a later reload, exactly as before; a lane this reload would have brought up is not, so it goes down through `_pause_delivery_lanes`. No new alert type; it reuses the page the door-4 gate already produces. Also down through `_pause_delivery_lanes` rather than calling `_stop_outbound_unsafe` directly. That helper's docstring says it is the one place that knows how the halt takes a delivery lane down and that a third copy of the loop is how the two halts drift. Reusing it inherits its per-lane except, so a lane that refuses to pause is logged and the rest of the reload's outbounds still reconcile, instead of the raise aborting the loop partway down the list. NEW INSTRUMENT. `/stats` grows `halted_claim_gate_hits`: OUTBOUND rows the pooled claim gate refused while the delivery tier was halted. This is the one signal the latch structurally cannot get from an enumeration -- its own argument, that counting doors is a claim nobody can verify, also means nobody can verify that every door is gated. A lane can only reach that gate if some path armed it unpaused while the halt held. Two qualifiers travel with the number and the runner property owns them: a small count at the moment of the halt is the documented window between setting the latch and pausing the lanes, and zero is not a clean bill because the counter is POOLED-only (the per_lane gate sits above the claim, and making it tick at the loop top would floor the signal with every running worker at halt time). TEST. The absence-of-a-spin window now patches `_WORKER_ERROR_BACKOFF_SECONDS` down and waits ten claim cycles rather than one. A window merely longer than one backoff bought a margin of one event. Measured with the gate removed: 9 claim-gate hits inside the window, in half the wall clock the un-patched single cycle needed. The /stats field is read through getattr-with-default, like the counters beside it, and test_api's stub-the-runner test now carries it with its own distinct value. That test exists because every field there was once replaceable with a literal 0 without a reader noticing; direct attribute access also 500'd the whole stats read against its SimpleNamespace stub. `_until_outbound_rows` folded into `_until_outbound_row` as a `count` kwarg, and the second outbound moved from a `_two_outbound_registry` builder that wrote `reg.handlers["h"]` directly -- bypassing `add_handler`'s duplicate guard and its `handler_accepts` bookkeeping -- to `added_outdir` / `added_auto_start` knobs on `_e2e_registry`. Two same-file restatements removed from the reconcile memo and the ADR.
…committed row
Found by running the wide selection, not by review: one failure in six, and it
surfaced three assertions away from its cause as the control arm's "delivered but
never finalized".
Phase 1 of the door-six test stopped its seeding runner as soon as the first
lane's file appeared. The file existing and the row being DONE are different
instants -- the connector writes, and the store write marking the row resolved
commits after it. Stopping the runner inside that gap left the row INFLIGHT,
phase 2's reset_stale_inflight reverted it to PENDING, and the message could then
never reach PROCESSED: that lane is paused by the halt for the rest of the test,
so nothing was ever going to deliver it again.
The wait is now on the STORE, through a new `_until_delivery_status` helper, and
the test also asserts its own premise after the stop -- exactly one row, PENDING,
for the lane phase 2 has never heard of.
Measured 8/8 green after the fix, having reproduced the failure at 1/6 before it.
The door-six assertions are unchanged and still RED in both claim modes with the
gate removed: "assert 'OB_TEST_ADDED' in {'OB_TEST'}" on _outbound_paused.
|
RESOLVING THE ONE OPEN QUESTION IN THIS PR BODY, from the Manager who dispatched it. The body records that the two THE So a lane can be armed without ever passing through SO NO ADR AMENDMENT IS NEEDED and the Builder was right not to take it on its own judgment. The altitude pass argument -- that one placement closes three arming sites at once -- is true about the sites it can see and wrong about the ones it cannot. Recording it here so the next reader does not reopen it from the PR body alone. TWO THINGS THIS BUILDER DID BETTER THAN THE BRIEF I GAVE IT, worth naming because both are the kind of deviation a reviewer should want:
IT ALSO FOUND A 1-IN-6 FLAKE IN ITS OWN NEW TEST, by wide selection rather than by review: phase 1 waited for a FILE where it needed a committed ROW, and stopping in that gap left the row INFLIGHT for Final state: 1366 passed, 131 skipped, zero failures; ruff, ruff format and mypy strict clean; glyph scan zero over 435 added lines against controls of 164 and 236. |
What this changes
The engine halts delivery when it cannot write its application log, because CLAUDE.md section 1
says every message a connection puts out is counted and logged. The inbound tier enforces that
with a latch (
_log_halted) read at each worker's loop top. The delivery tier had no state ofits own: the halt took a lane down by pausing it, and a pause lives in
_outbound_paused— which_teardown_bodydeliberately clears, and which no consumer can tell apart from an operator pause.So the rule could only be re-asserted at each door into resuming delivery, and four were found
one at a time, each only after the previous fix shipped:
start_outbound,restart_outbound, astop()+start()teardown, and a reload's_unpark_outbound_lane. Four gates are a completenessclaim nobody can verify (CLAUDE.md section 11, SDS-3.6).
This adds
_delivery_halted, read at the claim gate in both claim modes. Doors are unbounded;claim modes are two and closed.
per_lane_delivery_workerloop top, above the pause gate and before the claimreturn— no row left INFLIGHT, the same terminal state_router_worker's gate leavespooled_dispatch_delivery, the first runner-owned code a claimed row reachesreschedule_claimedthe head +RETRY— parks the laneThe four door gates are KEPT. They fail fast and page with the reason, which the claim gate
cannot. The latch is defence in depth behind them, never the recovery path. Removing working gates
in the same change would make a regression unattributable.
ADR: 0189,
allocated with
alloc.ps1and indexed in the same commit. No BACKLOG number was allocated for thischange;
#122is cited as the existing item this subsystem already belongs to.The fifth-door test, measured red first
tests/test_log_write_guard.py::test_an_unguarded_start_cannot_deliver_while_the_halt_is_latcheddeliberately uses none of the four doors. It drives
_start_outbound_unsafedirectly as astand-in for the fifth door — whatever it turns out to be — and asserts that with the halt latched
and both sinks genuinely dead, no bytes reach the output directory.
RED on
main, both claim modes:GREEN with the latch:
2 passed, 57 deselected in 4.51s.The red was measured with the three
_delivery_haltedassertions temporarily stripped, so thefailure is the bytes assertion rather than an
AttributeErroron a symbolmaindoes not have.Which assertions are real measurements, and which are harness properties
Real measurements (the engine, through a real File connector draining a real store):
list(outdir.iterdir()) == []while halted — the load-bearing one. This is the assertion thatwent red on
main.len(await store.outbox_for(message_id)) == 1— the row is retained PENDING, not dead-letteredand not stranded INFLIGHT by the refusal.
await _until(lambda: any(outdir.iterdir()))and_until_processed(...)in the control arm — thesame rig, the same queued row, delivered once the disk is repaired.
Harness properties (they check the rig reached the state under test, and would not by themselves
have caught the defect):
await _until(lambda: runner._log_write_stopped)— the halt fired.runner._delivery_haltedbefore and after the unguarded start — a read-back of the flag thischange adds. Kept only to attribute the refusal to the latch rather than to some other reason the
lane might be down.
assert list(outdir.iterdir()) == []before the sinks are killed — proves the lane isgenuinely loaded, so "no file was written" later cannot pass for the wrong reason (an empty
outbound stage).
Operator-surface assertions (real reads of the shipped API, but state and strings rather than
bytes):
outbound_status(...) == "log_halted",not outbound_running(...),outbound_status(...) == "running"after recovery, and the resend-409 wording — that last one withits own negative control, since a merely operator-paused lane must still get the start-it
instruction or the new branch would be a rewrite rather than a discrimination.
Claim modes
Both, via the file's existing
CLAIM_MODES = ["pooled", "per_lane"]. Neither is vacuous here,and the reason is structural rather than inherited: the gate lives in a different function in
each mode (
_delivery_workervs_dispatch_delivery) with a different refusal shape (pre-claimreturn vs post-claim reschedule-and-park), so a single-mode test would leave one of the two
implementations entirely unexercised. Both were measured red on
mainand green with the latch.What an operator now sees that differs
log_halted, notstopped. The halt takes a lane down by pausing it, anda pause is what an operator does — so
/connectionswas telling an operator the lane was waitingfor them to press start, when what it was waiting for was a writable disk. It renders in the BAD
colour, not the muted grey of "stopped".
log_haltedrather thanrunning. Such a lane isnot in
_outbound_paused, so it read as actively delivering while the claim gate refused everyone of its rows.
outbound_runningis false for every lane while the latch holds, so/stats' running/stoppedsplit counts halted lanes as not running rather than as running.
own fix: both resend guards read
outbound_runningand 409 with "start it before resending" —correct for an operator-paused lane, wrong for a halted engine, where the very door the message
points at is refused until the log works. That is the compensating-control-on-a-false-premise
shape (SDS-3.7), so the two identical guards now route through one helper that names the cause.
log_haltedjoinsstopping/stoppedinthe standalone-row loop, because the halt takes down every lane at once and a console listing
only the trafficked ones would silently omit the rest.
not_deployed,failedandfilteredstill outranklog_haltedon the display ladder — each is a fact about that one connection, fixable on thatrow, while the halt is process-wide and already has its own
log_write_failedpage.outbound_quiescedstill answers off the pause set, so purge-eligibility is untouched and ahalted-and-quiesced lane stays purgeable.
Surfaces updated: the
ConnectionRow.statusandGraphNode.statusvocabularies, the/connectionsstandalone-row loop, both resend guards,
.status-log_halted/.gnode-log_haltedin the webconsole CSS (the console derives both its class and its label from the raw string, matching how
not_deployedis handled),docs/SERVICE.mdanddocs/CONNECTIONS.md.Design note: the latch is DERIVED, not a second flag
_delivery_haltedis a property returning_log_write_stoppedrather than a parallel boolean."This process cannot log and has fail-closed" is one load-bearing fact, and
_log_write_stoppedalready states it: both halt sites set it, only
_log_recovery_okclears it, and that path re-teststhe sinks by writing to them. A second bool set and cleared at the same moments would be state that
must agree with this one, with nothing checking that it does — SDS-3.5. It inherits the property
that matters:
_teardown_bodydoes not clear it, so the latch survives a teardown, which is themechanism behind doors 3 and 4.
A
set[str]of lane names — mirroring_log_halted's per-inbound shape — was rejected onmeasurement, not taste.
_stop_all_for_log_failurepauses only the lanes not already paused, soa set built from it would omit exactly the engine-parked lane door 4 is about; and no set covers a
lane built after the halt. Per-lane recovery is meaningless here anyway: the broken sink is
process-global, so the moment one lane's door re-validates it, no lane's halt reason survives.
The full list of alternatives considered and rejected is in ADR 0189, Options considered.
Checks run
ruff check .ruff format --check .mypy messagefoundry(strict)pytest tests/test_log_write_guard.pypytest tests/ -k "outbound or log_write or connections"pytest tests/test_api.py tests/test_auto_start.py tests/test_not_deployed.py tests/test_wiring_engine.pypytest packaging/messagefoundry-webconsole/testsmessagefoundry adr-analyzeU+2192in the ADR's Acceptance Criteria — the character the ADR template mandates andadr-analyzeparses. Positive control ondocs/FEATURE-MAP.mdthrough the same instrument: 179 occurrences found, so the 4 is not a false zero/simplifyLegs a reader still has to check after this process exits. The full
pytestsuite was not run inone pass — the selections above cover the changed surfaces and the web console, but not the tree. And
every leg that only exists on a hosted runner (
windows-service-smokeand the other NSSM/servicelegs) is invisible from here.
/simplifyfindingsRun on the first commit's diff. Reuse, simplification and efficiency all came back clean on the
questions that matter here; the altitude pass had not returned when this turn ended, and its question
— whether the two-site gate is the right depth, or whether anything belongs inside
StageDispatcher— is answered explicitly in ADR 0189, Options considered 5, so a reviewer can check that reasoning
directly rather than take it on trust.
no new stored state, nothing to fall out of sync), and two sites is the structural minimum: pooled's
claim happens inside
stage_dispatcher.py, a moduleRegistryRunnerdoes not own and which has nogeneric pre-claim hook, while pushing the check down into
_process_delivery_item/_process_delivery_batchwould still be two sites and would stop preventing the per_lane claim.(commit
91c18a94b)._log_haltedidiom already in the three internalworkers; the pooled gate uses
reschedule_claimed, the documented primitive for exactly thismachinery-fault-head case.
.status-log_halted/.gnode-log_haltedshould join the existingvar(--bad)comma-lists rather than stand alone..status-not_deployedis already standalonewith its own explanatory comment, and the new rules follow that precedent so they can carry the
reason they read BAD rather than grey.
halted case it is a net win, since it skips the claim I/O entirely.
reschedule_claimedis asingle guarded UPDATE on an indexed predicate, awaited on an async cursor, bounded to a 1 s cadence
by
_WORKER_ERROR_BACKOFF_SECONDS.The third commit (the resend 409) landed after that pass; it is itself a de-duplication of two
byte-identical guards into one helper, so it moves in the direction
/simplifyasks for rather thanagainst it.
Door six, and two sentences that understated what they measured
Added 2026-09-15, on top of everything above. Three tasks and one optional extra.
The earlier headline was WRONG, and the banner at the top of this body still repeats it
The update banner says "a log halt spins every pooled lane once a second for its whole duration".
That is falsified.
_stop_all_for_log_failurepauses every OWNED outbound through_stop_outbound_unsafe->dispatcher.pause_lane, and a PAUSED lane is never claimed. On theordinary halt zero lanes spin. Every structural link of that finding held; only its headline did
not. Read the banner with this paragraph beside it.
But one spin path was reachable, and it is door six.
1. Door six is gated
_reconcile_outboundsbrought a newly ADDED outbound (deployed=True,auto_start=True) past bothconfig gates with no halt check. Such a lane is in neither
_gate_parkednor_outbound_paused, sothe un-park gate never asked about it, the branches below built its connector and armed its lane, and
the reload's own
notify_workseeded it READY. No bytes shipped -- the claim gate refuses every row,which is what the latch is for -- but the lane reached that gate once per
_WORKER_ERROR_BACKOFF_SECONDSfor the halt's whole duration. ADR 0189 option 4's rejection namesthis case in so many words ("a lane BUILT AFTER the halt (a reload adding an outbound), which is
door six") and did not close it.
It is closed by widening door 4's existing gate, not by adding a seventh:
The first commit used a raw
_delivery_haltedread beside the existing gate. The/simplifyaltitudepass found why that was wrong, and three independent reasons agree:
_outbound_start_permittedis a PROBE, not a predicate -- it re-validates the sinks by WRITINGto them, it can CLEAR the latch, and a refusal PAGES. Its docstring requires every caller to ask at
most once per operator action and to memoise across a loop. The raw read was documented as
deliberately un-memoised, which argued against the helper's own contract.
happened to be
_gate_parked. Fix the disk and reload a graph that only ADDS an outbound: with aparked lane present the probe runs, the latch clears and the lane comes up; with none, the latch
reads stale-True and the lane is stopped. Same repair, same reload, two outcomes.
_stop_all_for_log_failure's own docstring says "Fix the disk, reload, and the backlog drains."all is that a refusal there pages with a cause.
_stop_outbound_unsafe, never_park_outbound_lane, and that is the load-bearing detail: a parkwrites
_gate_parked, which this very gate LIFTS the moment a probe succeeds, so a park would re-openthe hole one reload later off a marker the method wrote itself. It goes down through
_pause_delivery_lanes-- the documented single owner of how the halt takes a delivery lane down --which routes through
_stop_outbound_unsafeand adds a per-laneexcept, so a lane that refuses topause is logged and the rest of the reload still reconciles.
The caller's
name not in self._outbound_pausedfilter is what keeps the two doors apart, and it isthe same filter
_stop_all_for_log_failureuses. Every_gate_parkedlane is already paused, so ittakes the no-write path and its marker survives for a later reload -- byte-identical to before.
Re-stopping an already-paused lane would CLEAR its quiescence Event, flipping a drained lane's status
back from
stoppedtostoppingand withdrawing its purge-eligibility for a reload that changednothing.
Measured RED, both claim modes
tests/test_log_write_guard.py::test_a_reload_that_adds_an_outbound_into_a_dead_log_lands_it_paused,on ADR 0189's own rig. Phase 1 seeds a genuine PENDING row for a connection through an engine-parked
lane; phase 2 brings a second runner up on the same store without that connection, halts it, then
reloads the connection in. A fresh runner over the same store is the only way to get a real queued row
for a lane this process has never heard of, which is the definition of the door.
RED with the gate removed, both modes:
GREEN with it:
2 passed, 59 deselected.WHAT IS RED IS THE LANE STATE, NOT THE BYTES, and the test says so. The claim gate this PR already
adds refuses every row, so the output directory stays empty with or without the new gate -- the byte
assertion is a regression guard on the latch, not the discriminator for this door. The spin is the
other half, and it is measured separately: with the lane-state assertions neutralised, the counter and
row-untouched assertions fail in pooled only (
assert 9 == 0), and pass in per_lane, where theworker returns above the claim and there is no spin to find.
9, not1, because the window patches_WORKER_ERROR_BACKOFF_SECONDSdown and waits ten claimcycles. A window merely longer than one backoff buys a margin of one event, and a test that
discriminates by one event is a scheduling hiccup from proving nothing. Ten cycles is a real margin
and finishes in half the wall clock a single un-patched cycle needed.
2. ADR 0189's risk sentence understated the cost
It called a spin cycle "a bounded store round-trip per lane per second". Verified against
store.py, it is two write transactions and a payload decrypt:self._lock?claim_fifo_heads: SELECT, UPDATE toinflightwithattempts+1, re-SELECT,delivered_keysprobe_outbox_item_from_row: decrypts the payload where at-rest encryption is on (plus ashared_bodyread for abody_refrow)reschedule_claimed: UPDATEThat lock is the one
enqueue_ingressand every stage handoff serialize behind -- both reach itthrough
_writer_txn, group-committer enabled or not -- so the cost is not confined to a tier alreadyrefusing to work.
3.
LaneItemResult's docstring promised the wrong producerIt said RETRY means the body "has already
mark_failed'd the head and returns that additivenext_attempt_at"._dispatch_delivery's halt gate returns areschedule_claimeddeadline instead.The sentence is fixed, not the type, because the widening is in the safe direction:
mark_failedspends a retry and can eventually dead-letter,
reschedule_claimeddoes neither, and the dispatcherreads
retry_untilonly as a park deadline. It now names both producers and says which one spends aretry. The
LaneResultKind.RETRYenum comment repeated the same wrong half and now links to thedocstring instead of restating it.
4. (optional extra)
/statshalted_claim_gate_hitsOUTBOUND rows the pooled claim gate refused while halted. This is the one signal the latch
structurally cannot get from an enumeration: its own argument -- that counting doors is a claim
nobody can verify -- also means nobody can verify that every door is now gated. A lane can only reach
that gate if some path armed it unpaused while the halt held.
Two qualifiers travel with the number, and the runner property owns them rather than the API field:
_stop_all_for_log_failuretakes the reload lock and pauses the lanes, so a claim already in flightacross that window lands here legitimately. What names a door is a count that keeps climbing.
claim_lock_timeouts. The per_lane gateis at the worker's loop TOP, above the claim. It deliberately does not tick there: every running
per_lane worker passes that gate once at halt time, and the resulting floor would bury the signal.
No
lane_stuckalert was added. The operator is already pagedlog_write_failedplusconnection_stoppedper connection, and a second page restates a load-bearing fact (CLAUDE.mdsection 11, SDS-3.5).
A flake the wide selection found, and its cause
Worth recording because it failed three assertions away from its cause, and because it is the
shape the file's own
_until_processeddocstring already warns about.Phase 1 of the door-six test stopped its seeding runner as soon as the first lane's file appeared.
A written file and a resolved row are different instants: the connector writes, and the store write
marking the row DONE commits after it. Stopping inside that gap left the row INFLIGHT, phase 2's
reset_stale_inflightreverted it to PENDING, and the message could then never reach PROCESSED --that lane is paused by the halt for the rest of the test, so nothing was ever going to deliver it
again. It surfaced as the control arm's
delivered but never finalized.Measured 1 failure in 6 before the fix, 8/8 green after it. The wait is now on the store, through
a
_until_delivery_statushelper, and the test asserts its own premise after the stop: exactly onerow, PENDING, for the lane phase 2 has never heard of. The door-six assertions are unchanged and
still RED in both claim modes with the gate removed.
It was found by running the wide selection, not by review, and not by the file's own suite --
which passed 61/61 every time. That is the argument for the wider
-krather than the narrow one.Explicitly NOT done
is_infra_faultis not decoupled from eventual STOP._apply_retry's streak semantics are ADR0070's own declared sharp edge, its blast radius is ingress/routed/response/outbound, and the lane
population that would benefit is zero on the ordinary path.
_apply_retry,_infra_backoffandinfra_fault_policyare untouched, and ADR 0070 is not amended -- its scope is the T17 machinerypath and it says so.
argued the spin belongs in
_pooled_lane_provider-- filter its OUTBOUND branch on_delivery_halted, and the three arming sites (start's seed loop,notify_work's union,_run_sweep_once's owned set) all close at once, where this gate closes one path. Its case is thatADR 0189 option 5 rejected that as "a partial gate that LOOKS total", which is a claim about a
correctness gate misleading a reader, and a cost reducer sitting behind an acknowledged
correctness backstop cannot mislead that way. I did not do it: it reverses an accepted ADR's
recorded decision, it needs an amendment rather than a Builder's judgment, and it touches the
dispatcher's arming for every stage. Note the reviewer agrees the reconcile gate is still needed
either way -- a provider filter would not stop
build_destinationwarming an MLLP socket, DB poolor SMART token for a lane that cannot deliver.
/simplifypasses disagreed with each other about the merged gate, and thedisagreement is recorded rather than hidden. Simplification derived the same merge and rejected it,
on the grounds that a reload-while-halted would then probe and page where it is silent today, and
that it trades a live read for a memo. Altitude argued the silence and the asymmetry are the
defect. I went with altitude, because the memo is
_outbound_start_permitted's own documentedcontract for a looping caller and the silence contradicts the ADR's stated reason for keeping door
gates -- but a reviewer who weighs the extra probe higher has a real argument and it is above.
Checks run on this change
ruff check messagefoundry testsruff format --check messagefoundry testsmypy messagefoundry(strict)pytest tests/test_log_write_guard.pypytest tests/test_log_write_guard.pyx8 (the new test)pytest tests -k "outbound or log_write or reconcile or connections or stats or dispatcher or api"docs/FEATURE-MAP.md164,docs/CONNECTIONS.md236 -- so the zero is not a false one. The 4U+00B7in the ADR's Related line predate this change/simplifyLegs a reader still has to check after this process exits. The full tree was not run in one pass,
and every leg that only exists on a hosted runner (
windows-service-smoke, the other NSSM/servicelegs) is invisible from here. The web console suite was not re-run for this change; it touches no
console surface, but
halted_claim_gate_hitsis a newStatsResponsefield.