Give the draft pile an automatic consumer, so captured knowledge reaches the corpus (#765) - #767
Merged
Conversation
…hes the corpus (#765) 113 articles sat in draft, growing about 11 a night, with zero automatic consumers. Under the governing constraint - there is no human approver and never will be - that is not a safety gate, it is total loss: an agent never sees a draft, so a held article is indistinguishable from one that was never captured. This is the last item of #765 and the blocker for claude-config#221. TWO PREMISES IN THE SPEC DID NOT SURVIVE MEASUREMENT, and the design follows the measurement. First, the spec says to dispose of drafts using the near-neighbours already stamped in metadata.proposal_novelty. ZERO of the 113 carry that key - their metadata is empty. They are direct orchestrator deposits that never went through propose_article's novelty branch, so a consumer written to the spec would read a nil and do nothing on every one of them. This consumer ASSESSES, through the existing ProposalGate rather than a second novelty implementation. Second, the spec says a near-duplicate may be merged or linked and the twin then archived, because archive is a soft delete and therefore reversible. Archive is TERMINAL: Article's valid transitions carry no entry out of archived and there is no unarchive function, which is why the consolidation pass retracts with unpublish and never archive. An unattended writer may not take a one-way door. So the disposition is publish-and-link. Novel drafts publish. Near-duplicates publish AND carry a relates_to edge to their nearest published neighbour with the cosine score, handing the redundancy to the potential_conflict promoter and judge - a pipeline that is reversible end to end and judged 226 of 226 pairs on 2026-08-27. Nothing is archived, no body is discarded, and unpublish_article is the exact inverse of the publish this takes. AN ASSESSMENT THAT FAILS PUBLISHES NOTHING. A provider outage, a shed heavy read or a verdict outside the behaviour's vocabulary leaves the draft held and counts it. Defaulting an unrecognised verdict to novel would be the same unassessed publish the unknown branch already refuses. BOUNDS, in the worker that died of getting this wrong twice. The per-item cost is an embedding call plus a vector read - only 1 of the 113 drafts had an embedding, because publishing enqueues one and being created as a draft does not - so the bound is a wall clock and the budget is carved OUT of the existing job timeout rather than added beside it. timeout/1 does not move. The judge's ceiling falls from 18 to 16 minutes, which is what this step may spend, and the arithmetic stays in the one clamp that already does it. The drain cap sits above the producer rate on purpose: about 11 drafts arrive a night and the cap is 25, so the 113 backlog clears inside a week instead of holding at whatever level it reached. It resolves from a SystemConfig row first, so 0 pauses the drain fleet-wide without a deploy. Every night reports what it was OFFERED and whether the clock truncated it, in the log line and the audit event, because a truncated night and a night with nothing to do are otherwise the same number. Mutation-verified: making a gate-unavailable verdict publish reddens the failed-assessment test, and defaulting an off-contract verdict to novel reddens its own.
…at were held on purpose draft_scope/1 treated every :draft row as an abandoned capture. Three classes are staged deliberately and are now excluded: - a RETRACTION. The audit guard was narrowed to worker:consolidation, so unpublish_article/3 and bulk_unpublish/3 - the role: user lever - were undone unattended within 24 hours. The NOT EXISTS now spares an article.unpublished row written by ANY actor. - a MERGE. A draft carrying metadata.merged_from is never offered. The KB-content carve-out that lets an agent-role key record a merge verdict rests on that synthesised draft never being auto-published. - a FRESH HOLD. A draft is offered only once it is 48h old, so the advertised draft:true and ingestion publish:false staging opt-ins stay observable. Also in this round: - the candidate window reserves a fifth of the cap for the NEWEST drafts. With no per-draft attempt counter, a purely oldest-first window stalls at zero forever once a cap's worth of permanently unconsumable drafts reaches the head. - a keyless tenant is stopped BEFORE the candidate query, on has_embedding_key?/1 rather than the Anthropic has_api_key?/1 the warning was reading. Offering wrote one llm.blocked_no_api_key audit row per draft per night, forever, for a known-zero outcome. - the annotation edge is scoped to the auto-linker's own neighbour scope (same project or global, never private/owner), so this step adds no edge class the graph has never held and cannot flag a cross-project or private pair into curated suppression. - an off-contract verdict is named instead of tagged as the literal string unknown, which is what ExitTag.tag/1 returns for a map. - the sizing prose now states the NET drain: 30 offered against ~11 arriving clears 113 in about six nights, not four. Corrected in all three sites. - the moduledoc and the call-site ordering comment no longer claim the promoter and judge handle the redundancy reversibly. The judge's dismiss is terminal and retires neither article, consolidation's duplicate_capture reaches only colliding titles and idempotency keys, and a flagged pair suppresses both its articles from curated answers until it is judged.
… the clock can reach Six root causes from the round-2 review, all in the draft consumer. Retraction guard was only as durable as the audit partitions. The column is stamped by consolidation alone and only since 20260818055453, the audit_log is DROPped past audit_retention_days, and a user+ PATCH to status draft writes article.updated and no unpublished row at all - so a human retraction came back unattended after ~90 days, or immediately by that PATCH. A third record closes both: a draft carrying an embedding was PUBLISHED once (embeddings are enqueued at published only, ingestion only under publish true) and that row outlives every partition. It also splits the queue cleanly - the weekly sweep takes the embedded drafts, this step the unembedded captures. Merge guard hung on metadata.merged_from, which an agent-role PATCH casts and whole-map-replaces - the previous_title and lifecycle_entered_at lesson, third instance. Excluded now by the executor's own conflict_resolutions row, which no caller can cast. The reserved NEWEST slice was appended after the backlog, and the wall clock truncates the tail - so a slow unconsumable head discarded exactly the slice added to keep the drain positive. The two slices are interleaved now. linkable? judged the head neighbour only, so an out-of-scope rank 1 threw away an in-scope rank 2 the auto-linker links anyway; the whole scored list is walked. A near-duplicate published with no annotation is counted as published_link_failed rather than hiding inside published, and the warning names the real cause instead of blaming a malformed assessment. The keyless-tenant warning fired nightly for tenants with no drafts, asserting drafts were held when none were; it speaks only when something is actually held. Comments corrected rather than qualified: the visibility half of linkable? is STRICTER than the auto-linker (which passes no visibility scope at all), not a mirror of it, and the moduledoc says where a merge draft's exit now is.
mkreyman
added a commit
that referenced
this pull request
Aug 27, 2026
… could only reach what it meant to spare (#765) (#768) Six confirmed findings from #767's enhanced review. They sat outside that diff's write scope but every one is a consequence of it, so they are fixed here. AN UNASSESSED DRAFT COULD PUBLISH AS NOVEL. The consumer refuses to publish when assessment fails, but one layer down ProposalGate could not tell "no neighbours found" from "the neighbour search was unavailable" - a shed or degraded vector read returns an empty list, which scores as maximally novel. The guard above was therefore defeated by the layer below it. Every assessment now carries comparison: :complete or :unavailable, and the consumer treats :unavailable as NOT ASSESSED, exactly as it already treats :unknown. The CREATE path is deliberately unchanged and still falls open: a write must not 500 because a novelty read shed. The two callers want opposite things from the same failure, which is why the fix is to distinguish rather than to change the default. THE SWEEP WORKER COULD ONLY REACH THE CLASS IT DOCUMENTED IT MUST SPARE. This is the finding worth reading. DraftDuplicateSweepWorker archives drafts whose nearest published neighbour clears a high threshold, and it sweeps only drafts that HAVE an embedding. Embeddings are enqueued at status published only, so every row it could ever archive was published once and then retracted - precisely the class its own "what it must NOT sweep" section says to protect. It spared only the half consolidation wrote, leaving a human retraction reachable. It is now PARKED rather than rewritten. Nothing was swapped in for archive because a draft is already the bottom of the reversible lifecycle - published to draft and back is the pair, archived is the exit from it - so there is no reversible retirement to move to. Parking changes the AUTHORITY instead of the action: a human may take a terminal act, and an operator unparking it is a human asking for exactly that. The design below it is preserved intact for a revive, including the measurement behind the threshold. A DELIBERATELY STAGED DRAFT NOW HAS A VETO, not a 48-hour floor. staged_draft_at is a column stamped programmatically and absent from every cast list, so an ordinary PATCH cannot erase it - the same lesson previous_title and lifecycle_entered_at each paid for. Plus a covering index for the candidate scan, which was reading the tenant's articles on the three-connection admin pool that every authenticated request also checks out of; and the pre-marker retraction residual, verified against the code and documented at the guard site as the deliberate bounded cost of not stranding every pre-marker draft permanently. The CHANGELOG entry that the previous two changes correctly omitted is warranted here, and it names its categories: two migrations, and OBAN_UNPARK_CRONS as the lever that revives the parked worker.
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last item of loopctl#765, and the blocker for mkreyman/claude-config#221.
The leak
113 articles sat in
status: :draft, growing ~11/night, with zero automatic consumers. Under the governing constraint (owner decision, KB837daaa0) that is not a safety gate — it is total loss. An agent never sees a draft, so a held article is indistinguishable from one that was never captured. Holding is the harm; publishing is recoverable.Two of the spec's premises did not survive measurement
1. There is no
proposal_noveltystamp to read. #765 says to dispose of drafts using the near-neighbours "already stamped inmetadata.proposal_novelty". Zero of the 113 carry that key — their metadata is empty. They are direct orchestrator deposits (article.created) that never went throughpropose_article/3's novelty branch. A consumer written to the spec would read a nil and do nothing on all 113. This one assesses, through the existingProposalGaterather than a second novelty implementation.2. Archive is not reversible. #765 says a near-duplicate twin may be archived because "archive is soft-delete, therefore reversible, therefore allowed".
Article's@valid_transitionscarries no{:archived, _}entry and there is no unarchive function —:archivedis terminal, reachable back only by auser-role PATCH. That is why the consolidation pass retracts withunpublishand neverarchive(#608), and CLAUDE.md says so outright (#605/#606). An unattended writer may not take a one-way door.The disposition: publish-and-link
:novel:low_novelty/:duplicaterelates_toedge carrying the cosine score:unknown, or a verdict outside the contractThe near-duplicate case hands redundancy to the existing
potential_conflictpromoter + judge — reversible end to end, and demonstrably working (226/226 pairs judged on 2026-08-27). Nothing is archived, no body is discarded, andunpublish_article/3is the exact inverse of thepublish_article/3this takes, both writing audit events.A failed assessment publishes nothing. A provider outage, a shed heavy read, or an off-contract verdict leaves the draft held. Defaulting an unrecognised verdict to
:novelwould be the same unassessed publish the:unknownbranch already refuses — mutation-verified in both directions.Bounds — third time in this worker, same discipline
Only 1 of the 113 drafts had an embedding (publishing enqueues one; being created as a draft does not), so the per-item cost is an embedding call plus a vector read. Therefore a wall clock, and the budget is carved out of the existing job timeout rather than added beside it:
timeout/1does not move (Lifeline's 30-minrescue_afteris the real ceiling)@job_reserve_msbecomes@prelude_reserve_ms + @retitle_reserve_ms + @draft_reserve_ms; the judge's ceiling falls 18 → 16 min, which is exactly what this step may spendSystemConfigrow first, so0pauses the drain fleet-wide without a deployVerification
7,942 tests green via the pre-commit hook (credo --strict, dialyzer included). Guards mutation-verified independently of the implementer's report:
:unknownpublishes instead of holding:novelNo CHANGELOG entry: by that file's own rule (five categories, "ONLY") a behavioural change routes to
git log --first-parent.