Engine improvements: fix loops, votes, packet pins, gates, and registry - #38
Open
erikreinert wants to merge 123 commits into
Open
erikreinert wants to merge 123 commits into
erikreinert wants to merge 123 commits into
Conversation
erikreinert
force-pushed
the
feature/engine-improvements
branch
2 times, most recently
from
September 1, 2026 23:39
583fae1 to
30eb937
Compare
56 commits hardening the run engine and the CLI around it. Workflow grammar — `when` predicates gain `or` and `contains_any`; steps can declare a severity floor and stall limit; cross-issue linked input form; `max_fix_loops` counting documented. Fix loops — entry scoped to per-trigger clusters; loop parks when the routing verdict repeats or the fix commit didn't move; steering notes carry into the next round; run-scoped batch gate-override covers repeated identical failures (DKT-546). Votes — approved votes route on concerned casts; vote records exposed; retry refused once a proposal has decided; silent seats named with their path; weighted score separated from ballot count; vote-step spend counted against the run budget. Claims and budget — claims can declare a variant-scaled cost; a forced reap no longer spends the attempt budget; routing metadata recorded at claim; run report pairs requested and resolved tier; aggregate builtin gets a routing floor; warning when usage is backfilled against an unclaimed step. Packet pins — pin only the closure a run's workflows reach; check the pin set is closed; validate a packet before taking a lease; repin drops gone files and adds newly required ones; registered workflow source drift detected on disk and shown in run reports. Gates — gate processes get the step's base commit; stub gate results marked with a reason and tracking issue; override-pass refused when it would skip a gate; failed gate named when a step parks; advisory pre-gate verdicts marked; gates named as make targets. Registry and scope — cross-project workflow/schema registration and a drift audit; orphaned registrations surfaced; `--deprecated` hides retired versions and `show NAME` skips them (DKT-616); a widened scope can reach an active run, with a warning when it can't; unresolvable stale targets flagged and waivable. CLI fixes — `events follow --tail` starts at the newest N; `events list` finds a run in any project; `next --run` returns the full ready set; dispatch close does backfill and verify in one step; trust changes refuse a missing actor/cwd; loop redirects no longer bind an unrelated step's output; operator pause survives a step record; in-repo docket skill copy retired.
erikreinert
force-pushed
the
feature/engine-improvements
branch
from
September 1, 2026 23:47
30eb937 to
10b6f6a
Compare
…ainst built dist Removes the standalone test/qa jobs — go build/test already run inside build-shell/build via the vorpal pipeline, and qa.sh now runs against the actual docket-dist-* artifact from the build job instead of a throwaway `go build` binary, so e2e exercises what actually ships. repo-gates is renamed to gates to match its new position as the first, fast-fail job; secret-scan.sh's comment is updated to match.
…breaks - a tree comparison on a moving branch tip read any later commit on the same paths as a divergence, so clean cherry-pick integrations warned "integration diverged" on every review row - a disproved ancestry now asks first whether HEAD carries the target's patch, via git cherry refined by zero-context patch-id comparison, including a squashed integration's combined diff - the tree probe remains as a fallback and may only acquit; the advisory says "diverged" only when the patch probe measured the patch missing
…olve - step resolve --worktree, with override-pass or rerun-gates, recomputes the recorded issue.diff and target sha from the named checkout and records it as a new artifact superseding the step's previous one - no resolution re-recorded the diff before, so after an out-of-band patch every downstream review packet still rendered the pre-patch commit and a full fix round ran to report nothing needed fixing - the re-pin is event-logged as issue-diff-repinned with both shas and rides the resolve output as issue_diff_repin - an empty diff over a recorded change is refused rather than pinned - extract computeIssueDiff from the routing stage so the re-pin and step record share one diff implementation - fold the resolve variants into ResolveStepWith on a ResolveOptions struct
- a type-filtered listing of eleven documents was 83KB of JSON because every row carried its full body - rows now carry id, type, status, title, author, timestamps, and body_bytes; bodies stay in doc show - --with-body restores the previous full-document output for bulk callers
- issue list, next, plan and board now drop description from each JSON row and carry description_bytes in its place - a filtered listing of a few dozen described issues ran to tens of kilobytes to answer a question that needs ids, titles and status - --with-body restores the previous full-row shape on all four verbs; issue show is unchanged
- the previous-round pass matched on emitted kind alone, so a re-synthesis received the prior round's whole raw review fanout beside the digest of it - a re-entry step now reads only its own name at the prior ordinal plus the standing set the loop body declares it acted on - one security-change round-2 synthesis carried a 263KB packet of four raw judge payloads it was also carrying as the reconciled aggregate
- a vote panel is seated from step show and only probes the context bundle when the row names a target - the bundle carried target_sha and target_worktree but the step view did not, so every panel read its own HEAD instead of the judged tree - the view now resolves the same pair in the same transaction, emitting both keys only when the step consumes issue.diff and a round record exists - steps with no target emit neither key rather than an empty or approximated value
(cherry picked from commit f01b09faa828b3229e413832023f7ffd261c726f)
…ng gap - derive the reserved-label set (selected by no workflow, declined by every workflow) from the live corpus instead of hand-listing it, so the sweep tracks corpus changes automatically - fail loudly, naming the label, when one is declined by only some workflows and selected by none (an orphaned exclusion) instead of silently reporting it as an unrouted gap - apply the non-vacuity floor to the routable vocabulary only, after reserved labels are set aside, across all three sweep tests
- ParseRelationType no longer accepts related_to/related-to; it backs the wire format and the workflow-spec vocabulary, both of which are meant to stay exactly the documented set - move the typo tolerance to a new parseRelationTypeArg helper shared by link add and link remove, where it only ever affects a human typing at a terminal - add tests pinning the parser, the JSON decode, and the CLI helper so a future alias can't widen the wire format silently again
- new run_notes table (schema v26): append-only, run-scoped, capped at 16 KiB per note, dead with the run - docket run note add RUN-N --text|--file records one; refuses an empty or oversized note, an unknown run, or a done/abandoned run - docket run note list RUN-N renders them back, human and --json - context assembly reads notes as a sixth source alongside the existing five, so step context/claim/render all carry them - the default packet template renders each note verbatim as its own section right after the request body, so an operator ruling made before dispatch reaches every step of the run instead of getting rediscovered per step - documented in the engine spec and reliability delta
…g it - a gap file's leading header block (Severity/Priority/Kind/Labels lines right after the title, ending at the first line that isn't one) now sets the filed issue's priority, kind, and labels instead of every gap landing at priority none, kind task, no labels - Severity maps to priority (blocker->critical, high->high, medium->medium, else none); an explicit Priority line wins either way it's ordered against Severity - unrecognized keys are skipped without ending the block, so a Home: line composes with the header; invalid values fall back to the default rather than refusing the whole issue - the body is stored verbatim, header included - documented in step complete/record's help text
A gate step that ends up skipped (routing resolved elsewhere, an on_fail=skip rejection, a false `when`, or a quorum miss) used to leave its downstream `after` successors ready to run anyway, since a skipped predecessor still counts as terminal. Successors had no way to say "only run me if that predecessor actually fired." Add a step-level `after_fired` list: naming a predecessor there means this step is skipped in the same transaction the moment that predecessor is skipped, cascading transitively through the graph. Every `after_fired` entry must also appear in `after`, so the step still waits for the predecessor to reach a terminal state before the skip (or the run) is decided.
- the claim wrote step_inputs but no read verb used it, so step context and step show re-resolved a handed-out step over the run's current artifacts and reported inputs and a target sha the worker never saw - a claimed step now reads back the bindings its claim recorded; a pending or never-claimed step still resolves live - a re-claim clears the last attempt's bindings before recording its own - the claim records its bindings before pre-gates run so a read that lands mid-claim finds them - step context --live keeps the current-state resolution reachable
…e with it - golangci-lint and staticcheck cache issues by package content but store the absolute source path and re-open it to find a suppressing nolint comment - a pre-gate reconstruction is deleted within the minute, so a later run over the same content replayed a stale entry, could not find the nolint, and failed a clean tree - point GOLANGCI_LINT_CACHE and STATICCHECK_CACHE at a scratch sibling of the reconstruction that is removed with it; durable trees keep their shared caches
…ays blocked - Expansion gated on every issue in every earlier topological level, so an unrelated phase-1 sibling left at todo by abandon-issue stalled a whole chain whose only real predecessor was done - An issue now expands once its own depends_on predecessors are done, matching how the planner reads the same graph - Activation reports every issue it left unexpanded with the predecessors holding it, on the summary line and as blocked_issues in the JSON envelope
…'s domain - a workflow may declare `[match] domain_paths`; it binds nothing and is read only by an activation lint - an issue whose whole scope lies inside another workflow's domain, lacking only that workflow's labels, is named in the activation report - exactly-one-match refuses zero and several matches but cannot see one wrong match; a mis-labelled TUI test issue bound the baseline pipeline and silently lost the UI gates - warns and never refuses, stays quiet on partial overlap, kind mismatch, or a firing unless_labels
- a gate needing an earlier step's artifact had to list the issue's steps and pick itself out by an instance-name convention, which broke silently - exports the step reference on both completion gates and pre-gates, and leaves the variable absent rather than STEP-0 when no step is known
…s reaped - failed_attempts/reaped_claims tally every ending a step ever had and go ambiguous once a history mixes both; a router needs only the last one - a session killed mid-wave had ten leases reaped and an on_failure policy escalated all ten as if they had failed - schema v27 adds steps.last_claim_end, stamped by the fail and reap paths, surfaced as prior_attempt_end on next, dispatch open, step show and list - next.go also makes IssueStepList return an empty slice instead of nil so a stepless issue serializes "steps": [] rather than null
- a nil slice marshals as "steps": null and forces every JSON consumer to guard against it
`docket trust probe [--run RUN-N] --json` runs every non-action trust-roster entry once against a throwaway worktree of clean HEAD, using the same resolve/spawn/timeout path gates use at record time, and removes the worktree unconditionally including on interrupt. `docket trust list --json=v2` items now carry `class: "gate"|"action"`, resolved from the workflow corpus's own `action = "<name>"` declarations, so a caller can tell the two apart without grepping workflow TOMLs itself. This replaces the gate-probe.js workflow script's equivalent, which spent one agent per gate and could not honor per-entry timeouts.
`docket next --run` executor rows now carry model/effort/variant resolved from the run's pinned policy.toml, and vote rows carry the same per voter. Resolution walks the attempt- and round-based escalation ladder, redirects around [security]-forbidden models via [escalation.fallback], and clamps to [security].ceiling on a sensitive row -- a direct port of wave.js's and tribunal.js's resolve()/resolveSeat(), verified against tests/wave-escalation-walk.test.sh's own table and structural assertions. A run with no pinned policy.toml leaves every row exactly as it was before this feature existed. policy.toml was already pinned per run (packet_closure.go's policyPinRef); this reads those bytes back instead of leaving routing to a second, independently-maintained parser downstream.
`dispatch close` now verifies, for every write-class step recorded in the run, that its own recorded commit is an ancestor of the shared branch's HEAD or patch-equivalent to a commit on it (a cherry-pick mints a new sha for identical content), refusing CONFLICT with the step, sha, and worktree otherwise. A cherry error counts as unintegrated rather than being assumed equivalent. `--skip-integration-check REASON` is the operator override, and the close event records integration: verified|skipped plus the checked shas either way. Reuses the same ancestor/patch-equivalence probes the stale-target advisory already shells out with, rather than a second implementation of `git cherry` plumbing.
- Replaces the six read-only probes a conductor cleared by hand before the first dispatch of an attach with one read-only verb - Every check runs; a missing --run or --source reads as SKIP and leaves the report unclean rather than silently checking fewer things - Straggler worktrees are reported but never move the clean bit
- The run report could not show how much of a review round agreed rather than merely occurred; `aggregate` has always written `members` on every cluster and nothing read it back. - Count unique versus corroborated clusters from `members` over the emitted payload plus any below-floor clusters `route_at` recorded, and read the input-artifact count off the producer step's `step_inputs`.
- DKT-2452's complementarity rollup could not attribute a cluster to the executor that produced a member: `members` is core's own reduced record with no per-member origin, and a finding's own id is unique only within its producing worker's payload, so core cannot resolve attribution by matching ids across judge artifacts. - Add `source_field`, a sixth optional `aggregate` param naming a property of each input element holding an array of opaque source labels. G3's existing pass-through already carries that array to the output untouched; the run report groups clusters by it and resolves each value to the producing step's declared executor hint. Absent the param, the report section is omitted, matching `route_at`'s convention. - Core reads the param only to learn which key to read — it never assumes a literal field name — the same discipline `route_at` keeps by taking its floor as a value rather than a field.
- step approve/reject/resolve/reap and run pause/resume/abandon were token-free, so under a harness any executor sharing the operator's checkout could approve the gate guarding its own commit or park the run - a first activation mints a run-scoped token (schema v29 stores only its hash) and returns it once; the seven verbs require it via DOCKET_TOKEN or stdin, refusing VALIDATION_ERROR when absent and AUTH_ERROR when wrong - `run conduct RUN-N` re-mints it for a session that does not hold it, retiring the standing token and recording a `conductor-seated` event - callers must hold the activation envelope's `conductor_token` or take the seat with `run conduct`; runs activated before the column existed stay open until conducted
- The unscoped form answers over every active run of the project, so one run's approval opened the gate for a caller working under a second run whose own gate was still undecided. - With --run only that run's gate answers; a run that does not exist is NOT_FOUND rather than a verdict, and a run that has ended denies. - Denials name the run searched. The unscoped form stays as the reading for callers with no run context, such as an operator session's hook.
- A finding may cite `artifact:ARTIFACT-N` or `gate:<name>`; `vote cast` resolves each against the run whose vote step or reap acknowledgment opened the proposal and refuses an unresolvable one by name before the cast records. - `run report` gains a findings section listing every recorded entry with its evidence and marking one that cites nothing `unsupported`; sealed-open ballots stay withheld. - An entry with no evidence keeps its bare-string wire form, so stored rows, exports, and vote-record packets are unchanged until a cast cites something.
- `docket report executors [--since RUN-N|DATE] [--all-projects]` groups, across every run in the window, fix-loop routings, override-pass rulings, reaps and forced reaps, and unique, corroborated, and held cluster contributions per executor hint, plus casts by verdict and the fate of the panels sat on per voter name. - Operator-facing only: the verb writes nothing, `next` never consults it, and nothing in it reaches a seat, since a track record fed back into a panel becomes an incentive to agree with the majority. - The ruling columns read the event log, because only the event says which reap a relay forced and which resolution an operator chose. - Extract parseVoteStepKey so the ledger and the bulk proposal reader parse a vote-step key the same way.
- `--active` is removed; the bare list now shows only runs that are not done or abandoned - `--all` restores the full list; scripts that read the unfiltered default must pass it
- add a normalized-output fingerprint to every recorded gate result row - require gate, exit, reason and fingerprint to agree before a grant covers - refuse a grant that predates the column rather than treating it as a wildcard - name the covered signature in both override ledger events (cherry picked from commit 0900d0fc941a40fdadc0e141163dc79e5c7e6753)
(cherry picked from commit 437199f67f4c5bc265e5e1b52070f29b85171778)
(cherry picked from commit 080ae88fc5ca394a4da2672dfb75aabe75507765)
(cherry picked from commit 728e81612f9db46547950bc91a7b9263a9a7f115)
(cherry picked from commit e5b8eae29a1ff70bf89fa86031b9f4d6098c37eb)
- add dispatches.extended_seq (v31) beside opened_seq - carry ReapHold on Extension and warn on it, as dispatch open does - test the expiry refusal, the reap, the seq oracle and the guard's no-counterpart branch - align the guard clause table, the event tables and the extend spec (cherry picked from commit 6bd92ad9dccf022d302b14e7d5b560a6809acc5e)
(cherry picked from commit a55fa38223925ecf7cb67a98ee0b5663b9c24a6d)
(cherry picked from commit fc94a05f9700ddbc8a883e922e0553cc6a891a14)
(cherry picked from commit 1dcfbfe3b15839bffd08703266a65e866f0b2aa3)
(cherry picked from commit 7abe9b0f93c78fc66f4c52a144a4ee6aec1ad101)
STEP-9464's implementation was out of scope to touch docs/tdd/**, so the schema-span tripwire correctly refused the v32 bump on its own.
- add a closed operator/standing-grant/conductor authority to resolve, approve, reject, pause and abandon - carry it on the emitted event, with a reference on a standing grant - tally the run report's resolutions by authority (cherry picked from commit eebc8833e1313c11ec6666ed5d38cb6d8c1c4878)
(cherry picked from commit 3511c2b240c989a8a898360ca0cfc451089e7618)
(cherry picked from commit 37ce304be4e2eacf7db19387b8f66635feb1dc90)
(cherry picked from commit b424b1b1c6c20c65986b8535f4653ab4b9a71343)
(cherry picked from commit ab277278529f3d95053737b0a91cbe27482cc37d)
(cherry picked from commit 7e0d81a45cb7902bcc7abe24862396e18e36db6c)
(cherry picked from commit 154da598c863d32f8e834e0d18c25fba6639cb1e)
(cherry picked from commit 037236da2ab300d8243aeac2491913289847c5b5)
…_class Both writers branched from the same base concurrently. DKT-1900 added a required ParkClass argument to SetStepRoutingTx; DKT-1901's triage.go called the pre-change signature. Added ParkClassTriageUndecided for the one genuine park site with no existing class fit, passed the empty string at the five non-waiting-human call sites where class is unused, and ratified the resulting v33 schema span (DKT-1900's park_class column was never doc-amended, same scope-boundary gap as DKT-1898 earlier).
- Adds an optional [sizes] policy.toml table mapping an issue label to the [variants] row a row should stand on instead of its [executors] entry's own variant. - Consulted in ResolveExecutor and ResolveSeat before the [security] ceiling clamp, so a size-derived variant is a new starting point for the escalation walk rather than an escalation hop, and still clamps to the ceiling on a sensitive row. - An issue with no size label, or a policy with no [sizes] table, resolves exactly as before.
- Adds model.Size (trivial, small, bounded, needs-design, unknown) as a closed enum on Issue, mirroring Priority's shape, with the v1 wire format emitting it only when declared (schema v34, additive migration, amendment recorded in docs/tdd/reliability-delta.md). - docket issue create|edit take --size, validated against the closed set; docket issue show/list/next/plan/board take --size as a filter, the same way --priority already works across those verbs. - Adds [match].sizes_any to the workflow grammar: a workflow can bind on an issue's declared size directly, the engine-side replacement for the small/trivial label convention as routing input. Evaluated as another inclusion clause alongside kind/labels_any/labels_all, before unless_labels. - Fixes activation's own narrow issue read (issueTx) to select size, a gap an activation-level test caught: without it, sizes_any could parse and unit-test correctly while never actually influencing which workflow a run bound.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Workflow grammar —
whenpredicates gainorandcontains_any; steps can declare a severity floor and stall limit; cross-issue linked input form;max_fix_loopscounting documented.Fix loops — entry scoped to per-trigger clusters; loop parks when the routing verdict repeats or the fix commit didn't move; steering notes carry into the next round; run-scoped batch gate-override covers repeated identical failures (DKT-546).
Votes — approved votes route on concerned casts; vote records exposed; retry refused once a proposal has decided; silent seats named with their path; weighted score separated from ballot count; vote-step spend counted against the run budget.
Claims and budget — claims can declare a variant-scaled cost; a forced reap no longer spends the attempt budget; routing metadata recorded at claim; run report pairs requested and resolved tier; aggregate builtin gets a routing floor; warning when usage is backfilled against an unclaimed step.
Packet pins — pin only the closure a run's workflows reach; check the pin set is closed; validate a packet before taking a lease; repin drops gone files and adds newly required ones; registered workflow source drift detected on disk and shown in run reports.
Gates — gate processes get the step's base commit; stub gate results marked with a reason and tracking issue; override-pass refused when it would skip a gate; failed gate named when a step parks; advisory pre-gate verdicts marked; gates named as make targets.
Registry and scope — cross-project workflow/schema registration and a drift audit; orphaned registrations surfaced;
--deprecatedhides retired versions andshow NAMEskips them (DKT-616); a widened scope can reach an active run, with a warning when it can't; unresolvable stale targets flagged and waivable.CLI fixes —
events follow --tailstarts at the newest N;events listfinds a run in any project;next --runreturns the full ready set; dispatch close does backfill and verify in one step; trust changes refuse a missing actor/cwd; loop redirects no longer bind an unrelated step's output; operator pause survives a step record; in-repo docket skill copy retired.