feat(outbound): enforce sending policy at fire time (B6) - #999
Merged
Conversation
…asons Two additive local failure reasons for the sending-protection holds: submission.policy_budget_expired (a sending-budget hold reached its seven-day deadline) and submission.sending_setup_expired (SES tenant readiness did not land within the 72-hour setup deadline). Both are local, correctable outcomes like submission.local_retries_exhausted, and neither may ever be reported as a recipient rejection or a provider outage. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
… paths Two callers hold provider evidence but no token: the worker that finds provider-accept evidence already recorded on a row it is about to re-drive, and the terminal reconciler settling a stranded row from that same evidence. Neither can name an ordinal. SettleOperation applies the outcome to the latest attempt whose provider call started — never a later ordinal that was only reserved, and nothing when no attempt ever dialed — through the same body SettleProvider uses. LookupOperation recovers a reference for an operation that already exists; it is not a constructor, and every Gate method still reloads the row under lock. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The send claim now returns the message's persisted hold class and anchor (migration 116) plus the owning account's last_resumed_at and ses_tenant_ready_at, so every worker execution can re-derive the same deadline. RecordOutboundHold writes the pair only while the message is pre-terminal; every terminal write — sent, failed, evidence-settled, trash-cancelled — clears it, so a stale hold can never outlive its message's outcome. The two new local expiry reasons are recognized as complete terminal fallbacks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The outbound send worker now authorizes every provider call through the sending-protection Gate, in the fixed order the design names: Reserve the durable attempt; snooze on an early hold without provider I/O; DeferAttempt on a rate deferral and CancelAttempt on a final suppression match; ConsumeAttempt as the last serialized decision; then the authorized submitter, which redeems the token immediately before the socket opens and settles the provider's answer. A later execution after a confirmed attempt returns to Reserve, which allocates the next ordinal. The worker-owned RampGate and agent.NewOutboundRampGate are removed: the ramp is composed inside the gate and its progress moves only through settlement. The Deliverer contract carries the token; the production deliverer is outbound.ProviderSubmitter and refuses to dial without one. A lost 250 (ErrProviderAcceptanceUnknown) is retried as a new ordinal and never settled. Enqueue prepares the operation in the accept transaction, between the message insert and the River insert; a paused account is refused there (ErrSendingPaused, HTTP 403 sending_paused) rather than queued. Jobs from a pre-floor slot carry no reference and resolve at fire time through the same Prepare path. Finite holds persist a class and anchor on the message and derive the deadline every execution: 72 hours for rate/ramp/provider and tenant setup, seven days for policy budget. The first finite hold anchors at the latest of accept, schedule, review, and last resume; a budget hold promotes any class and keeps the anchor; policy_budget never changes again; tenant readiness landing inside the setup deadline moves the class to rate/ramp/provider exactly once; a pause has no clock but a running deadline keeps running. Expiry emits the class's own reason. Terminal reconciliation is settlement-only: an evidence-settled row also settles the attempt that dialed through Gate.SettleOperation. cmd/e2a gains one composition root (newOutboundSending) and a wiring test that proves the registered send path holds the concrete gate and the ProviderSubmitter-backed deliverer. The test servers build the same composition with the disabled policy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The machine-checked contracts caught three vocabularies the worker cutover widened without saying so: the error-code catalog and the ErrorBody.Code documentation (sending_paused, 403, auth family), the lifecycle reason table in docs/api.md (submission.policy_budget_expired, submission.sending_setup_expired), and the OpenAPI description the two generated SDK models embed. Both SDK error maps classify sending_paused as a non-retryable permission error, with tests. The email-eval integration runner's job-args parser insisted on exactly one key; the accept transaction now stamps operation_ref beside message_id, so the parser admits that key and still rejects any other. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Two parallel reviews (correctness + adversarial) over the first cut. Every item has a named test. Contract surfaces (the blocker in both): the two new lifecycle reasons join the hand-maintained reason_code enum tag, both closed-vocabulary tests, the regenerated spec, both generated SDK models (mirrored by hand; the generator needs Docker, and the description now carries no apostrophe so the two generators agree), the web lifecycle parser and timeline, docs/api.md and docs/events.md. sending_paused is registered in the error catalog, the ErrorBody.Code doc, docs/api.md, and both SDK error maps. Worker: - A gate outage is a bounded rate/ramp/provider hold, not an unbounded snooze. - A paused job evaluates no deadline; a persisted deadline is not extended and the first hold after resume applies it. - A provider outage never emits the setup reason (expiryReasonFor). - markFailed's evidence-settle branch settles the dialed attempt, as the reconciler already did; a failed post-acceptance settlement is retried before it is logged as critical; a provider-id conflict is surfaced as an invariant alarm. - The job's operation reference must name its own message; a mismatch cancels before any ledger call. Enqueue refuses a zero reference. - HoldClassFor maps reasons by name; the armed worker RegisterJobs builds is exposed (Jobs.SendWorker) so the wiring test can prove it carries the gate and the legacy resolver and the submitter carries the configuration set. Gate: SettleOperation prefers the oldest dialed attempt with no provider id yet, so evidence arriving in send order binds each attempt when several dialed. Paused accounts on every enqueue path: 403 sending_paused on the direct, platform-test, and HITL-approve paths; the TTL auto-approve sweep defers a paused account's expired review by an hour (DeferReviewExpiry) instead of re-picking it first every cycle and starving the batch. The email-eval integration runner admits the operation_ref args key. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Mutation-tested re-review of the previous fix round: no blockers, six should-fixes. - SettleOperation without a token resolves the attempt as: one already bound to this exact provider id (a replay stays home), else the oldest dialed attempt with no id, else the latest dialed. The earlier oldest-unbound-first rule let a replay for attempt one bind attempt two; the test now covers that shape. - snoozeOnGateError threads the live reservation into the bounded hold, so an expiry at final authorization gives the attempt back instead of stranding it under an enforcing policy. - MarkFailed returns the evidence's provider id, and the worker's evidence settle under a terminal write carries it — the reconciler already did. The two evidence paths now agree. - resettle logs at critical level when the context ends mid-retry; a dedicated test covers the retry itself. - The wiring test registers workers exactly as main does and inspects the worker River received (Jobs.RegisteredSendWorker), so a RegisterJobs that bypassed the armed constructor fails it. - sending_paused is marked experimental beside blocked_by_policy in the stability extension, the docs, and the description, since the pause control ships disabled and pre-GA. The Python forward-compat table gains the two lifecycle reasons. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Slice B6 of the sending abuse prevention plan (Task 6: primary outbound worker, hold deadlines, legacy resolver, composition root). Four commits, bottom-up.
internal/outboundsend):Reserve→ early hold snoozes without I/O → rate gateDeferAttempts → final suppressionCancelAttempts →ConsumeAttempt→ authorized submit. The worker-ownedRampGateandagent.NewOutboundRampGateare removed; the ramp lives inside the gate (B4) and moves only through settlement.Deliverernow carries the token; production's deliverer isoutbound.ProviderSubmitter(B5), which refuses to dial without one. A lost 250 (ErrProviderAcceptanceUnknown) retries as a new ordinal and is never settled.PrepareExternalTxbetween message insert and River insert). A paused account is refused at the door:ErrSendingPaused→ HTTP 403sending_paused, transaction rolled back. Legacy jobs (nooperation_refin args — a pre-floor slot) resolve at fire time viaJobs.ResolveLegacyOperation, the same Prepare path.messages.local_hold_class/local_hold_anchor(migration 116) ride the claim; the deadline is always derived (72h forrate_ramp_or_providerandtenant_setup, 7d forpolicy_budget). First-hold anchor = max(accept, schedule, review, last resume); budget promotes any class keeping the anchor;policy_budgetis sticky; tenant readiness inside the setup deadline moves to rate/ramp/provider once; a pause has no clock but a running deadline keeps running. Expiry reasons:submission.local_retries_exhausted,submission.sending_setup_expired,submission.policy_budget_expired(the last two are new catalog entries).Gate.SettleOperation(lateststartedattempt);Gate.LookupOperationrecovers a reference for an existing operation (not a constructor).cmd/e2a/outbound_wiring.go+sending_policy_wiring_test.go, which proves the registered send path holds the concrete*sendingpolicy.Moduleand theProviderSubmitter-backed deliverer. Test servers build the same composition with the disabled policy.Gate for this slice (per plan)
Hosted policy stays disabled; the primary path uses the authorization seam without enforcing budgets. Prod behaviour with the disabled policy: every send still gets a durable operation + attempt and a redeemed token, and the ramp remains pass-through. This is the first slice whose code runs on every prod send, so it needs the staging soak the plan calls for before promotion.
Review round 1
Two parallel Opus reviews (correctness + adversarial). Everything below is fixed in the follow-up commits, each with a named test:
sending_pausedregistered in the error catalog, theErrorBody.Codedoc,docs/api.md, and both SDK error maps; the two lifecycle reasons added to the hand-maintainedreason_codeenum tag, the two closed-vocabulary tests, the regenerated spec, both generated SDK models (mirrored by hand — the generator needs Docker), the web lifecycle parser and timeline labels,docs/api.md,docs/events.md.SettleOperation, matching the reconciler; a failed post-acceptance settlement is retried before it is logged as critical;ErrProviderMessageIDConflictis surfaced as an invariant alarm, never a plain log line.SettleOperationattribution: prefers the oldest dialed attempt with no provider id yet, so evidence arriving in send order binds each attempt correctly when several dialed.sending_pausedon the direct, platform-test, and HITL-approve paths; the TTL auto-approve sweep defers a paused account's expired review by an hour (DeferReviewExpiry) instead of re-picking it first every cycle and starving the batch.RegisterJobsregisters (gate + legacy resolver) and the submitter's configuration set, not just the bundle field.HoldClassFormaps reasons by name with a table test; the eval-runner integration test admits theoperation_refargs key.Known and deferred, recorded in the ops project memory: SES tenant readiness has no production writer yet —
tenant_header_modemust staydisableduntil the provisioning slice lands, or every customer send holds and expires as setup; reservations stranded by a best-effort cancel failure are day-scoped by design; ramp units stay reserved for an ambiguous-then-failed send by design (a message that might have been delivered must not release capacity); a first hold on a message already past its anchor's horizon expires at once (the anchor rule, by design).Review round 2
Mutation-tested re-review of the round-1 fixes: no blockers, six should-fixes, all addressed:
SettleOperationwithout a token now resolves the attempt in this order: one already bound to the exact provider id (a replay stays on its own attempt), else the oldest dialed attempt with no id, else the latest dialed; test extended to the replay-with-a-later-unbound-attempt shape.MarkFailedreturns the evidence's provider id, and the worker's evidence settle under a terminal write carries it (the reconciler already did).maindoes and inspects the worker River received, not a freshly constructed one.sending_pausedis marked experimental alongsideblocked_by_policy(stability extension, docs, and the description), since the pause control ships disabled and pre-GA.Nits recorded, not changed: the reviewers' page and pending-message API show a paused account's review TTL sliding forward an hour per sweep (silent, no event); the CLI exits with the generic request code for
sending_paused.Not in this slice
Sender.SubmitOnce*remains for them.-reconcile-legacy-sending-jobsoperator command (inventory); the fire-time resolver covers correctness.SettleOperationon SNS evidence (worker + reconciler do; the consumer is a follow-up).Tests
Unit (fake gate): allowed path order; every hold reason → class and snooze without I/O; pause indefinite; pause under a running budget deadline expires; promotion and stickiness; expiry reason per class; terminal holds cancel; rate deferral defers the attempt; suppression cancels it; late-gate hold; gate outage snoozes without burning an attempt; evidence settles the operation; legacy resolution (accept / paused / orphan / unwired); tenant readiness transition (in time / too late); anchor rule; acceptance-unknown retried unsettled; hold constant matches the policy default.
DB-backed (real gate): enqueue writes
operation_refand the operation row; paused account refused and rolled back; legacy job authorizes through the gate; a gated job authorizes with a real token and a re-drive allocates nothing; the reconciler settles the dialed attempt from evidence and binds the provider id.Full suites green for outboundsend, agent, cmd/e2a, testutil, sendingpolicy, sendramp, outbound, hitlworker, delivery, messagelifecycle;
-raceclean on outboundsend + sendingpolicy. Three engagement/outreach tests in agent/identity fail identically on cleanorigin/mainlocally (pre-existing, CI-green).🤖 Generated with Claude Code
https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX