Skip to content

feat(outbound): enforce sending policy at fire time (B6) - #999

Merged
jiashuoz merged 8 commits into
mainfrom
feat/sending-worker-cutover
Sep 5, 2026
Merged

feat(outbound): enforce sending policy at fire time (B6)#999
jiashuoz merged 8 commits into
mainfrom
feat/sending-worker-cutover

Conversation

@jiashuoz

@jiashuoz jiashuoz commented Sep 5, 2026

Copy link
Copy Markdown
Member

What

Slice B6 of the sending abuse prevention plan (Task 6: primary outbound worker, hold deadlines, legacy resolver, composition root). Four commits, bottom-up.

  • Worker order over the Gate (internal/outboundsend): Reserve → early hold snoozes without I/O → rate gate DeferAttempts → final suppression CancelAttempts → ConsumeAttempt → authorized submit. The worker-owned RampGate and agent.NewOutboundRampGate are removed; the ramp lives inside the gate (B4) and moves only through settlement. Deliverer now carries the token; production's deliverer is outbound.ProviderSubmitter (B5), which refuses to dial without one. A lost 250 (ErrProviderAcceptanceUnknown) retries as a new ordinal and is never settled.
  • Enqueue prepares the operation in the accept transaction (PrepareExternalTx between message insert and River insert). A paused account is refused at the door: ErrSendingPaused → HTTP 403 sending_paused, transaction rolled back. Legacy jobs (no operation_ref in args — a pre-floor slot) resolve at fire time via Jobs.ResolveLegacyOperation, the same Prepare path.
  • Durable hold deadlines: messages.local_hold_class / local_hold_anchor (migration 116) ride the claim; the deadline is always derived (72h for rate_ramp_or_provider and tenant_setup, 7d for policy_budget). First-hold anchor = max(accept, schedule, review, last resume); budget promotes any class keeping the anchor; policy_budget is sticky; tenant readiness inside the setup deadline moves to rate/ramp/provider once; a pause has no clock but a running deadline keeps running. Expiry reasons: submission.local_retries_exhausted, submission.sending_setup_expired, submission.policy_budget_expired (the last two are new catalog entries).
  • Settlement-only reconciliation: an evidence-settled row also settles the attempt that dialed via the new Gate.SettleOperation (latest started attempt); Gate.LookupOperation recovers a reference for an existing operation (not a constructor).
  • Composition root cmd/e2a/outbound_wiring.go + sending_policy_wiring_test.go, which proves the registered send path holds the concrete *sendingpolicy.Module and the ProviderSubmitter-backed deliverer. Test servers build the same composition with the disabled policy.

Gate for this slice (per plan)

Hosted policy stays disabled; the primary path uses the authorization seam without enforcing budgets. Prod behaviour with the disabled policy: every send still gets a durable operation + attempt and a redeemed token, and the ramp remains pass-through. This is the first slice whose code runs on every prod send, so it needs the staging soak the plan calls for before promotion.

Review round 1

Two parallel Opus reviews (correctness + adversarial). Everything below is fixed in the follow-up commits, each with a named test:

  • Contract surfaces (blocker in both reviews): sending_paused registered in the error catalog, the ErrorBody.Code doc, docs/api.md, and both SDK error maps; the two lifecycle reasons added to the hand-maintained reason_code enum tag, the two closed-vocabulary tests, the regenerated spec, both generated SDK models (mirrored by hand — the generator needs Docker), the web lifecycle parser and timeline labels, docs/api.md, docs/events.md.
  • Gate outage was an unbounded snooze: now a bounded rate/ramp/provider hold like every other wait.
  • Pause semantics: a paused job evaluates no deadline (spec), but a persisted deadline is not extended — the first hold after resume applies it.
  • Provider outage never emits the setup reason (spec).
  • Worker's evidence-settle under a terminal write now settles the dialed attempt through SettleOperation, matching the reconciler; a failed post-acceptance settlement is retried before it is logged as critical; ErrProviderMessageIDConflict is surfaced as an invariant alarm, never a plain log line.
  • SettleOperation attribution: prefers the oldest dialed attempt with no provider id yet, so evidence arriving in send order binds each attempt correctly when several dialed.
  • The job's operation reference must name its message: a mismatch cancels before any ledger call. Enqueue refuses a zero reference rather than emitting a legacy-looking job.
  • Paused account on every enqueue path: 403 sending_paused on the direct, platform-test, and HITL-approve paths; the TTL auto-approve sweep defers a paused account's expired review by an hour (DeferReviewExpiry) instead of re-picking it first every cycle and starving the batch.
  • Wiring test now inspects the armed worker RegisterJobs registers (gate + legacy resolver) and the submitter's configuration set, not just the bundle field.
  • HoldClassFor maps reasons by name with a table test; the eval-runner integration test admits the operation_ref args key.

Known and deferred, recorded in the ops project memory: SES tenant readiness has no production writer yettenant_header_mode must stay disabled until the provisioning slice lands, or every customer send holds and expires as setup; reservations stranded by a best-effort cancel failure are day-scoped by design; ramp units stay reserved for an ambiguous-then-failed send by design (a message that might have been delivered must not release capacity); a first hold on a message already past its anchor's horizon expires at once (the anchor rule, by design).

Review round 2

Mutation-tested re-review of the round-1 fixes: no blockers, six should-fixes, all addressed:

  • SettleOperation without a token now resolves the attempt in this order: one already bound to the exact provider id (a replay stays on its own attempt), else the oldest dialed attempt with no id, else the latest dialed; test extended to the replay-with-a-later-unbound-attempt shape.
  • A gate outage at final authorization threads the live reservation into the bounded hold, so an expiry gives the attempt back instead of stranding it.
  • MarkFailed returns the evidence's provider id, and the worker's evidence settle under a terminal write carries it (the reconciler already did).
  • The post-acceptance settlement retry logs at critical level when the context ends mid-retry.
  • The wiring test now registers workers exactly as main does and inspects the worker River received, not a freshly constructed one.
  • sending_paused is marked experimental alongside blocked_by_policy (stability extension, docs, and the description), since the pause control ships disabled and pre-GA.

Nits recorded, not changed: the reviewers' page and pending-message API show a paused account's review TTL sliding forward an hour per sweep (silent, no event); the CLI exits with the generic request code for sending_paused.

Not in this slice

  • The AST closure guard and making raw relay methods package-private (Task 7).
  • HITL / webhook-health / public-feedback paths on the seam (Task 7). Sender.SubmitOnce* remains for them.
  • The -reconcile-legacy-sending-jobs operator command (inventory); the fire-time resolver covers correctness.
  • The delivery-feedback consumer calling SettleOperation on SNS evidence (worker + reconciler do; the consumer is a follow-up).

Tests

Unit (fake gate): allowed path order; every hold reason → class and snooze without I/O; pause indefinite; pause under a running budget deadline expires; promotion and stickiness; expiry reason per class; terminal holds cancel; rate deferral defers the attempt; suppression cancels it; late-gate hold; gate outage snoozes without burning an attempt; evidence settles the operation; legacy resolution (accept / paused / orphan / unwired); tenant readiness transition (in time / too late); anchor rule; acceptance-unknown retried unsettled; hold constant matches the policy default.

DB-backed (real gate): enqueue writes operation_ref and the operation row; paused account refused and rolled back; legacy job authorizes through the gate; a gated job authorizes with a real token and a re-drive allocates nothing; the reconciler settles the dialed attempt from evidence and binds the provider id.

Full suites green for outboundsend, agent, cmd/e2a, testutil, sendingpolicy, sendramp, outbound, hitlworker, delivery, messagelifecycle; -race clean on outboundsend + sendingpolicy. Three engagement/outreach tests in agent/identity fail identically on clean origin/main locally (pre-existing, CI-green).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX

jiashuoz and others added 8 commits September 4, 2026 22:01
…asons

Two additive local failure reasons for the sending-protection holds:
submission.policy_budget_expired (a sending-budget hold reached its
seven-day deadline) and submission.sending_setup_expired (SES tenant
readiness did not land within the 72-hour setup deadline). Both are
local, correctable outcomes like submission.local_retries_exhausted, and
neither may ever be reported as a recipient rejection or a provider
outage.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
… paths

Two callers hold provider evidence but no token: the worker that finds
provider-accept evidence already recorded on a row it is about to
re-drive, and the terminal reconciler settling a stranded row from that
same evidence. Neither can name an ordinal. SettleOperation applies the
outcome to the latest attempt whose provider call started — never a
later ordinal that was only reserved, and nothing when no attempt ever
dialed — through the same body SettleProvider uses. LookupOperation
recovers a reference for an operation that already exists; it is not a
constructor, and every Gate method still reloads the row under lock.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The send claim now returns the message's persisted hold class and
anchor (migration 116) plus the owning account's last_resumed_at and
ses_tenant_ready_at, so every worker execution can re-derive the same
deadline. RecordOutboundHold writes the pair only while the message is
pre-terminal; every terminal write — sent, failed, evidence-settled,
trash-cancelled — clears it, so a stale hold can never outlive its
message's outcome. The two new local expiry reasons are recognized as
complete terminal fallbacks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The outbound send worker now authorizes every provider call through the
sending-protection Gate, in the fixed order the design names: Reserve
the durable attempt; snooze on an early hold without provider I/O;
DeferAttempt on a rate deferral and CancelAttempt on a final suppression
match; ConsumeAttempt as the last serialized decision; then the
authorized submitter, which redeems the token immediately before the
socket opens and settles the provider's answer. A later execution after
a confirmed attempt returns to Reserve, which allocates the next ordinal.

The worker-owned RampGate and agent.NewOutboundRampGate are removed: the
ramp is composed inside the gate and its progress moves only through
settlement. The Deliverer contract carries the token; the production
deliverer is outbound.ProviderSubmitter and refuses to dial without one.
A lost 250 (ErrProviderAcceptanceUnknown) is retried as a new ordinal and
never settled.

Enqueue prepares the operation in the accept transaction, between the
message insert and the River insert; a paused account is refused there
(ErrSendingPaused, HTTP 403 sending_paused) rather than queued. Jobs from
a pre-floor slot carry no reference and resolve at fire time through the
same Prepare path.

Finite holds persist a class and anchor on the message and derive the
deadline every execution: 72 hours for rate/ramp/provider and tenant
setup, seven days for policy budget. The first finite hold anchors at
the latest of accept, schedule, review, and last resume; a budget hold
promotes any class and keeps the anchor; policy_budget never changes
again; tenant readiness landing inside the setup deadline moves the
class to rate/ramp/provider exactly once; a pause has no clock but a
running deadline keeps running. Expiry emits the class's own reason.

Terminal reconciliation is settlement-only: an evidence-settled row also
settles the attempt that dialed through Gate.SettleOperation.

cmd/e2a gains one composition root (newOutboundSending) and a wiring
test that proves the registered send path holds the concrete gate and
the ProviderSubmitter-backed deliverer. The test servers build the same
composition with the disabled policy.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The machine-checked contracts caught three vocabularies the worker
cutover widened without saying so: the error-code catalog and the
ErrorBody.Code documentation (sending_paused, 403, auth family), the
lifecycle reason table in docs/api.md (submission.policy_budget_expired,
submission.sending_setup_expired), and the OpenAPI description the two
generated SDK models embed. Both SDK error maps classify sending_paused
as a non-retryable permission error, with tests.

The email-eval integration runner's job-args parser insisted on exactly
one key; the accept transaction now stamps operation_ref beside
message_id, so the parser admits that key and still rejects any other.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Two parallel reviews (correctness + adversarial) over the first cut.
Every item has a named test.

Contract surfaces (the blocker in both): the two new lifecycle reasons
join the hand-maintained reason_code enum tag, both closed-vocabulary
tests, the regenerated spec, both generated SDK models (mirrored by
hand; the generator needs Docker, and the description now carries no
apostrophe so the two generators agree), the web lifecycle parser and
timeline, docs/api.md and docs/events.md. sending_paused is registered
in the error catalog, the ErrorBody.Code doc, docs/api.md, and both SDK
error maps.

Worker:
- A gate outage is a bounded rate/ramp/provider hold, not an unbounded
  snooze.
- A paused job evaluates no deadline; a persisted deadline is not
  extended and the first hold after resume applies it.
- A provider outage never emits the setup reason (expiryReasonFor).
- markFailed's evidence-settle branch settles the dialed attempt, as the
  reconciler already did; a failed post-acceptance settlement is retried
  before it is logged as critical; a provider-id conflict is surfaced as
  an invariant alarm.
- The job's operation reference must name its own message; a mismatch
  cancels before any ledger call. Enqueue refuses a zero reference.
- HoldClassFor maps reasons by name; the armed worker RegisterJobs
  builds is exposed (Jobs.SendWorker) so the wiring test can prove it
  carries the gate and the legacy resolver and the submitter carries the
  configuration set.

Gate: SettleOperation prefers the oldest dialed attempt with no provider
id yet, so evidence arriving in send order binds each attempt when
several dialed.

Paused accounts on every enqueue path: 403 sending_paused on the direct,
platform-test, and HITL-approve paths; the TTL auto-approve sweep defers
a paused account's expired review by an hour (DeferReviewExpiry) instead
of re-picking it first every cycle and starving the batch.

The email-eval integration runner admits the operation_ref args key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Mutation-tested re-review of the previous fix round: no blockers, six
should-fixes.

- SettleOperation without a token resolves the attempt as: one already
  bound to this exact provider id (a replay stays home), else the oldest
  dialed attempt with no id, else the latest dialed. The earlier
  oldest-unbound-first rule let a replay for attempt one bind attempt
  two; the test now covers that shape.
- snoozeOnGateError threads the live reservation into the bounded hold,
  so an expiry at final authorization gives the attempt back instead of
  stranding it under an enforcing policy.
- MarkFailed returns the evidence's provider id, and the worker's
  evidence settle under a terminal write carries it — the reconciler
  already did. The two evidence paths now agree.
- resettle logs at critical level when the context ends mid-retry; a
  dedicated test covers the retry itself.
- The wiring test registers workers exactly as main does and inspects
  the worker River received (Jobs.RegisteredSendWorker), so a
  RegisterJobs that bypassed the armed constructor fails it.
- sending_paused is marked experimental beside blocked_by_policy in the
  stability extension, the docs, and the description, since the pause
  control ships disabled and pre-GA. The Python forward-compat table
  gains the two lifecycle reasons.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
@jiashuoz
jiashuoz merged commit 2f6cdec into main Sep 5, 2026
29 checks passed
@jiashuoz
jiashuoz deleted the feat/sending-worker-cutover branch September 5, 2026 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant