JITSU-227: Core snapshot-mirror orchestration - #1519
Merged
Merged
Conversation
![jitsu-code-review[bot]](https://avatars.githubusercontent.com/in/3391383?s=60&v=4){kind=small}
This was referenced Sep 15, 2026
absorbb
added a commit
that referenced
this pull request
Sep 16, 2026
## Scope Third implementation slice of `JITSU-227`, on merged #1509. #1512 was closed by the owner and is deliberately excluded. Adds server-only Node/PostgreSQL delivery persistence under `services/retl-runner`; no executable runner, CronJob, provider adapter or live delivery is enabled. - Eight canonical `reverse_sync_*` models in the existing console Prisma schema; no separate `retl` namespace or standalone migration. Compact JSON-text cursor/store/generation state reuses `source_state`. - Ordinary `prisma db push` manages all tables, enums, keys and indexes. No supplemental CHECK, custom schema-update wrapper, Docker/Helm changes, or extra deployment bundle. - Runtime `pg` transactions use the same configured schema with a transaction-local search path and narrowly scoped runtime-role grants. - Kubernetes leases and syncctl are the sole worker coordination mechanism; no `Database.owned()`, database worker lease/epoch, or `fencingEpoch` in provider context. Ordinary transactions and short lifecycle row locks retain atomic receipts/checkpoints/counters. Persistent exclusive mirror audience ownership still prevents different syncs from managing the same target. - Prisma-generated row types cover control, generations, operations, batches, membership, saved state and target owners. A small `PgRow` mapping accounts for pg bigint strings and byte buffers; Prisma Client is type-only, not instantiated or bundled at runtime. - Durable prepared manifests, provider state and recoverable identities; bounded recovery reads and explicit core reconciliation acknowledgements. No blind replay. - Acceptance atomically persists receipts, effective membership and reservation/accounting updates. Partial accepted-then-failed outcomes remain recoverable. Per identity, the latest accepted source sequence wins even when staged work resolves later; indexed projected identity hashes on operations preserve ordering for both upserts and removals without blocking multi-batch staging or adding tombstones/tables. - Exact batch-receipt retries are read-only whether terminal, staged or mixed, and cannot roll back the run store. Replacements must advance a staged outcome, preserve terminal outcome details and keep original job IDs/checkpoint while any staged outcomes remain. Final terminal metadata may change; recovered transitions require core reconciliation. - Pending finish receipts are immutable until accepted. Matching retries preserve the store; changed job IDs/checkpoints are rejected through both facade and core recovery. - Interrupted initialization has a core-only reconciled retry path: verify absence or safe cleanup, atomically reset to new with reconciled store and cleared provider state, then reopen persistence for the same run before invoking the ordinary lifecycle. Recovery sessions cannot blindly acknowledge/restart initialization; unknown sessions remain blocked. - Existing `DeliveryJournal` facade with no database/snapshot access. Contiguous checkpoint validation, safe abort boundaries and restartable staged-finish resolution. - Snapshot creation is retryable for the current unsealed candidate. Core status reports extraction progress; sequential page appends use an atomic latest-page sequence/content hash on the existing generation table. Exact retries are read-only; changed pages, sequence gaps and duplicate keys in new pages fail closed. No association or page-receipt table. - Core-owned desired generations, duplicate source-key validation, shared-identity conflict checks, keyset diff reads, sealed-source removal guards and atomic promotion with final state. - Full-snapshot diffs compare deduplicated desired identities with acknowledged membership; no source-to-identity association table is needed. Shared identities remain present until their last source row disappears. - Conservative pre-submission storage reservations and bounded retention under the caller-held Kubernetes lease. ## Application-level encryption removed Per owner request, payloads are now persisted as readable, versioned UTF-8 JSON in the existing bytea columns. No Cipher, AAD, encryption keyring or key rotation is required; no column migration was added. source_state version 2 stores checkpoint JSON text inside its scoped JSON envelope (not base64), preserving NUL and unpaired-surrogate strings that PostgreSQL JSONB cannot represent directly. Legacy encrypted development data is rejected; no automatic reset, deletion or migration is performed. Workspace/revision/target checks, recovery receipts, transactional acceptance, redacted errors and byte limits remain. Membership/result reservations account for serialized JSON bytes. Confidentiality now relies on restricted database/backup access, infrastructure encryption and retention—not application-level payload encryption. Regression tests cover plain payload readback, exact UTF-8 byte accounting, unsupported formats, checkpoint scope checks and Unicode checkpoint recovery. Both Unicode cases failed before the JSON-text correction and pass afterward. ## Billing explicitly excluded Per owner request, this PR contains no billing-period inputs/validation, activation ledger, usage events/outbox/publisher, billing-service integration, entitlement enforcement, quota reservations, pricing or invoicing. All billing work is deferred to a separate slice. Receipts remain for delivery recovery, not billing. `accepted_at` is the database-clock time at which Jitsu acknowledged the verified outcome, not a remote delivery timestamp or billing-period attribution. Recovery no longer requires a remote timestamp or billing proof; only core reconciliation methods can resolve any unresolved recovered operation, whether accepted, rejected or staged. Accepted recovered finish also requires reconciliation even when no rows were staged. Interrupted finish resolution reuses its saved acknowledgement time. ## Other boundaries The caller must already hold the Kubernetes sync lease. Executable runner/syncctl wiring, independent task/log/status observation, lease-renewal loops and cancellation follow in subsequent slices. There is no database stale-worker fencing: a paused worker or already-started remote call is not forcibly stopped after lease expiry. The runtime must abort on Kubernetes lease loss; exactly-once remote delivery is not promised. Core projection supplies provider-ready identity/upsert/removal values; no vendor normalization rule is invented here. Core snapshot-mirror orchestration and SQL pagination validation are included via #1519. The executable runner/syncctl integration follows in stacked #1521; the first verified Google adapter remains a later slice. The existing upsert lifecycle still refuses mirror execution. Snapshot page sequences are not warehouse cursors: callers must reproduce an ambiguous page from a stable/replayable source and await acknowledgement before advancing; extraction resumption remains part of runner orchestration. Controlled config/target/mode changes fail closed until the reset/transfer workflow exists. Retention is explicit and runs under the caller-held Kubernetes lease; no background sweep is started by this package. ## Intentional empty mirror projections A mirror source projection may return `[]` to exclude a valid row from the desired audience. Its source key remains validated, stored and unique, and counts toward byte/entry budgets. Shared identities stay while any source row projects them. Previously tracked members are removed only after the complete snapshot is sealed and additions are accepted. All-excluded snapshots still finish and promote normally. Invalid rows, thrown projections and malformed payloads still fail; no error-to-empty fallback. Prepared delivery projections remain non-empty. Regression tests cover shared-member retention/removal, empty desired generations, validation, duplicate keys, retries/recovery and accounting rollback. ## Control reads, transitions and caching Read-only control observations no longer take row locks. Initialization, provider-state saves, and finish/abort preparation use scoped conditional `UPDATE … RETURNING` instead of pre-reading control. SQL phase/run guards remain; validation failures roll back phase and store together. A process-local per-sync cache and queue are shared by journal, snapshots, recovery and maintenance. Cache hits for `state()` and `recoveryStatus()` make no database calls. Simple writes refresh the cache from returned rows only after acknowledged commit. Multi-table accounting/checkpoint/snapshot/retention mutations retain fresh locked reads and invalidate afterward. Failed or uncertain transactions invalidate too; reopening refreshes durable state. No database worker ownership or cross-process cache is introduced. Regression checks cover cold-cache nonblocking reads, zero-query cache hits, one reload after complex writes, failed/uncertain commits, shared queues, competing transitions, wrong scope/phase, and old logical runs. ## Validation - 143 persistence, mirror, serialization and row-type tests pass, including the optional million-row SQL diff check. Disposable PostgreSQL 18 uses the canonical Prisma schema and restricted runtime grants. - Regression coverage retains atomic rollback, exact receipt retries, recovery reconciliation, mirror additions-before-removals, workspace/run/config boundaries, Unicode checkpoints, bounded storage and retention. Removed worker lease/epoch columns are explicitly checked absent. - Runner, protocols and destination-functions typechecks pass; 53 Reverse ETL destination lifecycle tests pass. Changed-file ESLint, formatting and diff checks pass. - Prisma Client generation and frozen-lockfile installation pass. Generated types preserve pg return values and nullability; no runtime Prisma client is introduced. - Incremental risky-tier review: independent persistence and schema/type finders, followed by a fresh final review; no findings. - No production schema changes, deployment or advertising writes. The executable integration is validated separately in stacked #1521. Next: runnable Node service and syncctl/CronJobs in stacked #1521, then verified provider adapters and UI rollout. Billing remains separate.
absorbb
added a commit
that referenced
this pull request
Sep 16, 2026
## Summary JITSU-227. Executable Node runner and syncctl/CronJob integration, following the merged core-mirror PR #1519. Stacked on the still-open persistence PR #1516; merge that first. - One non-root Node application container, runtime image/build target, and compiled-in provider binding boundary. No Go sidecar or dynamic module loading. - Wire full/cursor upsert and core snapshot mirroring to warehouse readers and PostgreSQL persistence; reconcile interrupted delivery from exact durable evidence. - Acquire/renew the Kubernetes per-sync lease (the sole worker ownership mechanism), persist source_task/task_log lifecycle state, handle cancellation/deadlines and bounded shutdown without clearing uncertain delivery. - Add independent reverse-syncs export and kind-scoped shared CronJob reconciliation. Failed exports retain the prior feed; connector jobs stay isolated. - Reuse syncctl read/cancel APIs for scoped manual runs and cancellation. Fresh console admission blocks disabled/deleted/stale configurations and maintenance/read-only execution. - Independently detect stale heartbeats and retry Pod termination after transient API failures. Pod exit never implies delivery success; terminal task state is preserved. ## Boundaries - No billing, schema/association tables, UI enablement, live ads adapters, target reset/transfer or deployment. - Provider registry is deliberately empty until API, normalization and recovery contracts are verified. Existing reverse-link creation guard remains. - Provider raw messages are suppressed; core-owned redacted lifecycle logs persist. Pending delivery is a FAILED task with a recovery message; the next manual/scheduled attempt reconciles. - Runtime requires an explicitly provisioned DB/admission Secret and lease RBAC. Credential/delivery configuration changes remain revision-bound and require controlled reset; scheduling changes do not. - Controller REVERSE_ENABLED is opt-in wiring; disabling link/workspace feature flags blocks fresh admission for existing CronJobs too. ## Encryption-removal follow-up Updated to the plaintext persistence base from #1516. Runner startup no longer parses an encryption keyring or requires RETL_ACTIVE_KEY/RETL_KEYS; syncctl Pods require only RETL_DATABASE_URL, RETL_CONSOLE_URL and RETL_CONSOLE_TOKEN from the runtime Secret. Actual credentials remain in Kubernetes Secrets. Recovery reads scoped JSON-text checkpoints through the shared persistence decoder. Added coverage for empty cursor runs recovering pending finish while preserving a NUL-containing checkpoint. This earlier encryption-removal step left scheduling, provider reconciliation and billing unchanged. ## Kubernetes-only coordination and typed persistence Per owner request, remove the PostgreSQL worker lease/epoch and all `Database.owned()` calls. The runner renews only its Kubernetes lease plus the task heartbeat; lease/heartbeat loss aborts execution. Task/log writes use ordinary transactions and preserve terminal statuses. Database lifecycle checks and cross-sync mirror target ownership remain; they are not worker leases. A paused worker or an already-started remote call is not forcibly fenced after lease expiry. Persistence rows and recovery reads use Prisma-generated types with pg-specific bigint/byte mappings. Imports are type-only; the compiled runtime contains no Prisma client. Initialization recovery now reopens persistence instead of releasing/reacquiring a database owner. ## Validation - 136 Node tests pass for this revision, including persistence/mirror suites, executable/lease tests, a denied-Kubernetes-admission regression, and the optional million-row SQL pagination check. Also 53 Reverse ETL destination lifecycle tests pass. - At the preceding checkpoint, 8 console integration tests passed (new scoped export/admission data tests and existing admin exports). - At the preceding checkpoint, 7 Go controller tests passed with race detection and fake Kubernetes clients. Used `-vet=off` because unchanged `fmt.Errorf(stderr.String())` in existing job_runner.go fails the current Go vet check. - Current runner/protocols/destination TypeScript checks, targeted ESLint/format checks and bundled runner build pass. No Prisma runtime import in the bundle. Console and missing-config startup checks passed at the preceding checkpoint; their implementation is unchanged. - Incremental risky-tier local review: independent persistence/schema and runner finders, followed by fresh final reviews. No findings to verify. No production databases, Kubernetes clusters, advertising APIs or release/deployment workflows were invoked. Next slice: first verified Google Ads adapter, followed by complete Reverse sync UI and end-to-end rollout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
JITSU-227. Stacked on #1516 (
feat/jitsu-227-persistence); merge persistence first. This PR adds the server-side snapshot-mirror library, not an executable runner or production enablement.Billing, schema changes, source-to-identity association tables, live destination adapters, UI, native replacement, and syncctl/CronJobs are intentionally excluded. The next slice wires the executable Node runner into syncctl/CronJobs.
Validation
RETL_MIRROR_SCALE_TEST=1(80 persistence + 25 mirror).No live advertising APIs or production databases were accessed.