Skip to content

JITSU-227: Execute Reverse ETL with Node runner and syncctl jobs - #1521

Open
absorbb wants to merge 2 commits into
feat/jitsu-227-persistencefrom
feat/jitsu-227-runner
Open

absorbb wants to merge 2 commits into
feat/jitsu-227-persistencefrom
feat/jitsu-227-runner

Conversation

@absorbb

@absorbb absorbb commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Summary

JITSU-227. Executable Node runner and syncctl/CronJob integration, following the merged core-mirror PR #1519. Stacked on the still-open persistence PR #1516; merge that first.

  • One non-root Node application container, runtime image/build target, and compiled-in provider binding boundary. No Go sidecar or dynamic module loading.
  • Wire full/cursor upsert and core snapshot mirroring to warehouse readers and PostgreSQL persistence; reconcile interrupted delivery from exact durable evidence.
  • Acquire/renew Kubernetes and PostgreSQL ownership, persist source_task/task_log lifecycle state, handle cancellation/deadlines and bounded shutdown without clearing uncertain delivery.
  • Add independent reverse-syncs export and kind-scoped shared CronJob reconciliation. Failed exports retain the prior feed; connector jobs stay isolated.
  • Reuse syncctl read/cancel APIs for scoped manual runs and cancellation. Fresh console admission blocks disabled/deleted/stale configurations and maintenance/read-only execution.
  • Independently detect stale heartbeats and retry Pod termination after transient API failures. Pod exit never implies delivery success; terminal task state is preserved.

Boundaries

  • No billing, schema/association tables, UI enablement, live ads adapters, target reset/transfer or deployment.
  • Provider registry is deliberately empty until API, normalization and recovery contracts are verified. Existing reverse-link creation guard remains.
  • Provider raw messages are suppressed; core-owned redacted lifecycle logs persist. Pending delivery is a FAILED task with a recovery message; the next manual/scheduled attempt reconciles.
  • Runtime requires an explicitly provisioned DB/encryption/admission Secret and lease RBAC. Credential/delivery configuration changes remain revision-bound and require controlled reset; scheduling changes do not.
  • Controller REVERSE_ENABLED is opt-in wiring; disabling link/workspace feature flags blocks fresh admission for existing CronJobs too.

Validation

  • 122 Node tests pass, including existing persistence/mirror suites, new executable/lease tests, and optional million-identity SQL pagination check.
  • 8 console integration tests pass (new scoped export/admission data tests and existing admin exports).
  • 6 Go controller tests pass with race detection and fake Kubernetes clients. Used -vet=off because unchanged fmt.Errorf(stderr.String()) in existing job_runner.go fails the current Go vet check.
  • Runner + console TypeScript checks, targeted ESLint/format checks, bundled runner build and missing-config startup smoke check pass.
  • Local independent finders, finding verification and fresh final review before push.

No production databases, Kubernetes clusters, advertising APIs or release/deployment workflows were invoked. Next slice: first verified Google Ads adapter, followed by complete Reverse sync UI and end-to-end rollout.

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the reverse-sync export, controller scheduling, runner admission/leases, and task cleanup paths.

Findings:

  • Manual Pod creation failures leave credential-bearing Secrets behind.
  • The enabled runner image has no production adapter bindings, so every admitted reverse run fails before delivery.

Comment thread bulker/sync-controller/reverse_tasks.go
Comment thread services/retl-runner/src/adapters.ts

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the reverse-ETL runner, console export/admission, Kubernetes lifecycle, and image/release wiring.

No new actionable findings. Existing resolved review threads were inspected and not re-raised.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant