JITSU-163: klauspost/gzip on the gzip write paths - #1504
Merged
Merged
Conversation
![jitsu-code-review[bot]](https://avatars.githubusercontent.com/in/3391383?s=60&v=4){kind=small}
sahiltyagi-jitsu
added this pull request to stack #1505
September 9, 2026 11:21
absorbb
approved these changes
Sep 10, 2026
Base automatically changed from
feat/jitsu-163-klauspost-gzip-readers
to
newjitsu
September 15, 2026 07:11
sahiltyagi-jitsu
force-pushed
the
feat/jitsu-163-klauspost-gzip-writers
branch
from
September 15, 2026 07:11
405eb25 to
f7f452b
Compare
sahiltyagi-jitsu
added a commit
that referenced
this pull request
Sep 15, 2026
`Bulker Test (app)` fails on every branch that actually runs it, and **`newjitsu` is currently red** for the same reason. ``` panic: create container: Error response from daemon: pull access denied for minio/minio, repository does not exist or may require 'docker login' ``` ## Cause `docker.io/minio/minio` is **no longer publicly pullable**. An anonymous pull token from `auth.docker.io` is rejected with `UNAUTHORIZED`, and Docker Hub's API has no public record of the repository. Docker's *"repository does not exist or may require 'docker login'"* is its generic message for an unresolvable reference — so this is **not** rate limiting, and adding registry credentials would not have fixed it. Reproduced locally with the identical error: ``` $ docker pull minio/minio:latest Error response from daemon: pull access denied for minio/minio, repository does not exist or may require 'docker login' ``` ## Fix MinIO also publish to **quay.io**, which is public. One line in `minio_container.go`. Pinned to `RELEASE.2025-09-07T16-13-09Z` rather than a floating tag. That tag is **multi-arch** (amd64 / arm64 / ppc64le) and its manifest list is currently **byte-identical** to quay's `:latest`, so pinning costs nothing today and keeps the tests runnable on Apple Silicon. Both verified by pulling: ``` $ docker pull quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z # arm64, native Status: Downloaded newer image $ docker pull --platform linux/amd64 quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z Status: Downloaded newer image ``` There is precedent for quay in the tree — `docker/docker-compose.yml:288` pulls Keycloak from it. ## Why this went unnoticed for six days The unpinned `:latest` meant the break needed no commit in this repo. And the bulker shard **skips whenever a PR does not touch `bulker/`**, reporting `pass` in ~11s — a real run takes 4-5 minutes. Recent merges (#1502, #1507, #1509, #1510) all show `Bulker Test (app): pass` at 11s; none of them touched `bulker/`. The last real run was #1503 on **9 September**. The JITSU-163 PRs were the first to exercise the suite since, which is why they surfaced it — and merging #1503 turned `newjitsu` red. Worth considering separately: a scheduled run of the full suite on `newjitsu` would have caught this within a day instead of six. ## Verification CI on this PR is the real test — it runs the exact job that is failing. Locally the image now pulls natively on both architectures, and the previous reference reproduces the CI failure exactly. Unblocks #1504, #1506, #1508 and the default branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Second half of the klauspost swap in JITSU-163, covering the write sites the ticket names: gzip-encoded ingest responses, the gzipped script served to browsers, and operator's connection objects. klauspost emits standard gzip, so every consumer decodes it unchanged and no deploy ordering is needed. Verified live: the served /p.js decompresses with GNU gunzip and Python's gzip module, and parses as valid JavaScript. Stacked on the readers branch — merge that first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sahiltyagi-jitsu
force-pushed
the
feat/jitsu-163-klauspost-gzip-writers
branch
from
September 15, 2026 08:54
f7f452b to
9e2a6ab
Compare
sahiltyagi-jitsu
added a commit
that referenced
this pull request
Sep 15, 2026
… change (#1506) Third of the small PRs for [JITSU-163](https://linear.app/maroo/issue/JITSU-163), covering the warehouse batch path: the JSON, JSONArray and CSV marshallers, and the transactional SQL stream. **Stacked on #1504** (itself on #1503) — merge in order. No `go.mod` change; `bulkerlib` already carries klauspost as a direct dependency from #1503. ## The level change is part of this, deliberately The library swap and the compression level cannot be separated. Per the ticket's own benchmarks: | Codec | Output | |---|---| | `compress/gzip` L4 — what we run today | 185.5 MB | | `klauspost/gzip` L4 | 203.2 MB | | `klauspost/gzip` L6 | 177.4 MB | Swapping the library while keeping level 4 would emit **9.5% larger** files than today — the ticket says as much: *"klauspost L4 buys its speed with 9.5% larger output — don't take it."* Level 6 is the equivalent of the stdlib level 4 it replaces: 4.4% smaller and 25% cheaper. I originally planned these as two PRs for separate revert boundaries, which was wrong — it would have allowed the regression to merge on its own. The level now lives in a named `gzipLevel` constant with the reasoning inline, so it doesn't get "optimised" back down. ## Which files, and why these The ticket scopes this as "all five warehouse adapters", but `snowflake.go`, `redshift.go`, `clickhouse.go`, `bigquery.go` and the Mixpanel adapter contain no gzip calls. `GetBatchFileCompression()` only returns a declared value (`sql_adapter_base.go:166`), and redshift's `gzip` at line 34 is the `COPY … csv gzip` SQL keyword in a template string. The compression happens where a marshaller is constructed, so the set closes on `marshaller.go` and `sql/abstract_transactional.go`. Happy to narrow it if you meant something else. ## Verification `bulkerlib/implementations/sql` passes against real database containers — it writes, compresses and loads batch files, which is exactly this path. `bulkerlib/types` and `redshift_driver` pass. Build and `gofmt` clean. ## Risk Highest blast radius of the three: this output feeds Snowflake `COPY`, Redshift `COPY … csv gzip`, ClickHouse `url()`, BigQuery's loader, and customers reading their own S3/GCS objects. The format is unchanged — only the size and CPU move — and the CI warehouse suites are the real check. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
birhantprkc
pushed a commit
to birhantprkc/jitsu
that referenced
this pull request
Sep 15, 2026
`Bulker Test (app)` fails on every branch that actually runs it:
panic: create container: Error response from daemon: pull access denied for
minio/minio, repository does not exist or may require 'docker login'
`docker.io/minio/minio` is no longer publicly pullable — an anonymous pull token
is rejected with UNAUTHORIZED and Docker Hub has no public record of the repo.
Docker's "repository does not exist or may require 'docker login'" is its generic
message for an unresolvable reference, so this is not rate limiting and adding
registry credentials would not have fixed it. Reproduced locally with the same
error.
MinIO also publish to quay.io, which is public. Pinned to
RELEASE.2025-09-07T16-13-09Z rather than a floating tag: it is multi-arch
(amd64/arm64/ppc64le) and its manifest list is currently byte-identical to quay's
:latest, so pinning costs nothing today and keeps the tests runnable on Apple
Silicon.
The unpinned `:latest` is why this broke with no commit in the repo and stayed
hidden for six days: the bulker shard skips whenever a PR does not touch
`bulker/`, reporting "pass" in 11s, and nothing between 9 and 15 Sep touched it.
The JITSU-163 PRs were the first to run the suite again, and merging jitsucom#1503 turned
newjitsu red.
Unblocks jitsucom#1504, jitsucom#1506, jitsucom#1508 and the default branch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of the small PRs for JITSU-163, covering the write sites the ticket names for the klauspost swap: gzip-encoded ingest responses, the gzipped script served to browsers, and operator's stored connection objects.
Stacked on #1503 — based on that branch, so merge it first.
ingest/go.modis already correct there, which is why this diff doesn't touch it.Verified live
Rebuilt ingest in a local cluster and confirmed the binary came from this branch, then fetched
/p.jswithAccept-Encoding: gzip:gunzip— 158,419 bytes, exit 0gzipmodule — same byte countnode --check— parses as valid JavaScriptSo the output isn't merely gzip-shaped, it's byte-correct. Headers unchanged:
Content-Encoding: gzip,Vary: Accept-Encoding, ETag intact. Modules build clean;ingesttests pass.Not covered, deliberately named
router_ingest_handler.go— reaching that branch needs a stream with tag/device destinations andShouldCompresstrue, which I couldn't configure locally.operator.go— the module has no test files at all, and it writes objects whose consumers the ticket itself declined to trace ("not worth the consumer archaeology"). Format-identical output makes that safe rather than risky, but it's the one site here with neither tests nor live coverage, so flagging it rather than glossing over it.Both are the same one-line change as the site proven three ways above, and klauspost's compatibility is a property of the library, not of a call site.
Risk
No format, config, schema or contract change; output stays standard gzip. No deploy ordering — mixed old/new pods are fine. Revert is one import line per file.
Dependency
operatorwas resolving klauspost transitively at v1.17.6 without declaring it; now declared direct at v1.19.1, matching #1503. Nogo mod tidy— it ignores the workspace and rewrites the graph.🤖 Generated with Claude Code