build: pin ko base images by digest - #1500
Open
Grant McCloskey (MushuEE) wants to merge 1 commit into
Open
Conversation
Every base image in .ko.yaml was referenced by tag only, so two builds of the same commit could produce different images depending on when they ran. debian:stable-slim additionally moves across Debian majors on its own. Pin all three by digest, keeping the tag alongside for readability and so a bump can re-resolve it. alpine and debian move to explicit major tags (alpine:3.24, debian:13-slim, both the same digests their floating tags resolve to today) so a routine bump never crosses a major; distroless only publishes latest/nonroot, so it stays latest plus digest. Add hack/verify/ko-base-images.sh, picked up by hack/verify-all.sh, which fails on any base ref without a digest. Fixes agent-substrate#1498
Grant McCloskey (MushuEE)
force-pushed
the
build/pin-ko-base-images
branch
from
September 4, 2026 22:05
df090bb to
5aae068
Compare
Grant McCloskey (MushuEE)
requested review from
Aditya Shantanu (aditya-shantanu) and
Anna Pendleton (annapendleton)
September 4, 2026 22:05
Anna Pendleton (annapendleton)
approved these changes
Sep 4, 2026
Taahir Ahmed (ahmedtd)
approved these changes
Sep 4, 2026
| # limitations under the License. | ||
|
|
||
| defaultBaseImage: gcr.io/distroless/static-debian13 | ||
| # Every base image is pinned by digest so that two builds of the same commit |
Collaborator
There was a problem hiding this comment.
I don't think we need the comments to justify pinning by SHA
Collaborator
There was a problem hiding this comment.
+1, we pin by sha for most images in the repo without any comment.
Also, are we sure these digests point at the tag we specified, if an agent generated this?
This is the sort of PR I would 100% hand generate. You can use crane digest on the intended image for TOFU.
| @@ -0,0 +1,53 @@ | |||
| #!/usr/bin/env bash | |||
Collaborator
There was a problem hiding this comment.
I don't think we need a defensive script for this. PR review should be sufficient. These cost time on every PR and we're already tight on CI resources.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1498
Every base image in
.ko.yamlwas referenced by tag only, so two builds ofthe same commit could produce different images depending on when they ran.
alpine→alpine:3.24anddebian:stable-slim→debian:13-slim: the samereleases the floating tags resolve to today, but a routine bump now stays
within a major; moving majors becomes a deliberate edit.
latest/nonroot, so it stayslatest+ digest.hack/verify/ko-base-images.sh(picked up byverify-all.sh) which failsany base ref without a digest.
How the digests were resolved
Each is the multi-arch index digest for the tag (what ko needs, since it
builds
linux/amd64andlinux/arm64from one base), resolved on 2026-09-04with the
cranethat ships in the go-containerregistry version ko alreadypins — no new dependency:
gcr.io/distroless/static-debian13:latestsha256:f2ea2709ac8db56323cbd7d014277f32cb572d9ea124b0076f7aafe5980678fealpine:3.24sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8bdebian:13-slimsha256:d7e12182ce18b85b93007c1dedf31f2d29e01ccf3182cc4017c709b6259bc132Cross-checks at resolve time:
alpine:latest,alpine:3, andalpine:3.24all resolve to the same digest, so the tag change does not change the image.
debian:13-slimanddebian:trixie-slimare the same digest;debian:stable-slimcurrently also points at Debian 13. Anyone can re-runthe command above to confirm (the digests will match until upstream
publishes a rebuild, which is what the bump job will pick up).
Verified locally
hack/verify/ko-base-images.shpasses, and fails correctly when a digestis removed from any ref.
hack/verify/boilerplate.shpasses.ko build --push=falseofcmd/podcertcontroller,cmd/ateom-microvm,and
demos/sandbox(one per base) resolves all three pinned bases andbuilds.
Follow-up: a scheduled job to re-resolve the digests weekly.