Skip to content

build: pin ko base images by digest - #1500

Open
Grant McCloskey (MushuEE) wants to merge 1 commit into
agent-substrate:mainfrom
MushuEE:build/pin-ko-base-images
Open

build: pin ko base images by digest#1500
Grant McCloskey (MushuEE) wants to merge 1 commit into
agent-substrate:mainfrom
MushuEE:build/pin-ko-base-images

Conversation

@MushuEE

Copy link
Copy Markdown
Collaborator

Fixes #1498

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Every base image in .ko.yaml was referenced by tag only, so two builds of
the same commit could produce different images depending on when they ran.

  • Pin all three by digest, keeping the tag alongside so a bump can re-resolve it.
  • alpinealpine:3.24 and debian:stable-slimdebian:13-slim: the same
    releases the floating tags resolve to today, but a routine bump now stays
    within a major; moving majors becomes a deliberate edit.
  • Distroless only publishes latest/nonroot, so it stays latest + digest.
  • Add hack/verify/ko-base-images.sh (picked up by verify-all.sh) which fails
    any base ref without a digest.

How the digests were resolved

Each is the multi-arch index digest for the tag (what ko needs, since it
builds linux/amd64 and linux/arm64 from one base), resolved on 2026-09-04
with the crane that ships in the go-containerregistry version ko already
pins — no new dependency:

cd hack/tools/ko
go run github.com/google/go-containerregistry/cmd/crane digest <ref>
Pinned ref Digest
gcr.io/distroless/static-debian13:latest sha256:f2ea2709ac8db56323cbd7d014277f32cb572d9ea124b0076f7aafe5980678fe
alpine:3.24 sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b
debian:13-slim sha256:d7e12182ce18b85b93007c1dedf31f2d29e01ccf3182cc4017c709b6259bc132

Cross-checks at resolve time: alpine:latest, alpine:3, and alpine:3.24
all resolve to the same digest, so the tag change does not change the image.
debian:13-slim and debian:trixie-slim are the same digest;
debian:stable-slim currently also points at Debian 13. Anyone can re-run
the command above to confirm (the digests will match until upstream
publishes a rebuild, which is what the bump job will pick up).

Verified locally

  • hack/verify/ko-base-images.sh passes, and fails correctly when a digest
    is removed from any ref.
  • hack/verify/boilerplate.sh passes.
  • ko build --push=false of cmd/podcertcontroller, cmd/ateom-microvm,
    and demos/sandbox (one per base) resolves all three pinned bases and
    builds.
  • The kind e2e lane builds every image on these bases.

Follow-up: a scheduled job to re-resolve the digests weekly.

Every base image in .ko.yaml was referenced by tag only, so two builds of
the same commit could produce different images depending on when they ran.
debian:stable-slim additionally moves across Debian majors on its own.

Pin all three by digest, keeping the tag alongside for readability and so a
bump can re-resolve it. alpine and debian move to explicit major tags
(alpine:3.24, debian:13-slim, both the same digests their floating tags
resolve to today) so a routine bump never crosses a major; distroless only
publishes latest/nonroot, so it stays latest plus digest.

Add hack/verify/ko-base-images.sh, picked up by hack/verify-all.sh, which
fails on any base ref without a digest.

Fixes agent-substrate#1498
Comment thread .ko.yaml
# limitations under the License.

defaultBaseImage: gcr.io/distroless/static-debian13
# Every base image is pinned by digest so that two builds of the same commit

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need the comments to justify pinning by SHA

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, we pin by sha for most images in the repo without any comment.

Also, are we sure these digests point at the tag we specified, if an agent generated this?

This is the sort of PR I would 100% hand generate. You can use crane digest on the intended image for TOFU.

@@ -0,0 +1,53 @@
#!/usr/bin/env bash

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need a defensive script for this. PR review should be sufficient. These cost time on every PR and we're already tight on CI resources.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pin ko base images by digest

4 participants