feat(snapshot s3): fingerprint buckets from metadata - #1069
Draft
mbevc1 wants to merge 7 commits into
Draft
Conversation
Contributor
|
Claude finished @mbevc1's task in 2m 42s —— View job Review —
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
VirtualDirSha256 returns the fingerprint DirSha256 would produce for a
directory containing a given set of files, from (path, content sha256)
pairs alone, without touching the disk. SingleVirtualFile reports the
one-file case that callers fingerprint with the file's own digest.
This lets a caller that can obtain per-file content digests some other
way -- an object store that already stores them, say -- reproduce an
artifact fingerprint without downloading and hashing every file.
The tree is built before sorting, deliberately. Object stores list keys
in byte order of the whole key, and '.' (0x2E) sorts before '/' (0x2F),
so keys "a.txt" and "a/z" list as [a.txt, a/z] while filepath.WalkDir
yields [a, a/z, a.txt]. Sorting the flat path list would diverge from
DirSha256 whenever a directory shares a name prefix with a sibling file.
Paths that cannot be mapped onto a tree unambiguously are rejected
rather than silently collapsed: unclean paths ("a//b", "..", absolute),
duplicates, and names used as both a file and a directory.
Every fingerprint test materialises the tree on disk, fingerprints it
with DirSha256, and requires VirtualDirSha256 of the same pairs to match,
so the two implementations cannot drift.
Fingerprinting a bucket from the checksums S3 already stores needs object metadata, not object content. Add S3HeadAPI to the S3API composite, backed by the same *s3.Client that already serves listing, and give FakeS3Client a HeadObject to match. S3MetadataAPI pairs listing with metadata and deliberately leaves out S3DownloadAPI, so a function typed against it cannot read object content even by accident. The fake models stored checksums sparsely, via a Checksums map rather than deriving them from object bytes: an object uploaded without an explicit checksum algorithm has none, and that is the common case a caller has to handle. It also withholds the checksum unless the request sets ChecksumMode, exactly as S3 does -- a fake that always returned it would hide a caller that forgets to ask. The contract tests gain a sha256ChecksumKey parameter and cover metadata retrieval, the missing-key error, and both sides of the ChecksumMode behaviour. They skip when no checksum-bearing object is available, which is the case for kosli-cli-public today: adding one there would change the golden fingerprints TestGetS3Data pins.
…tadata kosli snapshot s3 downloads every matching object into a temp dir and hashes it. For a large bucket that is a full egress, local disk equal to the bucket size, and a SHA256 pass over every byte. --fingerprint-source metadata reads the SHA256 checksum S3 already stores for each object instead, via HeadObject with ChecksumMode enabled. The fingerprint is identical to the downloaded one, so a snapshot still matches the artifact that was attested. This slice covers the one-object case; combining several objects into a directory fingerprint follows, and until then a multi-object selection fails with a message that says how to narrow it. Note what this does not buy: AWS requires s3:GetObject to read an object's checksum, the same permission downloading needs, so the saving is egress, disk and CPU rather than permissions. The flag help says so rather than implying otherwise. Objects must carry a full-object SHA256 checksum. A composite (multipart) checksum hashes the part checksums rather than the object, so it is rejected on both signals S3 gives -- the COMPOSITE type and the "-N" suffix, the latter being what the SDK's own response validation keys off. The error points at copy-object, which collapses a multipart object into a single part in place and needs no access to the original file. Both modes now list through listMatchingS3Objects, so what a filter selects cannot drift between them, and decodeLambdaFingerprint becomes decodeBase64Sha256 now that Lambda is not its only caller.
Combine the per-object checksums with digest.VirtualDirSha256, which reproduces the directory fingerprint the download path produces, so --fingerprint-source metadata now covers a whole bucket rather than a single object. One object is still fingerprinted as that file and named after it; several are fingerprinted as a directory named after the bucket, matching what containsSingleFile decides once the objects are on disk. A bucket-root .kosli_ignore is rejected rather than ignored. Its rules change the fingerprint, and applying them would mean downloading the file. Silently skipping them would produce a fingerprint that quietly differs from the downloaded one, which is worse than refusing. Only a root one matters: DirSha256 reads exactly one, at the artifact root, and a nested .kosli_ignore is an ordinary file to it. A lone .kosli_ignore is fine, because a one-object snapshot is fingerprinted as a file and the ignore rules never come into play. The equality tests now cover multi-object trees, including a prefix that shares a name prefix with a sibling object -- the case where sorting the flat key list instead of the tree would diverge, because '.' sorts before '/' in a byte-ordered listing.
Metadata mode made one HeadObject call per object, in sequence, so a thousand-object bucket paid a thousand round trips end to end. Fetch them concurrently instead, bounded by a semaphore. Unlike GetLambdaPackageData, which spawns one goroutine per function, this cannot be unbounded: an account holds tens of functions but a bucket can hold millions of objects. 16 in flight stays well under S3's per-prefix request ceiling, and the adaptive retryer already spreads throttling across the batch. Results go into preallocated slots rather than an appended slice, so the digest order follows the listing however the requests interleave. The same bucket then always fingerprints to the same value, and a failure is reproducible rather than depending on which goroutine won. Errors are split by what the user can do about them. A transport or permission failure aborts immediately, since every remaining request would fail the same way. An object whose checksum is missing or composite is collected instead, and one run reports every object that needs fixing -- capped at ten keys plus a count -- so migrating a bucket is not a guess-and-retry loop.
Spell out what --fingerprint-source metadata does and, more importantly, what it does not do. The obvious assumption is that reading metadata instead of object content needs weaker permissions; AWS requires s3:GetObject either way, so the help says so plainly rather than letting readers infer a benefit that is not there. The three conditions that make a bucket unusable in this mode -- objects without a stored checksum, composite multipart checksums, and a root .kosli_ignore -- are documented alongside the fix for each, so a reader can tell before running whether their bucket qualifies.
… error Two points from review. Unusual but legal object keys -- "a//b", or a leading-slash "/foo" -- are rejected by metadata mode while the content source quietly collapses them via filepath.Join, where "a//b" can even land on top of a real "a/b" object. Refusing beats a silently wrong fingerprint, so the behaviour stays, but a bucket that content mode appears to handle will now fail loudly under metadata. Say so in the help text, and cover both key shapes in the tests so the documented behaviour is pinned rather than described. The .kosli_ignore error also over-claimed. It said excluding the file with --exclude "would not help: the fingerprint would still differ", which is not true of the comparison that matters: with the file excluded, neither source applies its rules, so the two agree. What actually differs is that the resulting fingerprint covers the unfiltered bucket content rather than the content the rules were written to select. The message now says that, and an equality test pins the claim instead of leaving it as prose.
mbevc1
force-pushed
the
20260801_s3_metadata
branch
from
August 1, 2026 22:35
8aa8a82 to
e5e7859
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
--fingerprint-source content|metadatatokosli snapshot s3.Today the command downloads every matching object into a temp dir and hashes it
(
getS3DataFromClient). For a large bucket that means full egress, local disk equal to thebucket size, and a SHA256 pass over every byte. With
--fingerprint-source metadatathecommand reads the SHA256 checksum S3 already stores for each object instead, via
HeadObjectwith
ChecksumModeenabled — no download, no temp disk, no local hashing.Default is
content, so existing behaviour is unchanged.The fingerprint is identical, and that is the load-bearing property
A snapshot is only useful if its fingerprint matches the artifact that was attested, so
metadata mode has to reproduce the existing digest exactly rather than invent its own.
digest.VirtualDirSha256reconstructs whatDirSha256produces from(path, sha256)pairsalone. Every fingerprint test materialises the tree on disk, hashes it with
DirSha256, andrequires
VirtualDirSha256of the same pairs to match — the two cannot drift silently. Thesame equality assertion runs at the aws layer against the real download path.
One subtlety worth a reviewer's attention: the tree must be built before sorting. S3 lists
keys in byte order of the whole key, and
.(0x2E) sorts before/(0x2F), so keysa.txtand
a/zlist as[a.txt, a/z]whilefilepath.WalkDiryields[a, a/z, a.txt]. Sorting theflat key list would pass most tests and be wrong exactly when a prefix shares a name prefix
with a sibling file. There is a test for that case.
Conditions a bucket must meet
upload asked for it (
aws s3api put-object --checksum-algorithm SHA256).the object content, so they are not the object's digest. Detection keys off the observed
COMPOSITEtype and the-Nsuffix — the latter being what the SDK's own responsevalidation uses — rather than off "was this a multipart upload", so a full-object SHA256 is
accepted however it was uploaded. The error points at
aws s3api copy-object --checksum-algorithm SHA256, which collapses a multipart object into a single part in placeand needs no access to the original file.
.kosli_ignoreis rejected, not ignored. Its rules change the fingerprintand applying them would mean downloading it; silently skipping them would produce a digest
that quietly differs from content mode. A nested
.kosli_ignoreis an ordinary object, and alone one is fine (a one-object snapshot is fingerprinted as a file).
Objects that cannot be fingerprinted are reported together — every offending key in one run,
capped at ten plus a count — so migrating a bucket is not a guess-and-retry loop.
Checklist
charts/k8s-reporter/) updated, if needed. Note: these changes live in a separate PR