Skip to content

refactor(zip): make the archive's read callback re-entrant - #775

Merged
andiwand merged 1 commit into
mainfrom
refactor/zip-concurrent-reads
Aug 29, 2026
Merged

refactor(zip): make the archive's read callback re-entrant#775
andiwand merged 1 commit into
mainfrom
refactor/zip-concurrent-reads

Conversation

@andiwand

Copy link
Copy Markdown
Member

🤖 Generated with Claude Code

Stacked on #774 — review that first; the base flips to main when it merges.

util::Archive carried a mutable std::mutex that every operation took:
is_file, path, method, size, opening a stream, and each 4 KiB
underflow. It existed for one reason — the read callback drove a single shared
std::istream with seekg/read, so two readers fought over its position.

miniz needs none of that. mz_zip_reader_extract_iter_new heap-allocates state
per iterator, and all I/O goes through a pread-shaped
m_pRead(opaque, absolute_offset, buf, n), so the callback only has to be
re-entrant. ReadSource makes it so:

  • a memory-backed file is read straight out of its buffer, with no lock at all;
  • anything else takes a stream off a free list, reads, and puts it back — the
    lock covers the list, never the read.

The archive-wide mutex goes with it, so the metadata calls stop serialising too.
mz_zip_archive::m_last_error stays racy, which miniz documents; we never read
it.

Verification

  • ZipArchive.concurrent_entry_reads reads eight 64 KiB entries from eight
    threads, over a DiskFile and a MemoryFile.
  • It has teeth: pointed at a single unguarded shared stream it fails hard with
    FileNotFound: zip entry not found 6, not flakily.
  • Full suite: 1253 passed, 6 skipped, 0 failed, reference output unchanged.
  • No ThreadSanitizer run — a TSan build of odr_test segfaults at startup on
    macOS arm64 with this toolchain, before reaching any test.

Context: #767, which concludes we stay on miniz. This is the concurrency win
that issue wanted; no dependency swap delivers it, and libzip would have made it
worse (its docs tell you to open the archive once per thread).

Base automatically changed from chore/miniz-3.1.1 to main August 29, 2026 20:46
`util::Archive` carried a `mutable std::mutex` that every operation took —
`is_file`, `path`, `method`, `size`, opening a stream, and each 4 KiB
`underflow`. It existed for one reason: the read callback drove a single shared
`std::istream` with `seekg`/`read`, so two readers would fight over its
position.

miniz needs none of that. It keeps decompression state per iterator and
addresses the input by absolute offset, so the callback only has to be
re-entrant. `ReadSource` makes it so: a memory-backed file is read straight out
of its buffer with no lock, and anything else takes a stream from a free list,
reads, and puts it back — the lock covers the list, never the read.

The archive-wide mutex is gone with it, so the metadata calls no longer
serialise either. `mz_zip_archive::m_last_error` stays racy, as miniz documents;
we never read it.

`concurrent_entry_reads` reads eight 64 KiB entries from eight threads, over a
disk file and a memory file. Against a shared unguarded stream it fails hard
with `FileNotFound`, so it has teeth.
@andiwand
andiwand force-pushed the refactor/zip-concurrent-reads branch from 74c9827 to e4906f6 Compare August 29, 2026 20:48
@andiwand
andiwand merged commit 8e3f22b into main Aug 29, 2026
32 checks passed
@andiwand
andiwand deleted the refactor/zip-concurrent-reads branch August 29, 2026 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant