perf(parser): parallelize the cold Codex rollout parse on the same worker pool - #1010
Merged
Conversation
…rker pool Codex is the bigger half of a real cold parse (4 GB of rollouts against 1.8 GB of Claude sessions) and was still decoding one file at a time. A whole-file rollout decode now runs on the #1008 pool. parseCodexFileFull is the serial decode with the codex cache switched off: no hit lookup, and the entry it would have written comes back to the parent instead. The worker runs it against an EMPTY dedup set and returns the calls, the keys it claimed, and that entry; the parent installs all three in the serial loop's order, so an empty key intersection is the proof that a serial parse would have dropped nothing either. On overlap the whole file is discarded and re-parsed in-process -- which is what makes a forked rollout safe, since it replays its parent's token_count history under the parent's key namespace and collides outright. Nothing cross-file moves off the main thread: the dedup set, canonical project paths and the codex cache's per-directory state all stay in the parent, and a file the cache can serve exactly or resume into from a byte offset never reaches a worker. The decision is per provider -- the Claude scan and the provider loop run one after the other, so at most one pool is alive -- and the pool is terminated when its scan ends. The workload gate is now files OR bytes rather than both, and the count takes max(files / 50, bytes / 200 MB): a corpus of a few hundred multi-hundred-MB rollouts is as parallelisable as a few thousand small transcripts, and would otherwise have earned one thread or none.
A mixed Claude + Codex corpus with forked rollouts in both creation orders, asserting an identical payload, byte-identical cache shards and a byte-identical codex-results.json between CODEBURN_PARSE_WORKERS=0 and =3, with the codex discard count pinned above zero so the overlap path is really exercised. Plus: a resumable rollout never reaching a worker (the decision line reports no full parses pending after an append), the off-thread decode matching parseCodexFileFull exactly including the cache entry it hands back, no cache file written by the decode itself, no leaked threads, and the files-OR-bytes gate.
…t per parse Three fixes from review, all measured on this box. The workload gate was files OR bytes. The files arm is wrong: 250 pending files holding 117 KB between them spawned 5 threads and ran ~5% SLOWER than serial, and a file count only starts paying for itself around 400. Gate on bytes alone; the count still takes max(files / 50, bytes / 200 MB), so a few hundred huge rollouts keep their threads. The flat 256 MB per-worker memory budget was contradicted by the Codex workload: a 260 MB rollout peaks near 430 MB in its worker, linearly across the pool. It is now derived per parse as clamp(256 MB, 2 x average pending file + 128 MB, 1 GB), which leaves a corpus of small Claude transcripts where it was and stops over-subscribing on rollouts. The parent's buffer of up to pool.size finished results is part of that peak and is named in the comment. The worker/file pairing at both install sites was positional, guarded only by position (Claude) or a path membership check (Codex). Each worker now echoes its path and the parent asserts it, outside the per-file try: a misalignment would install one session's turns under another's path -- a wrong number nobody would ever notice -- so it fails the run rather than being swallowed as a parse failure. On the Claude side that meant hoisting the whole worker-result block above the try, which is safe because an append never consumes a result in either its shortcut or its straddled-fallthrough case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extends #1008's worker-thread cold parse to Codex rollouts (the larger half of many corpora), reusing the same pool, policy and install pattern.
parseCodexFileFull(source, seenKeys)is the existing CodexcreateParserwith acaptureargument: no cache-hit lookup (so never a resume — resume/exact-hit files stay in-process, decided by a pre-pass with the loop's exact eligibility predicate and order), and the codex-cache entry comes back throughcaptureinstead of being written. One decoder, no second code path.{path, calls, write?, keys}. Parent, in the serial loop's order:keys ∩ parserDedupempty → install calls, merge keys,writeCachedCodexResultsinto the parent's memCache (workers never touch codex-cache globals); overlap (a forked rollout replaying parent history) → discard and re-parse in-process. Worker failure → in-process fallback. Both install sites (Claude and Codex) now assert the echoedpathmatches the expected file and fail the run on mismatch.codex:${forkedFromId||sessionId}:${cumulativeTotal}:…), which is fully expressible as claimed keys; every other counter is a per-file parser local andcodex.tshas no module-level mutable state.Policy changes (
src/parse-workers.ts): gate on pending bytes ≥ 200 MB alone (the previous files-count arm made 250-tiny-file corpora ~5% slower than serial — measured; now they stay serial); worker countmax(files/50, bytes/200 MB)capped by cores−1 and the memory budget; per-worker RSS budget derived per parse —clamp(256 MB, 2 × avgFileBytes + 128 MB, 1 GB)— because a Codex worker peaks ~430 MB on a 260 MB rollout (measured) while a flat raise would have halved the Claude pool. Per-provider decision; at most one pool alive.Measured (real corpus: 15,892 Claude files / 1.8 GB + 940 Codex rollouts / 4.0 GB, APFS snapshot, 16 cores)
CODEBURN_PARSE_WORKERS=0workers=0 (below 210 MB pending)Identity vs serial at 4/8/auto: payload, all 10 cache shard bodies, and the 58 MB
codex-results.jsonbyte-identical.Review
Independent adversarial review: equivalence could not be broken (dedup key claim proven complete: the only
seenKeys.addsites on the full-decode path are returned in full; forked-rollout fixture in both filename orders with replays past the 5 s cutoff so the discard path runs; mutation-checked — disabling the overlap branch or reversing install order fails the tests). Its two findings (files-only gate pessimization; per-worker budget too low for Codex) are fixed above.Tests
tests/parse-workers.test.ts13: mixed Claude+Codex determinism e2e (payload + shard bodies +codex-results.jsonbyte-identical at 0 vs 3 workers, discard count > 0), bytes-only gate cases (250/917 KB → 0; 5000/10 MB → 0; 150/4 GB → 8; 250/65 GB → 3 via the derived budget), worker failure → serial fallback, resume-class files never off-thread, no leaked workers, echoed path assertions.tscclean ·npm test2713 ·test:locks26 · app 488.