perf(parser): parallelize the cold Claude parse across worker threads, hardware-adaptively - #1008
Merged
Conversation
Reading, decoding and line-parsing a Claude session JSONL is per-file work that touches nothing shared, so it moves onto worker_threads for a large cold parse. parseClaudeFileFull() is the extracted unit both sides run; a worker runs it against an empty dedup set and returns the result as a JSON string, and the parent installs results in the order the serial loop would. Everything with cross-file state stays on the main thread, and a file whose message ids were already claimed (or whose worker failed) re-parses in-process, so the output is identical to the serial path. Thread count is decided per parse: never with <=2 cores, under 2 GB free memory, fewer than 200 pending whole-file re-parses or under 200 MB behind them, so warm and incremental runs spawn nothing. CODEBURN_PARSE_WORKERS overrides it. The pool is terminated when the parse ends, so the resident serve child accumulates no threads.
Pins the gates that keep threads off low-spec machines and warm runs, that a forced worker count bypasses them, that results come back in submission order, that a dead pool reports failure instead of throwing (and the serial fallback lands on the same result), that no thread outlives a parse across back-to-back parses, and that a cold CLI parse with and without workers produces the same payload and the same cache shards.
os.freemem() reports free pages on macOS, not available memory: on an idle 128 GB machine it reads a few hundred MB, so the 2 GB gate switched the worker pool on and off between runs on the platform the desktop app ships to. The gate and the budget now use process.availableMemory() (cgroup/rlimit-aware in a container), falling back to os.totalmem(): serial under 4 GB available, budget min(0.25 * available, 2 GB). An 8 GB box earns 8 threads, a 4 GB box none. The verbose line now carries every decision input — cores, available GB, pending files and bytes — on both the gate and the go path, so one support log explains itself.
The comment at the install site claimed only that an overlapping worker result 'is discarded'. State why the empty-set result is installable at all — an empty id intersection is proof a serial parse would have dropped nothing — and why the tempting shortcut is wrong: parsedTurnsToCachedTurns delta-encodes gitBranch across turns, so dropping one turn changes whether a LATER turn carries a gitBranch key. Overlap discards the whole file, never individual turns. Tests: the end-to-end determinism check now runs both parses over the SAME corpus, so cache shard BODIES are compared byte for byte instead of just their keys, and a new resumed-session fixture (a transcript restating another file's message ids, in both filename orders) makes install order decide the answer. Verified by mutation: removing the discard guard fails it, and yielding worker results out of order fails it. CODEBURN_VERBOSE now reports how many worker results were re-parsed in-process on id overlap, which is what the new test asserts on. The worker bundle's source map is excluded from the published package (-1.8 MB).
This was referenced Aug 17, 2026
timothybrush
pushed a commit
to timothybrush/codeburn
that referenced
this pull request
Aug 17, 2026
…rker pool Codex is the bigger half of a real cold parse (4 GB of rollouts against 1.8 GB of Claude sessions) and was still decoding one file at a time. A whole-file rollout decode now runs on the getagentseal#1008 pool. parseCodexFileFull is the serial decode with the codex cache switched off: no hit lookup, and the entry it would have written comes back to the parent instead. The worker runs it against an EMPTY dedup set and returns the calls, the keys it claimed, and that entry; the parent installs all three in the serial loop's order, so an empty key intersection is the proof that a serial parse would have dropped nothing either. On overlap the whole file is discarded and re-parsed in-process -- which is what makes a forked rollout safe, since it replays its parent's token_count history under the parent's key namespace and collides outright. Nothing cross-file moves off the main thread: the dedup set, canonical project paths and the codex cache's per-directory state all stay in the parent, and a file the cache can serve exactly or resume into from a byte offset never reaches a worker. The decision is per provider -- the Claude scan and the provider loop run one after the other, so at most one pool is alive -- and the pool is terminated when its scan ends. The workload gate is now files OR bytes rather than both, and the count takes max(files / 50, bytes / 200 MB): a corpus of a few hundred multi-hundred-MB rollouts is as parallelisable as a few thousand small transcripts, and would otherwise have earned one thread or none.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The cold parse (first run, or after a cache-layout/parse-version change) was single-threaded: ~54% of it is per-file work (read, decode, JSONL line parse, turn grouping) that is independent across files. This runs the Claude JSONL per-file half on
worker_threads, and installs results in the parent in the exact order the serial loop would, so cross-file state (message-id dedup, spawn/parent links, PR correlation, canonical paths, progress) stays on the main thread untouched.Equivalence mechanism. A worker parses against an empty dedup set and returns the message ids it claimed. The parent, in serial order, checks the intersection with the global set: empty → the result is provably what serial would have produced → install and merge; any overlap (a resumed session restating another file's history) → discard and re-parse in-process. Worker error or crash → in-process fallback for that file; pool failure → serial. Appends never go off-thread.
Hardware-adaptive policy (
src/parse-workers.ts): gates in order — pending files ≥ 200, pending bytes ≥ 200 MB, cores > 2,process.availableMemory()≥ 4 GB (falls back tototalmem; notos.freemem, which is meaningless on macOS); thenworkers = min(cores − 1, min(0.25 × available, 2 GB) / 256 MB, files / 50). Warm/incremental parses never spawn workers.CODEBURN_PARSE_WORKERS=0|Noverrides. Decision inputs are logged underCODEBURN_VERBOSE=1, plus a summary with the discard count.Workers are created per qualifying parse and terminated in
finally— none outlive a parse or leak in the resident serve. Pricing/alias/override state crosses via a snapshot.dist/parse-worker.jsis a separate bundle entry (its sourcemap is excluded from the package); the Electron staging script copies it. Claude only — Codex (the generic provider loop + resume state) is the follow-up.Measured (real corpus: 15,890 Claude files / 1.8 GB, 16 cores; APFS snapshot so all runs see identical bytes)
CODEBURN_PARSE_WORKERS=0=4=8Per-worker RSS ≈ 20 MB average, ≈ 90 MB worst case on the largest (46 MB) file — well inside the 256 MB budget. 2000 files through one worker: no RSS growth.
Review
Independent adversarial review (no blockers/majors): confirmed the empty-set trick is exact (
msgIds= exactly what serial would have claimed; the only non-subtractive effect of a pre-seeded set —gitBranchdelta-encoding across turns — is why a whole-file discard, never a partial patch, is the fallback; that invariant is now pinned in the comment and a test), lifecycle is fault-tolerant (worker killed mid-parse → fallback, siblings continue; all workers dead → every file serial; no hang), no thread leaks across back-to-back parses, packaging complete (symlinked bin, Electron staging, snap/flathub don't enumerate dist files).Tests
tests/parse-workers.test.ts(9): policy incl. every 0-worker gate and the override; ordering (parseFilesInOrderyields in submission order); worker failure → serial fallback equals worker result; no liveWorkerafter a parse and after back-to-back parses; end-to-end determinism — payload and shard bodies byte-identical at 0 vs 3 workers, including a fixture with cross-file restated message ids in both filename orders (so the discard path runs in CI and install order matters). Verified by mutation: disabling the overlap branch or installing in completion order both fail.tscclean ·npm test2709 passed ·test:locks26 · app 488.