Record what a conversion actually did - #49
Merged
Conversation
The per-file sidecar answers "how do I put this one back?" and is written only where a backup exists. Nothing answered "why was this file not converted?" once the console output had scrolled away - which is the question people actually ask, and the one the ambiguity refusal creates more of. -Journal writes the run whole: every file EC looked at, what its encoding was detected or declared to be, whether the bytes identified it, which encodings competed, what EC decided, what it did, and the file's SHA-256 before and after. Refused and skipped files are in it, because they are the interesting ones. In the GUI it goes through Export report, as a second format alongside the CSV. Three things it does on purpose. It records the encoding the conversion read, not the detector's raw output, and keeps the detector's answer beside it. Those differ whenever somebody named the source encoding, and that difference is the whole of who was responsible for the reading. Getting this right needed a new field: a completed conversion re-labels its entry to the target so a second pass reads the new bytes correctly, so by journal time the effective label describes what the file now is. The first journal built reported a Shift_JIS file as having been read as UTF-8. The label is now captured when it is true rather than derived afterwards. It does not claim more than happened. Sha256After is present only where a file was actually rewritten, so the record can be checked against the disk rather than taken on trust. A -WhatIf run reports its rows as "would be converted"; a journal of one is marked a preview and records those files as decided, not converted. The hashes would have given that away - before and after would match - but a record should not need to be caught out to be read correctly. It costs an extra read per file, so it is opt-in. The plan-bound paths already carry the approved hash and pay nothing. Also adds the release checklist, with the GUI smoke-test matrix and a record template. The evidence for every destructive case is the source file's SHA-256 before and after, not the status message. That test stays manual because automating it would mean weakening the architecture to make Windows Forms drivable - but it is not optional, because "genuinely just UI" was the last description of this codebase that turned out to be wrong. 432 passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
amrali-eg
force-pushed
the
feat/conversion-journal
branch
from
August 27, 2026 03:51
573f50c to
f53d60a
Compare
The manual matrix needs an instrument, and the first one was wrong in a way worth writing down. It used a single folder and one final check for the whole sequence. That cannot work: the stale-plan case stops the entire run, so every "must have converted" expectation after it is unreachable by construction - and the state it leaves is byte-identical to "the tester cancelled everything", so the result cannot say which protection fired. Its first real run reported FAIL, and only reading the bytes by hand showed the product had been correct throughout. So the harness is four independent phases, each its own folder, click sequence and check, each proving one property. Every phase verifier was reproduced through the CLI before being handed over, and each was also checked to fail when it should - a verifier that cannot fail is the same defect as a test that silently asserts nothing. Phase C refuses to pass at all when the edit that makes the plan stale was never made, which is the failure mode a manual matrix is most prone to. Records the 2026-08-27 run on a201a08: all four phases pass. Two notes kept with it because they qualify what the phases establish. Phase B's French sample decodes identically under windows-1252 and iso-8859-1, so its text assertion cannot show which codec was used; what settles it is the recovery record's DetectedCodePage of 1252. And TextEquivalent is nearly unreachable - eight ASCII shapes all classify as StructurallyDetermined, and only a one-byte file reaches it - so the corpus has to be contrived to exercise that class at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase F — the conversion journal
The per-file
.ecmeta.jsonsidecar answers "how do I put this one back?" and is written only where a backup exists. Nothing answered "why was this file not converted?" once the console output scrolled away — which is the question people actually ask, and the one the ambiguity refusal creates more of.-Journal <path>records the run whole. In the GUI it goes through Export report as a second format alongside the CSV.{ "RelativePath": "notes.txt", "Sha256Before": "f19e0e0c…", "Sha256After": null, "DetectionMode": "Detected", "DetectedEncoding": "iso-8859-1", "SourceEncoding": "iso-8859-1", "Ambiguity": "TextChanging", "AmbiguityReason": "MultipleCodecsDifferentText", "DetectionCandidates": ["cp866", "ibm852", "ibm855", "…"], "PlannedAction": "Refuse", "Status": "Refused", "Reason": "The encoding could not be determined uniquely…" }Every field you listed is there, plus
Semantics/SemanticsVersion,AppliedPlan,Surface, and aSummarywhose counts sum to the run.It records the decision executed, not the detector's output
SourceEncodingis what the conversion actually read the file as;DetectedEncodingkeeps the detector's answer beside it. Those differ whenever somebody named the source encoding, and that difference is the whole of who was responsible for the reading.Getting this right needed a new field, and a smoke test caught it. A completed conversion re-labels its entry to the target so a second pass reads the new bytes correctly — so by journal time the effective label describes what the file now is. The first journal I built reported a Shift_JIS file as having been read as
utf-8. The label is now captured when it is true rather than derived afterwards.It does not claim more than happened
Sha256Afteris present only where a file was actually rewritten, so the record can be checked against the disk rather than taken on trust.A second honesty bug, also caught before merge: a
-WhatIfrun reports its rows as "would be converted", and the journal was copying that through asStatus: Convertedfor a directory nothing had touched. The hashes would have given it away — before and after would match — but a record should not need to be caught out to be read correctly. A preview journal is now marked"Preview": trueand records those files asNotAttempted, i.e. decided but deliberately not carried out.Cost
One extra read per file, so it is opt-in and rejected with
-DetectOnly/-Validate(use-Reportfor those). The plan-bound paths already carry the approved hash and pay nothing.10 new tests, 432 passing
Believed/decided/written recorded together; the read-as encoding rather than the became encoding; refusals with their competing encodings and hashes proving the file is untouched; explicit vs detected distinguished; every file accounted for; semantics recorded;
-Applynaming the plan it carried out; a failed conversion recorded asFailed, notRefused— both leave the file alone, and the difference is the whole of what a reader needs.Release checklist
RELEASE-CHECKLIST.mdcarries the GUI smoke-test matrix — your ten steps, plus the.bak-is-a-directory case and a journal-export check — and the record template.The evidence for every destructive case is the source file's SHA-256 before and after, not the status message.
It also says why the test stays manual: automating it would mean weakening the architecture to make Windows Forms drivable, which trades a real safety property for a test. And why it is not optional: "genuinely just UI" was the last description of this codebase that turned out to be wrong.
🤖 Generated with Claude Code