Write a self-describing record beside every backup - #44
Merged
Conversation
An audit measured 99.2% of bad conversions as byte-recoverable, then noticed the qualifier that made the figure hollow: recovery needs the codec the file was read with, and that lived solely in the conversion report. A GUI user has no reason to keep one. "Recoverable" was true and unusable. Each backup now carries a sidecar answering, on its own, how the original is restored and how the conversion can be reconstructed: original path, size and SHA-256; the backup's SHA-256; the detected encoding *and its code page*; BOM state; target encoding and code page; text hashes before and after; scalar count; tool version, timestamp, conversion id; and a schema version. The code page is there because a name is not an identifier. "cp949" and "ks_c_5601-1987" are one encoding and only the number says so - a lesson this project already learned when comparing detector output by label scored spelling disagreements as errors. Ordering is the part that matters. ConversionOptions.RecordConversion is invoked from inside Convert, after verification succeeds and before AtomicReplace. The backup exists, the conversion is proven, and the original is still in place, so a failure to record aborts with nothing lost and nothing needing recovery. Writing the record after installation would have reproduced the same gap in a new form. The record is also read back after writing, since a file that was written but cannot be parsed would leave a caller believing the conversion is reversible when it is not. Restore availability is a verified state rather than an inference from a ".bak" existing: Available, BackupMissing, MetadataMissing, MetadataUnreadable, or BackupCorrupted. The backup is hashed and checked against both the recorded backup hash and the recorded original hash - different questions, since the first says the backup is the file described and the second says that file is the original. A sidecar is written only when a backup exists, since without one there is nothing to restore from. A stale ".bak" from an earlier run is refused rather than recorded as this conversion's original. Chosen as a JSON sidecar over an NTFS alternate data stream: an ADS is lost by ordinary copying, archiving and cloud sync, which are exactly the operations that separate a backup from its origin. 330 tests passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The audit measured 99.2% of bad conversions as byte-recoverable — then found the qualifier that made the figure hollow: recovery needs the codec the file was read with, and that lived solely in the conversion report. A GUI user has no reason to keep one. "Recoverable" was true and unusable.
Every backup now carries a sidecar that answers, on its own, how to restore the original and reconstruct the conversion:
{ "MetadataVersion": 1, "OriginalSha256": "cd20729b…", "BackupSha256": "cd20729b…", "DetectedEncoding": "windows-1252", "DetectedCodePage": 1252, "TargetEncoding": "utf-8", "TargetCodePage": 65001, "SourceTextSha256": "ded8dc8b…", "OutputTextSha256": "ded8dc8b…", "ECVersion": "3.7.0.0", "ConversionId": "…", "ConversionTimestampUtc": "…" }The code page is recorded because a name is not an identifier:
cp949andks_c_5601-1987are one encoding and only the number says so. This project already learned that lesson when comparing detector output by label scored spelling disagreements as detection errors.Ordering is the part that matters
RecordConversionis invoked from insideConvert, after verification succeeds and beforeAtomicReplace. At that moment the backup exists, the conversion is proven, and the original is still in place — so a failure to record aborts with nothing lost and nothing needing recovery. Writing the record after installation would have reproduced the same gap in a new form.The record is also read back after writing: a file that was written but cannot be parsed would leave a caller believing the conversion is reversible when it isn't.
Restore is verified, not inferred
RestoreAvailabilitydistinguishes Available, BackupMissing, MetadataMissing, MetadataUnreadable, BackupCorrupted — because a.bakexisting is not evidence anything can be recovered.The backup is hashed and checked against both the recorded backup hash and the recorded original hash. Those are different questions: the first says the backup is the file described, the second says that file is the original.
A stale
.bakfrom an earlier run is refused rather than recorded as this conversion's original.Sidecar, not an alternate data stream
An ADS is lost by ordinary copying, archiving and cloud sync — exactly the operations that separate a backup from its origin. A sidecar survives them and can be read without EncodingChecker.
The sidecar is excluded from scanning alongside
.bak, so a later run never treats its own output as input.330 tests passing (was 320). Ten new tests, including one that reverses a real conversion using the sidecar alone.
🤖 Generated with Claude Code