From f53d60a393fb08d29f37e0d49b46dab939e596bc Mon Sep 17 00:00:00 2001 From: amrali-eg <32075105+amrali-eg@users.noreply.github.com> Date: Thu, 27 Aug 2026 05:47:46 +0300 Subject: [PATCH 1/2] Record what a conversion actually did The per-file sidecar answers "how do I put this one back?" and is written only where a backup exists. Nothing answered "why was this file not converted?" once the console output had scrolled away - which is the question people actually ask, and the one the ambiguity refusal creates more of. -Journal writes the run whole: every file EC looked at, what its encoding was detected or declared to be, whether the bytes identified it, which encodings competed, what EC decided, what it did, and the file's SHA-256 before and after. Refused and skipped files are in it, because they are the interesting ones. In the GUI it goes through Export report, as a second format alongside the CSV. Three things it does on purpose. It records the encoding the conversion read, not the detector's raw output, and keeps the detector's answer beside it. Those differ whenever somebody named the source encoding, and that difference is the whole of who was responsible for the reading. Getting this right needed a new field: a completed conversion re-labels its entry to the target so a second pass reads the new bytes correctly, so by journal time the effective label describes what the file now is. The first journal built reported a Shift_JIS file as having been read as UTF-8. The label is now captured when it is true rather than derived afterwards. It does not claim more than happened. Sha256After is present only where a file was actually rewritten, so the record can be checked against the disk rather than taken on trust. A -WhatIf run reports its rows as "would be converted"; a journal of one is marked a preview and records those files as decided, not converted. The hashes would have given that away - before and after would match - but a record should not need to be caught out to be read correctly. It costs an extra read per file, so it is opt-in. The plan-bound paths already carry the approved hash and pay nothing. Also adds the release checklist, with the GUI smoke-test matrix and a record template. The evidence for every destructive case is the source file's SHA-256 before and after, not the status message. That test stays manual because automating it would mean weakening the architecture to make Windows Forms drivable - but it is not optional, because "genuinely just UI" was the last description of this codebase that turned out to be wrong. 432 passing. Co-Authored-By: Claude Opus 5 --- README.md | 43 +++ RELEASE-CHECKLIST.md | 91 +++++ .../ConversionJournalTests.cs | 288 +++++++++++++++ sources/EncodingChecker/ConversionJournal.cs | 340 ++++++++++++++++++ sources/EncodingChecker/ConversionReport.cs | 27 ++ sources/EncodingChecker/MainForm.cs | 66 +++- sources/EncodingChecker/Program.cs | 86 +++++ sources/EncodingChecker/ScanEngine.cs | 29 ++ 8 files changed, 962 insertions(+), 8 deletions(-) create mode 100644 RELEASE-CHECKLIST.md create mode 100644 sources/EncodingChecker.Tests/ConversionJournalTests.cs create mode 100644 sources/EncodingChecker/ConversionJournal.cs diff --git a/README.md b/README.md index 543a270..3e7b924 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ Each [release](https://github.com/amrali-eg/EncodingChecker/releases) publishes - Lossless, safe conversion: every write is verified afterward by comparing a SHA-256 hash of the decoded content, so a silent encoder substitution (e.g. an unrepresentable character) is caught and reported as an error instead of corrupting the file. - Refuses to convert files whose encoding the bytes do not determine, naming the encodings actually in conflict, with `-From` (or the GUI's source-encoding selection) to supply the answer yourself. One policy engine decides for every surface. - The GUI confirms before writing, showing exactly what will happen to each file and carrying out that same plan rather than re-deciding. +- `-Journal` records what a run actually did — including the files it refused and why — with each file's SHA-256 before and after. - `-Plan`/`-Apply` preflight: review what a conversion would do, then carry out exactly that — the plan is bound to the files' hashes and is refused whole if they change. - Optional `.bak` backup before overwriting, and a `-WhatIf` dry-run mode that reports what would happen without touching any file. - Covered by an xUnit test suite exercising the detection/conversion engine, CLI argument parsing, and CSV report formatting across multilingual content and edge cases. @@ -70,6 +71,47 @@ easily hold refused files in different encodings — Cyrillic in koi8-r beside F windows-1252 — and one answer settles only the files it was given about. Imposing it on the rest would repeat, one level up, the mistake the refusal exists to prevent. +### The conversion journal + +`-Journal ` writes a JSON record of the run. For every file: what its encoding was +detected or declared to be, whether the bytes identified it, which encodings competed, +what EC decided, what it actually did, and the file's SHA-256 before and after. + +```json +{ + "RelativePath": "notes.txt", + "Sha256Before": "f19e0e0c…", + "Sha256After": null, + "DetectionMode": "Detected", + "DetectedEncoding": "iso-8859-1", + "SourceEncoding": "iso-8859-1", + "Ambiguity": "TextChanging", + "AmbiguityReason": "MultipleCodecsDifferentText", + "DetectionCandidates": ["cp866", "ibm852", "ibm855", "…"], + "PlannedAction": "Refuse", + "Status": "Refused", + "Reason": "The encoding could not be determined uniquely…" +} +``` + +Three things it does deliberately: + +- **Refused and skipped files are in it.** *Why was this file not converted?* gets asked + far more often than *how do I put this one back?*, and until now only the second had an + answer — in a sidecar written solely where a backup existed. +- **It records the encoding the conversion read, not the detector's raw output.** Those + differ whenever somebody named the source encoding, and `DetectedEncoding` keeps the + detector's answer beside it so the difference is visible. +- **It does not claim more than happened.** `Sha256After` is present only where a file was + actually rewritten, so the record can be checked against the disk. A `-WhatIf` run is + marked as a preview and its files are recorded as decided, not converted. + +In the GUI it is written through **Export report**, choosing *Conversion journal (\*.json)*. + +Together with the per-file `.ecmeta.json` sidecar — which exists so one conversion can be +undone — this completes the chain: what EC believed, what it decided, what was approved, +and what it wrote. + ### One policy engine The GUI and the CLI ask the same question of the same code: @@ -106,6 +148,7 @@ EncodingChecker.exe # detecting it (Convert mode only) [-Plan ] # Write a conversion plan; change nothing [-Apply ] # Carry out a plan written by -Plan + [-Journal ] # Record what the conversion actually did [-Report ] # Also write a CSV report to this path [-MaxParallelism ] # Default: min(logical processor count, 4) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md new file mode 100644 index 0000000..9124531 --- /dev/null +++ b/RELEASE-CHECKLIST.md @@ -0,0 +1,91 @@ +# Release checklist + +Automated coverage is the first gate and is enforced by CI. What follows is what CI +cannot answer. + +## Automated + +- [ ] `dotnet test sources/EncodingChecker.Tests/EncodingChecker.Tests.csproj -c Release` — all green. +- [ ] The 1,033-file oracle sentinel set still agrees with GNU libiconv and ICU. +- [ ] Detector-drift check passes (the detector sources are duplicated across three + repositories and nothing enforces the sync; a fix in one is a fix owed to all three). + +## Manual: the GUI smoke test + +**Why this is manual.** EC's conversion policy, the plan binding, and the whole +orchestration sequence are automated. What is left is Windows Forms itself — designer +layout, background-worker marshalling, and the dialog's behaviour under a real message +pump. Automating those would mean weakening the architecture to make it drivable, which +would trade a real safety property for a test. + +**Why it is not optional.** EC has already shipped a defect of exactly this shape: every +component was correct and tested while the GUI's *sequence* converted files the CLI +refuses. Nothing failed, because nothing ran the sequence. The orchestration is now +covered, so what remains is genuinely UI, but "genuinely UI" was also the last +description that turned out to be wrong. + +### The evidence that counts + +Status messages are not evidence. For every case below that must not modify a file, +record the file's SHA-256 before and after: + +```powershell +Get-FileHash -Algorithm SHA256 | Select-Object -ExpandProperty Hash +``` + +### Test files + +| name | contents | encoding | expected classification | +|---|---|---|---| +| `jp.txt` | `こんにちは世界。日本語のテキストです。` | Shift_JIS | unambiguous | +| `french.txt` | `Le café était déjà prêt` | windows-1252 | text-changing | +| `russian.txt` | `Привет мир, это русский текст` | koi8-r | text-changing | +| `plain.txt` | `plain ascii, no high bytes at all` | ASCII | text-equivalent | + +### Matrix + +| # | step | expected | +|---|---|---| +| 1 | **View** the directory | 4 files listed with their encodings | +| 2 | Tick all, **Convert** to utf-8 | confirmation appears; two files listed as needing an explicit source encoding, with competing encodings named | +| 3 | **Cancel** | nothing converted; **all four hashes unchanged**; no `.bak` files | +| 4 | Convert again; untick `russian.txt`; choose `windows-1252` | button reads "Use this encoding for 1 file(s)" | +| 5 | Confirm the re-planned conversion | `french.txt` converts and reads correctly as French | +| 6 | Check `russian.txt` | **hash unchanged**; still refused | +| 7 | Convert again; while the dialog is open, edit one selected file in another editor and save | — | +| 8 | Confirm | run stops; message names the changed file; **every hash unchanged** | +| 9 | Convert `jp.txt` alone, backups on | converts; `jp.txt.bak` and `jp.txt.ecmeta.json` present; text reads correctly | +| 10 | Create a **directory** named `.bak` beside a file, convert it | conversion refused; **source hash unchanged** | +| 11 | Export report → **Conversion journal (\*.json)** | journal written; refused files present with their competing encodings; `Sha256After` null for everything not converted | + +### Record + +Fill this in and keep it with the release. It is the auditable answer to the one question +the test suite cannot reach. + +```text +EC version: +Commit: +Windows version: +.NET version: +Date: +Tester: + +Test files and their SHA-256 before: + +Step-by-step observations (expected vs observed): + +SHA-256 after, per file: + +Cases where observed differed from expected: + +Result: PASS / FAIL +``` + +## Documentation + +- [ ] README figures match the current audit run; no stale counts. +- [ ] Version bumped in `Program.cs` usage text and the README heading. +- [ ] `SemanticsVersion` bumped **only** if conversion or classification behaviour + changed — it invalidates existing plans, and bumping it for a release that changed + neither teaches people to work around the check. diff --git a/sources/EncodingChecker.Tests/ConversionJournalTests.cs b/sources/EncodingChecker.Tests/ConversionJournalTests.cs new file mode 100644 index 0000000..004b2c5 --- /dev/null +++ b/sources/EncodingChecker.Tests/ConversionJournalTests.cs @@ -0,0 +1,288 @@ +using System.Text; + +namespace EncodingChecker.Tests; + +/// +/// The record of what a conversion actually did. +/// +/// The chain this completes is: what EC believed, what it decided, what was approved, and +/// what it wrote. Each link has been shown to matter. The audit found conversions that +/// reported success while the text had changed — believing and writing had come apart with +/// nothing recording it. The GUI defect found conversions EC had never decided on at all. +/// And "why was this file not converted?" is asked far more often than "how do I put this +/// one back?", yet only the second had an answer, in a sidecar written solely where a +/// backup existed. +/// +/// So the journal covers the run whole: refused and skipped files included, the encoding +/// the conversion actually read each file as rather than the detector's raw output, and +/// the file's SHA-256 before and after. +/// +public sealed class ConversionJournalTests : IDisposable +{ + private readonly string _root = + Directory.CreateTempSubdirectory("ec_journal_").FullName; + + private string JournalPath => Path.Combine(_root, "journal.json"); + + public void Dispose() + { + try + { + Directory.Delete(_root, recursive: true); + } + catch (IOException) + { + // Best-effort cleanup. + } + } + + private string Write(string name, string text, string charset) + { + string path = Path.Combine(_root, name); + File.WriteAllBytes(path, Encoding.GetEncoding(charset).GetBytes(text)); + return path; + } + + private static int Cli(params string[] args) + { + TextWriter originalOut = Console.Out; + TextWriter originalError = Console.Error; + + try + { + Console.SetOut(new StringWriter()); + Console.SetError(new StringWriter()); + + return Program.RunConsoleMode(args); + } + finally + { + Console.SetOut(originalOut); + Console.SetError(originalError); + } + } + + private ConversionJournal Load() + { + ConversionJournal? journal = ConversionJournal.Load(JournalPath, out string? error); + + Assert.Null(error); + Assert.NotNull(journal); + + return journal; + } + + private JournalEntry EntryFor(string name) => + Assert.Single( + Load().Entries, + e => string.Equals(e.RelativePath, name, StringComparison.Ordinal)); + + private int Convert(params string[] extra) => + Cli([ + "-BasePath", _root, "-Target", "utf-8", + "-Journal", JournalPath, "-Quiet", .. extra + ]); + + [Fact] + public void ItRecordsWhatWasBelievedDecidedAndWritten() + { + const string text = "こんにちは世界。日本語のテキストです。"; + string path = Write("jp.txt", text, "shift_jis"); + string before = ConversionMetadataStore.ComputeSha256(path); + + Assert.Equal(0, Convert("-Backup")); + + JournalEntry entry = EntryFor("jp.txt"); + + // Believed. + Assert.Equal("Detected", entry.DetectionMode); + Assert.Equal("shift_jis", entry.DetectedEncoding); + Assert.Equal(AmbiguityClass.Unambiguous, entry.Ambiguity); + + // Decided. + Assert.Equal(PlannedAction.Convert, entry.PlannedAction); + + // Written — and checkable against the disk, which is the point of recording it. + Assert.Equal(ConversionStatus.Converted, entry.Status); + Assert.Equal(before, entry.Sha256Before); + Assert.Equal(ConversionMetadataStore.ComputeSha256(path), entry.Sha256After); + Assert.NotEqual(entry.Sha256Before, entry.Sha256After); + Assert.Equal("jp.txt.bak", entry.BackupPath); + } + + [Fact] + public void ItRecordsTheEncodingTheConversionReadRatherThanWhatTheFileBecame() + { + // A completed conversion re-labels the entry so a second pass reads the new bytes + // correctly, which means by journal time the file's effective encoding is the + // target. Reporting that as what it was read as would invert what happened. + Write("jp.txt", "こんにちは世界。日本語のテキストです。", "shift_jis"); + + Assert.Equal(0, Convert()); + + JournalEntry entry = EntryFor("jp.txt"); + + Assert.Equal("shift_jis", entry.SourceEncoding); + Assert.Equal(932, entry.SourceCodePage); + Assert.NotEqual("utf-8", entry.SourceEncoding); + } + + [Fact] + public void ARefusalIsRecordedWithTheEncodingsThatCompetedForIt() + { + // The question a record most often has to answer is why something was left alone. + string path = Write("ambiguous.txt", "Le café était déjà prêt", "windows-1252"); + string before = ConversionMetadataStore.ComputeSha256(path); + + Assert.Equal(3, Convert()); + + JournalEntry entry = EntryFor("ambiguous.txt"); + + Assert.Equal(PlannedAction.Refuse, entry.PlannedAction); + Assert.Equal(ConversionStatus.Refused, entry.Status); + Assert.Equal(AmbiguityClass.TextChanging, entry.Ambiguity); + Assert.NotEmpty(entry.DetectionCandidates); + Assert.Contains("could not be determined uniquely", entry.Reason); + + // Nothing was written, so there is no "after" — and the file still is what the + // "before" says it is. + Assert.Null(entry.Sha256After); + Assert.Equal(before, entry.Sha256Before); + Assert.Equal(before, ConversionMetadataStore.ComputeSha256(path)); + } + + [Fact] + public void AnExplicitSourceIsDistinguishedFromADetectedOne() + { + // Detection can be wrong in ways an explicit choice cannot. A record that cannot + // tell them apart cannot say who was responsible for the reading. + Write("ambiguous.txt", "Le café était déjà prêt", "windows-1252"); + + Assert.Equal(0, Convert("-From", "windows-1252")); + + JournalEntry entry = EntryFor("ambiguous.txt"); + + Assert.Equal("Explicit", entry.DetectionMode); + Assert.Equal("windows-1252", entry.SourceEncoding); + Assert.Equal(AmbiguityReason.ExplicitlySpecified, entry.AmbiguityReason); + Assert.Equal(ConversionStatus.Converted, entry.Status); + Assert.Equal("windows-1252", Load().ExplicitSourceEncoding); + } + + [Fact] + public void EveryFileTheRunTouchedIsAccountedFor() + { + Write("jp.txt", "こんにちは世界。日本語のテキストです。", "shift_jis"); + Write("ambiguous.txt", "Le café était déjà prêt", "windows-1252"); + Write("already.txt", "already utf-8 世界", "utf-8"); + + Convert(); + + ConversionJournal journal = Load(); + + Assert.Equal(3, journal.Entries.Count); + Assert.Equal(3, journal.Summary.Values.Sum()); + Assert.Equal(1, journal.Summary["Converted"]); + Assert.Equal(1, journal.Summary["Refused"]); + Assert.Equal(1, journal.Summary["Unchanged"]); + } + + [Fact] + public void ItRecordsTheConversionBehaviourTheRunUsed() + { + // The same reason a plan carries it: what was done is only meaningful alongside + // the rules it was done under. + Write("jp.txt", "こんにちは世界。テキスト", "shift_jis"); + + Convert("-Backup"); + + ConversionJournal journal = Load(); + + Assert.Equal(ConversionJournal.CurrentJournalVersion, journal.JournalVersion); + Assert.Equal(ConversionSemantics.Current, journal.SemanticsVersion); + Assert.True(journal.Semantics.StrictDecoding); + Assert.True(journal.Semantics.AmbiguityRefusal); + Assert.Equal("CommandLine", journal.Surface); + Assert.Equal("utf-8", journal.TargetEncoding); + Assert.False(journal.TargetHasBom); + Assert.True(journal.BackupEnabled); + Assert.NotEmpty(journal.ECVersion); + } + + [Fact] + public void ApplyingAPlanRecordsWhichPlanWasCarriedOut() + { + Write("jp.txt", "こんにちは世界。日本語のテキストです。", "shift_jis"); + + string planPath = Path.Combine(_root, "plan.json"); + + Assert.Equal(0, Cli( + "-BasePath", _root, "-Target", "utf-8", "-Plan", planPath, "-Quiet")); + + Assert.Equal(0, Cli("-Apply", planPath, "-Journal", JournalPath)); + + ConversionJournal journal = Load(); + + Assert.Equal(planPath, journal.AppliedPlan); + Assert.Equal( + ConversionStatus.Converted, Assert.Single(journal.Entries).Status); + } + + [Fact] + public void AFailedConversionIsNotRecordedAsARefusal() + { + // Both leave the file alone, and the difference is the whole of what a reader + // needs: one is EC declining, the other is EC trying and not managing. + byte[] original = Encoding.UTF8.GetBytes("世界 مرحبا"); + string path = Path.Combine(_root, "unencodable.txt"); + File.WriteAllBytes(path, original); + + Assert.Equal(3, Cli( + "-BasePath", _root, "-Target", "windows-1252", + "-Journal", JournalPath, "-Quiet")); + + JournalEntry entry = EntryFor("unencodable.txt"); + + Assert.Equal(ConversionStatus.Failed, entry.Status); + Assert.Equal(PlannedAction.Convert, entry.PlannedAction); + Assert.Null(entry.Sha256After); + Assert.Equal(original, File.ReadAllBytes(path)); + } + + [Fact] + public void AJournalIsRefusedForModesThatConvertNothing() + { + Write("jp.txt", "こんにちは世界。テキスト", "shift_jis"); + + Assert.Equal(1, Cli( + "-BasePath", _root, "-DetectOnly", "-Journal", JournalPath)); + + Assert.Equal(1, Cli( + "-BasePath", _root, "-Validate", "utf-8", "-Journal", JournalPath)); + + Assert.False(File.Exists(JournalPath)); + } + + [Fact] + public void APreviewRecordsWhatWouldHaveHappenedWithoutClaimingItDid() + { + // -WhatIf reports rows as "would be converted". A journal of that run must not + // read as a record of files having been rewritten. + string path = Write("jp.txt", "こんにちは世界。テキスト", "shift_jis"); + byte[] before = File.ReadAllBytes(path); + + Assert.Equal(0, Convert("-WhatIf")); + + JournalEntry entry = EntryFor("jp.txt"); + + // The decision is recorded; the outcome is not claimed. + Assert.Equal(PlannedAction.Convert, entry.PlannedAction); + Assert.Equal(ConversionStatus.NotAttempted, entry.Status); + Assert.Null(entry.Sha256After); + Assert.True(Load().Preview); + + Assert.Equal(before, File.ReadAllBytes(path)); + Assert.Equal( + ConversionMetadataStore.ComputeSha256(path), entry.Sha256Before); + } +} diff --git a/sources/EncodingChecker/ConversionJournal.cs b/sources/EncodingChecker/ConversionJournal.cs new file mode 100644 index 0000000..bbfade6 --- /dev/null +++ b/sources/EncodingChecker/ConversionJournal.cs @@ -0,0 +1,340 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; + +namespace EncodingChecker; + +/// What became of one file. +[JsonConverter(typeof(JsonStringEnumConverter))] +internal enum ConversionStatus +{ + /// Rewritten in the target encoding. + Converted, + + /// Already in the target encoding; not touched. + Unchanged, + + /// Encoding not identified; not touched. + Skipped, + + /// Converting could not be shown to be safe; not touched. + Refused, + + /// Conversion was attempted and did not complete; not touched. + Failed, + + /// + /// Decided, and deliberately not carried out - a preview, or a run that stopped + /// before reaching this file. + /// + NotAttempted, +} + +/// +/// One file's line in the journal: what EC believed, what it decided, and what it wrote. +/// +internal sealed record JournalEntry +{ + public required string RelativePath { get; init; } + + /// The file's bytes before EC touched it. + public required string Sha256Before { get; init; } + + /// + /// The file's bytes afterwards, or when nothing was written. + /// + /// + /// Present only for a conversion that completed. A null here and a + /// other than + /// together say that the file on disk is still the one + /// describes. + /// + public string? Sha256After { get; init; } + + // ---- what EC believed + + /// Whether the source encoding was detected or supplied. + public required string DetectionMode { get; init; } + + /// What detection concluded, kept even when a person overrode it. + public required string DetectedEncoding { get; init; } + + /// The encoding the conversion actually read the file as. + public required string SourceEncoding { get; init; } + + public required int SourceCodePage { get; init; } + + public required bool SourceHasBom { get; init; } + + [JsonConverter(typeof(JsonStringEnumConverter))] + public required AmbiguityClass Ambiguity { get; init; } + + [JsonConverter(typeof(JsonStringEnumConverter))] + public required AmbiguityReason AmbiguityReason { get; init; } + + /// Encodings that also fit these bytes and read them differently. + public IReadOnlyList DetectionCandidates { get; init; } = []; + + // ---- what EC decided, and what happened + + [JsonConverter(typeof(JsonStringEnumConverter))] + public required PlannedAction PlannedAction { get; init; } + + public required ConversionStatus Status { get; init; } + + /// Why, when the outcome was not a plain conversion. + public string? Reason { get; init; } + + /// Where the original was kept, when it was. + public string? BackupPath { get; init; } +} + +/// +/// A record of one conversion run: every file EC looked at, what it concluded, what it +/// decided, and what it actually wrote. +/// +/// +/// Distinct from the per-file sidecar, which exists so a +/// single conversion can be undone and is written only where there is a backup to undo it +/// from. This is the run, whole: the files that were refused and the files that were +/// skipped are in it too, because "why did EC not convert this?" is a question people ask +/// more often than "how do I put this one back?", and until now nothing answered it after +/// the console output scrolled away. +/// +/// It records the decision that was carried out rather than the detector's raw output. +/// Those differ whenever somebody named the source encoding, and the difference is +/// exactly what an audit needs: what EC believed, what it decided, what was approved, and +/// what it wrote. +/// +/// +internal sealed record ConversionJournal +{ + internal const int CurrentJournalVersion = 1; + + public int JournalVersion { get; init; } = CurrentJournalVersion; + + /// The conversion behaviour this run was carried out under. + public int SemanticsVersion { get; init; } = ConversionSemantics.Current; + + public ConversionSemantics Semantics { get; init; } = new(); + + public required string ECVersion { get; init; } + + public required string StartedUtc { get; init; } + + public required string CompletedUtc { get; init; } + + /// Which interface ran it: the command line, or the application window. + public required string Surface { get; init; } + + public required string BaseDirectory { get; init; } + + public required string TargetEncoding { get; init; } + + public required bool TargetHasBom { get; init; } + + public required bool BackupEnabled { get; init; } + + /// The source encoding named by a person, if any. + public string? ExplicitSourceEncoding { get; init; } + + /// + /// The plan this run carried out, when it carried out a written one. + /// + public string? AppliedPlan { get; init; } + + /// Whether this was a preview, which wrote nothing. + /// + /// A preview reports its rows as "would be converted", and a journal that copied that + /// through would claim files had been rewritten when the directory is untouched. The + /// hashes would give it away - before and after would match - but a record should not + /// need to be caught out to be read correctly. + /// + public bool Preview { get; init; } + + public required IReadOnlyList Entries { get; init; } + + /// Counts by outcome, so the run can be read without tallying it. + public IReadOnlyDictionary Summary => + Entries + .GroupBy(entry => entry.Status.ToString()) + .OrderBy(group => group.Key, StringComparer.Ordinal) + .ToDictionary(group => group.Key, group => group.Count()); + + private static readonly JsonSerializerOptions Options = new() + { + WriteIndented = true, + }; + + /// + /// Builds the journal from the entries a run finished with. + /// + /// When the run began. + /// + /// Reads each converted file once more to record what was actually written. That is + /// the whole point of the last column: a journal that reports what EC intended is a + /// journal that cannot be checked against the disk. + /// + internal static ConversionJournal FromRun( + IEnumerable entries, + string baseDirectory, + string targetCharset, + bool targetHasBom, + bool backupEnabled, + string? explicitSource, + string surface, + DateTime startedUtc, + string? appliedPlan = null, + bool preview = false) + { + string root = Path.TrimEndingDirectorySeparator(Path.GetFullPath(baseDirectory)); + var lines = new List(); + + foreach (ConversionReportEntry entry in entries) + { + ConversionStatus status = entry.Result switch + { + // "Would be converted" is a decision, not an outcome. + ConversionRowResult.Converted when preview + => ConversionStatus.NotAttempted, + ConversionRowResult.Converted => ConversionStatus.Converted, + ConversionRowResult.Unchanged => ConversionStatus.Unchanged, + ConversionRowResult.Skipped => ConversionStatus.Skipped, + ConversionRowResult.Error when entry.Action == PlannedAction.Refuse + => ConversionStatus.Refused, + ConversionRowResult.Error => ConversionStatus.Failed, + _ => ConversionStatus.NotAttempted, + }; + + // What the conversion read, recorded at the time. Falling back to the + // effective label would report a converted file's new encoding as the one it + // was read as, which is the opposite of what happened. + ScanEngine.ParseCharsetLabel( + entry.ResolvedSourceLabel ?? entry.EffectiveSourceLabel, + out string sourceCharset, + out bool sourceHasBom); + + int codePage = 0; + + try + { + codePage = Encoding.GetEncoding(sourceCharset).CodePage; + } + catch (ArgumentException) + { + // Zero records that the label named no code page EC could resolve. + } + + string backupPath = entry.FilePath + ".bak"; + + lines.Add(new JournalEntry + { + RelativePath = Path.GetRelativePath(root, entry.FilePath), + // The plan-bound paths already carry the approved hash, so a journal + // over them costs no extra reading at all. + Sha256Before = entry.JournalSourceSha256 + ?? entry.ExpectedSourceSha256 + ?? Hash(entry.FilePath), + + // Only a completed conversion changed anything, so only it has an + // "after". Hashing the others would report a second reading of bytes + // nothing touched, which reads as evidence and is not. + Sha256After = status == ConversionStatus.Converted + ? Hash(entry.FilePath) + : null, + + DetectionMode = entry.SourceEncodingWasSpecified ? "Explicit" : "Detected", + DetectedEncoding = entry.SourceEncoding, + SourceEncoding = sourceCharset, + SourceCodePage = codePage, + SourceHasBom = sourceHasBom, + Ambiguity = entry.Ambiguity ?? AmbiguityClass.TextChanging, + AmbiguityReason = entry.AmbiguityReason, + DetectionCandidates = entry.CompetingEncodings, + PlannedAction = entry.Action ?? PlannedAction.Skip, + Status = status, + Reason = string.IsNullOrEmpty(entry.Diagnostic) ? null : entry.Diagnostic, + BackupPath = + status == ConversionStatus.Converted + && backupEnabled + && File.Exists(backupPath) + ? Path.GetRelativePath(root, backupPath) + : null, + }); + } + + return new ConversionJournal + { + ECVersion = typeof(ConversionJournal).Assembly.GetName().Version?.ToString() + ?? "unknown", + StartedUtc = startedUtc.ToString("O", CultureInfo.InvariantCulture), + CompletedUtc = DateTime.UtcNow.ToString("O", CultureInfo.InvariantCulture), + Surface = surface, + BaseDirectory = root, + TargetEncoding = targetCharset, + TargetHasBom = targetHasBom, + BackupEnabled = backupEnabled, + ExplicitSourceEncoding = explicitSource, + AppliedPlan = appliedPlan, + Preview = preview, + Entries = [.. lines.OrderBy(l => l.RelativePath, StringComparer.OrdinalIgnoreCase)], + }; + } + + /// A file's SHA-256, or empty when it cannot be read. + private static string Hash(string path) + { + try + { + return File.Exists(path) ? ConversionMetadataStore.ComputeSha256(path) : string.Empty; + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + return string.Empty; + } + } + + internal string? Save(string path) + { + try + { + File.WriteAllText( + path, JsonSerializer.Serialize(this, Options), new UTF8Encoding(false)); + return null; + } + catch (Exception ex) when ( + ex is IOException or UnauthorizedAccessException or JsonException) + { + return ex.Message; + } + } + + internal static ConversionJournal? Load(string path, out string? error) + { + try + { + ConversionJournal? journal = + JsonSerializer.Deserialize(File.ReadAllText(path)); + + if (journal is null) + { + error = $"'{path}' is empty."; + return null; + } + + error = null; + return journal; + } + catch (Exception ex) when ( + ex is IOException or UnauthorizedAccessException or JsonException) + { + error = ex.Message; + return null; + } + } +} diff --git a/sources/EncodingChecker/ConversionReport.cs b/sources/EncodingChecker/ConversionReport.cs index d23efc6..b1ad1cc 100644 --- a/sources/EncodingChecker/ConversionReport.cs +++ b/sources/EncodingChecker/ConversionReport.cs @@ -122,6 +122,33 @@ internal sealed class ConversionReportEntry CurrentCharsetLabel ?? ScanEngine.FormatCharsetLabel(SourceEncoding, SourceHasBom); + /// + /// The charset label this run actually read the file as, recorded when it was read. + /// + /// + /// Not derivable afterwards. A completed conversion sets + /// to the target so a second pass reads the new + /// bytes correctly, which means that by the time a journal is written the effective + /// label describes what the file now is, not what it was read as. + /// + internal string? ResolvedSourceLabel { get; set; } + + /// + /// Whether to record this file's bytes before converting it, for the journal. + /// + /// + /// Off unless a journal was asked for. A conversion overwrites the file, so its + /// original hash cannot be recovered afterwards - but hashing every source on every + /// run would charge an extra full read to the runs that never wanted one. + /// + internal bool CaptureSourceHash { get; set; } + + /// + /// The file's bytes before this run touched it, when they were recorded. + /// Internal state; not included in CSV output. + /// + internal string? JournalSourceSha256 { get; set; } + /// Additional error detail; not included in CSV output. internal string? Diagnostic { get; set; } } diff --git a/sources/EncodingChecker/MainForm.cs b/sources/EncodingChecker/MainForm.cs index e295a90..62f61ef 100644 --- a/sources/EncodingChecker/MainForm.cs +++ b/sources/EncodingChecker/MainForm.cs @@ -66,6 +66,10 @@ private enum CurrentAction // actual conversion from preview ("would be converted"). private bool _convertWasPreview; + // When the last conversion began, so a journal exported afterwards can say. Null + // until one has run: there is nothing to journal before that. + private DateTime? _lastConversionStartedUtc; + // Held so the completion handler can read how the run ended, as reported by // ConversionOrchestrator: converted, previewed, cancelled, or stopped because the // files moved underneath the plan. The worker method is static and writes into it. @@ -335,7 +339,11 @@ private void OnExportReport(object? sender, EventArgs e) var saveFileDialog = new SaveFileDialog { Title = @"Export Conversion Report", - Filter = @"CSV files (*.csv)|*.csv", + + // Two records, not two formats of one. The CSV is the results table as + // shown; the journal is what EC believed, decided and wrote for every file, + // including the ones it refused. + Filter = @"CSV files (*.csv)|*.csv|Conversion journal (*.json)|*.json", FileName = "Conversion report.csv", RestoreDirectory = true, }; @@ -343,15 +351,21 @@ private void OnExportReport(object? sender, EventArgs e) if (saveFileDialog.ShowDialog(this) != DialogResult.OK) return; - try - { - var entries = - new List(lstResults.Items.Count); + var entries = + new List(lstResults.Items.Count); - // Tag is always set to the entry when the row is added (ActionWorkerProgressChanged). - foreach (ListViewItem item in lstResults.Items) - entries.Add((ConversionReportEntry)item.Tag!); + // Tag is always set to the entry when the row is added (ActionWorkerProgressChanged). + foreach (ListViewItem item in lstResults.Items) + entries.Add((ConversionReportEntry)item.Tag!); + + if (saveFileDialog.FilterIndex == 2) + { + ExportJournal(entries, saveFileDialog.FileName); + return; + } + try + { using var writer = new StreamWriter(saveFileDialog.FileName, false, Encoding.UTF8); @@ -366,6 +380,41 @@ private void OnExportReport(object? sender, EventArgs e) } } + /// Writes the record of what the last conversion actually did. + private void ExportJournal(List entries, string path) + { + if (_lastConversionStartedUtc is not { } startedUtc) + { + ShowWarning( + "There is no conversion to journal yet. Run Convert first; a journal " + + "records what a conversion did, which a detection scan has not."); + + return; + } + + ScanEngine.ParseCharsetLabel( + (string)lstConvert.SelectedItem!, + out string targetCharset, + out bool targetWriteBom); + + string? error = ConversionJournal.FromRun( + entries, + lstBaseDirectory.Text, + targetCharset, + targetWriteBom, + chkCreateBackup.Checked, + explicitSource: entries.Count > 0 + && entries.TrueForAll(e => e.SourceEncodingWasSpecified) + ? entries[0].ResolvedSourceLabel + : null, + surface: "Gui", + startedUtc) + .Save(path); + + if (error != null) + ShowWarning("Failed to export the journal: {0}", error); + } + private void OnBaseDirectoryDragEnter(object? sender, DragEventArgs e) { e.Effect = @@ -551,6 +600,7 @@ private void OnConvert(object? sender, EventArgs e) }; _convertArgs = args; + _lastConversionStartedUtc = DateTime.UtcNow; _actionWorker.RunWorkerAsync(args); } diff --git a/sources/EncodingChecker/Program.cs b/sources/EncodingChecker/Program.cs index 63c6f61..af8a4bc 100644 --- a/sources/EncodingChecker/Program.cs +++ b/sources/EncodingChecker/Program.cs @@ -102,6 +102,7 @@ internal sealed class CliOptions internal string? From; internal string? PlanPath; internal string? ApplyPath; + internal string? JournalPath; internal string? ValidateCharsets; internal bool DetectOnly; internal string? ReportPath; @@ -196,6 +197,18 @@ combined with -DetectOnly or -Target. -FailOnChanges, -Quiet, and -Verbose have no effect. Cannot be combined with -Validate. + Record: + [-Journal ] + Write a JSON record of the run: for every file, what + its encoding was detected or declared to be, whether + the bytes identified it, which encodings competed, + what EC decided, what it actually did, and the file's + SHA-256 before and after. Refused and skipped files + are included - "why was this not converted?" is the + question a record most often has to answer. + + Costs one extra read per file, so it is opt-in. + Report: [-Report ] Write a CSV report in addition to normal console @@ -301,6 +314,8 @@ private static int ApplyPlan(CliOptions options) return 3; } + DateTime startedUtc = DateTime.UtcNow; + List entries = [ .. plan.Files @@ -371,6 +386,29 @@ .. plan.Files // Every planned file is accounted for. A file that the plan scheduled but that // the run left alone is the interesting case, so it must not disappear into a // difference between two totals. + if (!string.IsNullOrWhiteSpace(options.JournalPath)) + { + string? journalError = ConversionJournal.FromRun( + completed, + plan.BaseDirectory, + plan.TargetEncoding, + plan.TargetHasBom, + plan.BackupEnabled, + plan.ExplicitSourceEncoding, + surface: "CommandLine", + startedUtc, + appliedPlan: options.ApplyPath) + .Save(options.JournalPath!); + + if (journalError != null) + { + Console.Error.WriteLine( + $"The conversion ran, but the journal could not be written: " + + journalError); + failed++; + } + } + Console.Out.WriteLine( $"Applied plan: {Count(ConversionRowResult.Converted)} converted, " + $"{Count(ConversionRowResult.Unchanged)} already in the target encoding, " @@ -462,12 +500,17 @@ .. options.ValidateCharsets! TargetWriteBom = targetWriteBom, WhatIf = options.WhatIf, Backup = options.Backup, + + // The original bytes have to be taken before anything overwrites them. + CaptureSourceHashes = !string.IsNullOrWhiteSpace(options.JournalPath), }; // onEntry fires concurrently, so collect into a ConcurrentBag, then sort by // path once scanning finishes for deterministic downstream output. var collectedEntries = new ConcurrentBag(); + DateTime startedUtc = DateTime.UtcNow; + using var cancellation = new CancellationTokenSource(); ConsoleCancelEventHandler cancelHandler = (_, e) => @@ -525,6 +568,32 @@ .. collectedEntries.OrderBy( if (options.Verbose) PrintVerboseSummary(entries); + if (!string.IsNullOrWhiteSpace(options.JournalPath)) + { + string? journalError = ConversionJournal.FromRun( + entries, + options.BasePath!, + targetCharset ?? options.Target ?? string.Empty, + targetWriteBom, + options.Backup, + options.From, + surface: "CommandLine", + startedUtc, + appliedPlan: null, + preview: options.WhatIf) + .Save(options.JournalPath!); + + if (journalError != null) + { + // Exit 3, not 1: in Convert mode the files have already been rewritten, + // and a conversion nothing recorded is exactly what this option exists + // to prevent. + Console.Error.WriteLine( + $"Failed to write the journal: {journalError}"); + return 3; + } + } + if (!string.IsNullOrEmpty(options.ReportPath)) { try @@ -708,6 +777,14 @@ internal static bool TryParseArguments( } break; + case "journal": + if (!TryTakeValue(args, ref i, out options.JournalPath)) + { + error = "-Journal requires a value."; + return false; + } + break; + case "apply": if (!TryTakeValue(args, ref i, out options.ApplyPath)) { @@ -794,6 +871,7 @@ internal static bool TryParseArguments( new(StringComparer.OrdinalIgnoreCase) { "basepath", "include", "exclude", "target", "from", "plan", "apply", + "journal", "validate", "detectonly", "report", "maxparallelism", "failonchanges", "whatif", "backup", "quiet", "verbose", @@ -880,6 +958,14 @@ internal static bool TryValidateOptions( } } + if (!string.IsNullOrWhiteSpace(options.JournalPath) && + (options.DetectOnly || !string.IsNullOrWhiteSpace(options.ValidateCharsets))) + { + error = "-Journal records what a conversion did; it cannot be combined " + + "with -DetectOnly or -Validate. Use -Report for those."; + return false; + } + if (!string.IsNullOrWhiteSpace(options.PlanPath) && (options.DetectOnly || !string.IsNullOrWhiteSpace(options.ValidateCharsets))) { diff --git a/sources/EncodingChecker/ScanEngine.cs b/sources/EncodingChecker/ScanEngine.cs index 490228d..8bd9fa5 100644 --- a/sources/EncodingChecker/ScanEngine.cs +++ b/sources/EncodingChecker/ScanEngine.cs @@ -77,6 +77,17 @@ internal sealed class ScanDirectoryOptions /// internal string? SourceCharset { get; init; } + /// + /// Record each file's bytes before converting it, so a journal can report what the + /// file was as well as what it became. + /// + /// + /// Off by default: a conversion overwrites the original, so the hash has to be taken + /// in advance, and taking it for every run would charge an extra full read to the + /// runs that never asked for a journal. + /// + internal bool CaptureSourceHashes { get; init; } + internal bool TargetWriteBom { get; init; } /// Simulate conversion without writing. @@ -356,6 +367,7 @@ internal static string FormatCharsetLabel( // default - correctly, since there is nothing ambiguous about an answer somebody // gave. entry.SourceEncodingWasSpecified = sourceWasSpecified; + entry.CaptureSourceHash = options.CaptureSourceHashes; switch (options.Action) { @@ -429,6 +441,23 @@ private static void ApplyConversion( { entry.TargetEncoding = targetCharset; entry.TargetHasBom = targetWriteBom; + entry.ResolvedSourceLabel = FormatCharsetLabel(sourceCharset, sourceHasBom); + + // Before anything can overwrite it. A converted file's original bytes are not + // recoverable from the file afterwards, so a journal that wants them has to say + // so in advance. + if (entry.CaptureSourceHash && entry.JournalSourceSha256 is null) + { + try + { + entry.JournalSourceSha256 = + ConversionMetadataStore.ComputeSha256(path); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + // Left null; the journal records an empty hash rather than a wrong one. + } + } // Classified here, at the point where the decision is actually made, so that // every caller reaching a conversion has it - the CLI's Convert scan, a plan From cc6af921072a7aaedd9d1ea31c95fa62a35edda6 Mon Sep 17 00:00:00 2001 From: amrali-eg <32075105+amrali-eg@users.noreply.github.com> Date: Thu, 27 Aug 2026 06:53:24 +0300 Subject: [PATCH 2/2] Add the GUI smoke-test harness, and record the run that passed The manual matrix needs an instrument, and the first one was wrong in a way worth writing down. It used a single folder and one final check for the whole sequence. That cannot work: the stale-plan case stops the entire run, so every "must have converted" expectation after it is unreachable by construction - and the state it leaves is byte-identical to "the tester cancelled everything", so the result cannot say which protection fired. Its first real run reported FAIL, and only reading the bytes by hand showed the product had been correct throughout. So the harness is four independent phases, each its own folder, click sequence and check, each proving one property. Every phase verifier was reproduced through the CLI before being handed over, and each was also checked to fail when it should - a verifier that cannot fail is the same defect as a test that silently asserts nothing. Phase C refuses to pass at all when the edit that makes the plan stale was never made, which is the failure mode a manual matrix is most prone to. Records the 2026-08-27 run on a201a08: all four phases pass. Two notes kept with it because they qualify what the phases establish. Phase B's French sample decodes identically under windows-1252 and iso-8859-1, so its text assertion cannot show which codec was used; what settles it is the recovery record's DetectedCodePage of 1252. And TextEquivalent is nearly unreachable - eight ASCII shapes all classify as StructurallyDetermined, and only a one-byte file reaches it - so the corpus has to be contrived to exercise that class at all. Co-Authored-By: Claude Opus 5 --- RELEASE-CHECKLIST.md | 58 ++++++++- tools/gui-smoke-test.py | 262 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 315 insertions(+), 5 deletions(-) create mode 100644 tools/gui-smoke-test.py diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 9124531..5237760 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -42,6 +42,21 @@ Get-FileHash -Algorithm SHA256 | Select-Object -ExpandProperty Hash | `russian.txt` | `Привет мир, это русский текст` | koi8-r | text-changing | | `plain.txt` | `plain ascii, no high bytes at all` | ASCII | text-equivalent | +### Structure it in phases, not one long sequence + +The first version of this test used one folder and one final check for the whole matrix. +That cannot work, and the reason is worth keeping: the stale-plan case **stops the entire +run**, so every "must have converted" expectation after it is unreachable by construction. +Worse, the state it leaves is byte-identical to "the tester cancelled everything", so the +result cannot say *which* protection fired. The first real run produced a FAIL that was +entirely the instrument's fault, and only inspecting the bytes by hand showed the product +had behaved correctly. + +Each phase therefore gets its own folder, its own short click sequence, and its own check, +and proves exactly one property. [`tools/gui-smoke-test.py`](tools/gui-smoke-test.py) does the setup and the +verification; it also refuses to pass a phase whose defining action was skipped — a phase +that silently tests nothing is the failure mode a manual matrix is most prone to. + ### Matrix | # | step | expected | @@ -71,17 +86,50 @@ Windows version: Date: Tester: -Test files and their SHA-256 before: - -Step-by-step observations (expected vs observed): - -SHA-256 after, per file: +Phase A (refuse + cancel change nothing): PASS / FAIL +Phase B (explicit source, scoped): PASS / FAIL +Phase C (stale plan stops the whole run): PASS / FAIL +Phase D (backup + record; backup failure): PASS / FAIL Cases where observed differed from expected: Result: PASS / FAIL ``` +### Run of 2026-08-27 + +```text +EC version: 3.7.0.0 +Commit: a201a08 +Windows version: Microsoft Windows NT 10.0.26200.0 +.NET version: 10.0.400 +Date: 2026-08-27 +Tester: amrali-eg + +Phase A (refuse + cancel change nothing): PASS +Phase B (explicit source, scoped): PASS +Phase C (stale plan stops the whole run): PASS +Phase D (backup + record; backup failure): PASS + +Result: PASS +``` + +Notes from that run, kept because they qualify what the phases actually establish: + +- **Phase B proves less on its own than it appears to.** Its French sample decodes + identically under windows-1252 and iso-8859-1, so "the text is preserved" cannot show + which codec was used. What settles it is the recovery record: `french.txt.ecmeta.json` + gives `DetectedCodePage: 1252`, the encoding chosen in the dialog rather than the + `iso-8859-1` that detection proposed. A future revision should use content where the two + encodings genuinely disagree, so the assertion stands without the sidecar. +- **Text-equivalent ambiguity is nearly unreachable.** Eight ASCII shapes — short strings, + digits, JSON, code, newlines — all classify as `StructurallyDetermined`, because ASCII + constrains every byte below 0x80. Only a **one-byte file** reaches `TextEquivalent`, + where no codec that decodes it at all can read it differently. The middle class of the + three-way taxonomy is far rarer in practice than the taxonomy suggests. The classifier + is right in both cases; the corpus has to be contrived to exercise it, and `tiny.txt` + exists for that reason alone. + ## Documentation - [ ] README figures match the current audit run; no stale counts. diff --git a/tools/gui-smoke-test.py b/tools/gui-smoke-test.py new file mode 100644 index 0000000..43b787a --- /dev/null +++ b/tools/gui-smoke-test.py @@ -0,0 +1,262 @@ +"""GUI smoke test, in phases. + +The first version used one corpus and one final check for the whole matrix. That cannot +work: the stale-plan case stops the entire run, so every "must have converted" +expectation after it is unreachable by construction. Worse, the state it leaves is +identical to "the tester cancelled everything", so it cannot say which protection fired. + +Each phase is therefore its own folder, its own short click sequence, and its own check, +and each proves exactly one property. The evidence is always the bytes on disk. + + python smoke.py setup A # then do phase A in the GUI + python smoke.py verify A +""" +import hashlib +import json +import os +import shutil +import sys + +BASE = r"C:\Users\Amr\Desktop" +HERE = os.path.dirname(os.path.abspath(__file__)) + +JP = ("こんにちは世界。日本語のテキストです。", "shift_jis") +FRENCH = ("Le café était déjà prêt", "cp1252") +RUSSIAN = ("Привет мир, это русский текст", "koi8-r") +PLAIN = ("plain ascii, no high bytes at all", "ascii") +TINY = ("A", "ascii") +MOVING = ("さようなら世界。これも日本語のテキストです。", "shift_jis") +BACKUPFAIL = ("これはバックアップ失敗の試験です。", "shift_jis") + +PHASES = { + "A": { + "title": "Refusing and cancelling change nothing", + "files": {"jp.txt": JP, "french.txt": FRENCH, "russian.txt": RUSSIAN, + "plain.txt": PLAIN, "tiny.txt": TINY}, + "bak_dirs": [], + "steps": [ + "View the folder, tick every row, click Convert.", + "The confirmation should say 2 file(s) need an explicit source encoding,", + " listing french.txt and russian.txt with the encodings in conflict.", + "Click Cancel.", + ], + "unchanged": { + "jp.txt": "cancelled, so nothing may be written", + "french.txt": "refused, and cancelled", + "russian.txt": "refused, and cancelled", + "plain.txt": "cancelled", + "tiny.txt": "cancelled", + }, + "converted": {}, + "no_artifacts": True, + }, + "B": { + "title": "An explicit source applies only to the files it was given for", + "files": {"jp.txt": JP, "french.txt": FRENCH, "russian.txt": RUSSIAN}, + "bak_dirs": [], + "steps": [ + "View, tick every row, Convert.", + "In the confirmation, UNTICK russian.txt so only french.txt stays ticked.", + "Choose windows-1252. The button should read 'Use this encoding for 1 file(s)'.", + "Click it, then click Convert on the plan that comes back.", + ], + "unchanged": { + "russian.txt": "not answered for, so it stays refused", + }, + "converted": { + "jp.txt": JP[0], + "french.txt": FRENCH[0], + }, + "no_artifacts": False, + }, + "C": { + "title": "A file changed while the dialog is open stops the whole run", + "files": {"jp.txt": JP, "moving.txt": MOVING}, + "bak_dirs": [], + "steps": [ + "View, tick both rows, Convert.", + "LEAVE THE CONFIRMATION OPEN. In another editor, append anything to", + " moving.txt and save it.", + "Now click Convert in the confirmation.", + "It should refuse and name moving.txt.", + ], + "unchanged": { + "jp.txt": "the run must stop whole, not convert the files that still match", + }, + "converted": {}, + "no_artifacts": True, + "edited": "moving.txt", + }, + "D": { + "title": "A conversion leaves a backup and a record; a failed backup aborts", + "files": {"jp.txt": JP, "backupfail.txt": BACKUPFAIL}, + "bak_dirs": ["backupfail.txt.bak"], + "steps": [ + "Tick 'Back up original files before converting'.", + "View, tick both rows, Convert, and confirm.", + ], + "unchanged": { + "backupfail.txt": "its .bak path is a directory, so the backup cannot be written", + }, + "converted": {"jp.txt": JP[0]}, + "artifacts": ["jp.txt.bak", "jp.txt.ecmeta.json"], + "no_artifacts": False, + }, +} + + +def root(phase): + return os.path.join(BASE, "EC-smoke-" + phase) + + +def state_path(phase): + return os.path.join(HERE, "smoke-state-" + phase + ".json") + + +def sha256(path): + with open(path, "rb") as handle: + return hashlib.sha256(handle.read()).hexdigest() + + +def snapshot(directory): + return { + name: sha256(os.path.join(directory, name)) + for name in sorted(os.listdir(directory)) + if os.path.isfile(os.path.join(directory, name)) + } + + +def setup(phase): + spec = PHASES[phase] + directory = root(phase) + + if os.path.isdir(directory): + shutil.rmtree(directory) + os.makedirs(directory) + + for name, (text, encoding) in spec["files"].items(): + with open(os.path.join(directory, name), "wb") as handle: + handle.write(text.encode(encoding)) + + for name in spec["bak_dirs"]: + os.makedirs(os.path.join(directory, name)) + + with open(state_path(phase), "w", encoding="utf-8") as handle: + json.dump({"root": directory, "before": snapshot(directory)}, handle, indent=1) + + print("Phase " + phase + " - " + spec["title"]) + print("\n folder: " + directory) + print(" files : " + ", ".join(spec["files"])) + for name in spec["bak_dirs"]: + print(" plus : " + name + "/ (a directory, so the backup must fail)") + print("\n in the GUI:") + for step in spec["steps"]: + print(" " + step) + print("\n then: python smoke.py verify " + phase) + return 0 + + +def verify(phase): + spec = PHASES[phase] + + with open(state_path(phase), encoding="utf-8") as handle: + state = json.load(handle) + + directory = state["root"] + before = state["before"] + after = snapshot(directory) + failures = [] + + for name, why in spec["unchanged"].items(): + if name not in after: + failures.append(name + ": MISSING (" + why + ")") + elif after[name] != before[name]: + failures.append(name + ": CHANGED but must not have - " + why) + else: + print(" ok %-18s unchanged (%s)" % (name, why)) + + for name, expected in spec["converted"].items(): + if name not in after: + failures.append(name + ": MISSING") + continue + + raw = open(os.path.join(directory, name), "rb").read() + + try: + text = raw.decode("utf-8") + except UnicodeDecodeError as ex: + failures.append( + name + ": still not UTF-8, so it was never converted (" + str(ex) + ")") + continue + + if text != expected: + failures.append( + name + ": converted, but the text changed\n" + " expected " + repr(expected) + "\n" + " actual " + repr(text)) + else: + print(" ok %-18s converted, text preserved exactly" % name) + + # The file the tester edited must carry that edit and nothing else - EC must not have + # converted it either. Telling the tester's write apart from a conversion is precisely + # what the single-corpus version of this script could not do, and why its result was + # unreadable. + edited = spec.get("edited") + + if edited: + raw = open(os.path.join(directory, edited), "rb").read() + text, encoding = spec["files"][edited] + original = text.encode(encoding) + + if raw == original: + failures.append( + edited + ": unchanged - the edit that makes the plan stale was never " + "made, so this phase tested nothing") + elif raw.startswith(original): + print(" ok %-18s carries your edit, not a conversion" % edited) + else: + try: + raw.decode("utf-8") + failures.append(edited + ": looks converted rather than merely edited") + except UnicodeDecodeError: + print(" ok %-18s not converted" % edited) + + for name in spec.get("artifacts", []): + if os.path.exists(os.path.join(directory, name)): + print(" ok %-18s present" % name) + else: + failures.append(name + ": missing") + + if spec.get("no_artifacts"): + strays = [ + n for n in os.listdir(directory) + if n.endswith((".bak", ".ecmeta.json")) and n not in spec["bak_dirs"] + ] + + if strays: + failures.append( + "nothing should have been written, but found: " + ", ".join(strays)) + else: + print(" ok %-18s no backups or records written" % "(folder)") + + print() + + if failures: + print("PHASE " + phase + ": FAIL") + for failure in failures: + print(" " + failure) + return 1 + + print("PHASE " + phase + ": PASS - " + spec["title"]) + return 0 + + +if __name__ == "__main__": + mode = sys.argv[1] if len(sys.argv) > 1 else "setup" + which = (sys.argv[2] if len(sys.argv) > 2 else "A").upper() + + if which not in PHASES: + print("phases: " + ", ".join(PHASES)) + sys.exit(2) + + sys.exit(setup(which) if mode == "setup" else verify(which))