Refuse to convert when the bytes do not identify the encoding

[amrali-eg]

The audit's largest remaining risk category: 262 of 5,078 files where several encodings read the bytes and disagree about what they say. Single-byte code pages map 256 values independently, so a file valid in windows-1252 is equally valid in iso-8859-1, and nothing in the bytes decides. Detection answers anyway — and on short or ASCII-heavy input that answer is close to a guess, which EC then acted on.

Three outcomes, not two

Class	Meaning	Action
Unambiguous	One codec reads these bytes	convert
TextEquivalent	Several read them, all agree on the text	convert
TextChanging	Several read them and disagree	refuse

The middle case matters as much as the last. A pure-ASCII file is ambiguous in label and identical in text — refusing it would protect nothing. The test is whether the candidates produce different Unicode, not whether more than one can decode.

Observed end to end:

ambiguous_cp1252.txt   iso-8859-1  →  Error      (refused)
ambiguous_koi8r.txt    koi8-r      →  Error      (refused)
ascii_only.txt         us-ascii    →  Converted
determined_utf8.txt    utf-8       →  Unchanged
determined_sjis.txt    shift_jis   →  Converted
determined_big5.txt    big5        →  Converted
determined_utf16.txt   utf-16      →  Converted

Candidate set and placement

Candidates come from EC's supported encodings, deduplicated by code page — never from a corpus. The audit already demonstrated that deriving candidates from test data makes the answer depend on what the tests happened to contain. That list moved from MainForm to TextEncoding so the engine can reach it.

The decision is made beside the detection that produced it and carried on the entry, so the conversion path reads a verdict rather than re-deriving one. A caller who supplies the source encoding rather than detecting it is unaffected — there is nothing ambiguous about an answer somebody gave.

Two mistakes on the way, both caught by the existing tests

Counting rival readings alone refused every UTF-8 file, because iso-8859-1 "reads" it too. A codec that cannot refuse anything is not offering an alternative. Fixed by requiring the detected codec to have no hold on the bytes before rivals count.

Measuring that hold by bit flips alone refused UTF-16, which survives nearly any flip — most produce another valid character. Fixed by also deleting a byte, which tests alignment, the only structure a fixed-width encoding has.

The measurement is deterministic — evenly-spaced probes, not random ones — after an earlier version gave answers that moved with the seed on short files. Whether a conversion is refused should not depend on a random draw.

The threshold sits just above zero because the separation is not a matter of degree: unconstrained single-byte pages reject exactly nothing, and every structured encoding measured on real text lands at 0.111 or above.

The message

The encoding could not be determined uniquely from the file's contents. iso-8859-1 and cp866, ibm852, ibm855, iso-8859-13, and 17 more all match this file and would produce different text. No conversion was performed; specify the source encoding explicitly to convert it.

Rather than "low confidence", which gives a user nothing to act on.

345 tests passing (was 330).

🤖 Generated with Claude Code

---

Update: -From folded in

The refusal told users to specify the source encoding, and the CLI had no way to do it — a safety feature issuing instructions its own interface could not follow. Fixed before merge:

EncodingChecker.exe -BasePath . -Include "*.txt" -From "windows-1252" -Target "utf-8"

It replaces detection and nothing else. Verified end to end:

Auto-detect on ambiguous file	Error + advice
Same file with -From windows-1252	Converted, text intact
-From euc-jp on undecodable bytes	still refused — SourceDecodeError
-From + target that cannot hold the content	still refused
-From + failing backup	still refused
-From + -DetectOnly / -Validate	rejected as meaningless

It changes the interpretation, not the permission: naming koi8-r for windows-1252 bytes produces koi8-r text. The user is choosing a reading, not waiving a check. The recovery record is still written.

Also adds AmbiguityReason (SingleCandidate, StructurallyDetermined, MultipleCodecsSameText, MultipleCodecsDifferentText, ExplicitlySpecified) and records per entry whether the encoding was detected or specified — different claims, since detection can be wrong in ways an explicit choice cannot.

354 tests passing (was 345).

[amrali-eg]

Phase C status: CLI-complete, GUI override pending.

Automatic refusal on text-changing ambiguity	✓
Actionable CLI override (-From)	✓
Explicit codec still subject to every safety check	✓
GUI source-encoding override	✗ — tracked for Phase E

The GUI plumbing exists (ScanDirectoryOptions.SourceCharset) but has no UI, so a GUI user hitting an ambiguity refusal currently has no in-app escape. That belongs with the risk-aware confirmation work, where the interface needs to surface ambiguity, alternatives, consequences and the source-selection control together rather than as an isolated dropdown.