Skip to content

expected_changes.toml promises the harness will fail on a Ph. D. regression; it won't #328

Description

@derek73

tools/differential/expected_changes.toml:301-311 deliberately leaves trailing Ph. D. healing unclassified, and says why:

Adding a suppression rule for it would risk masking a real regression in this exact shape, so it is intentionally left unclassified: if it ever starts diffing, the harness must fail.

It does not. Reproduced — a three-name corpus containing John Smith, Jr. Ph. D.:

corpus: 296 names; intentional diffs: 103; unexplained: 0
## fix(comma-family) lone post-comma piece routes to suffix/title, not first (8)
  'John Smith, Jr. Ph. D.'

The divergence is real — 1.4.0 gives suffix "Ph. D., Jr.", 2.x gives "Jr. Ph. D." — and it is absorbed silently by a rule whose prose is about something else entirely.

Why

fix(comma-family) is declared as:

name_regex = ","
fields = ["first", "title", "suffix"]

Any suffix-only diff on a comma-bearing name is a subset of that, so the rule claims it. classify() (compare.py:40-48) returns the first matching rule, and compare.py:69 sorts on only two tiers:

rules.sort(key=lambda r: not isinstance(r.get("name_regex"), str))

name_regex rules ahead of fields-only rules, stable within tier. Twelve of the thirteen rules are in the name_regex tier, so for almost every rule precedence is file order.

That is what makes this hard to fix locally. A narrower rule for the Ph. D. shape could only win by being written earlier in the file — which makes file position load-bearing again, the exact thing the sort's own comment says it exists to prevent ("sort is stable, so rules within a tier keep the order they were written").

Scope, not just this shape

The problem is structural. , and / are among the declared name_regex values, so any rule sitting above a narrower one can absorb its cases. Nothing detects it: a rule that swallows more than it describes produces unexplained: 0 and looks like a pass. The Ph. D. case surfaced only because a reviewer probed the file's own written promise.

Candidate directions

  • Give classify() a real specificity order — e.g. rank by name_regex length or by an explicit priority key — so a narrow rule beats a broad one regardless of position.
  • Report the runner-up. If two rules match, say so. Silent absorption is the failure mode; visibility may be enough without reordering anything.
  • Let a rule declare itself non-absorbing — a flag meaning "only claim a diff no other rule matches."
  • Assert the promise directly — a test that adds the Ph. D. shape to a scratch corpus and requires it to come back unexplained. Narrow, but it makes this comment honest without a harness redesign.

Interim

The divergence itself is now pinned by suffix_comma_split_phd_after_another_suffix in tests/v2/cases.py (classification fix(credential-pair-order)), and the toml comment records that the case table — not the harness — is what guards it. So nothing is unguarded today; what is wrong is the file's claim about its own behavior, and the general absorption hazard behind it.

Found during the #319 review (PR #327).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions