Skip to content

Honorifics written with a fullwidth or ideographic full stop aren't recognized (김민준 씨.) #322

Description

@derek73

A CJK honorific reaches the vocabulary only when the name is typed with the exact codepoints the lexicon expects. Three compatibility spellings miss it, all silently — the honorific lands in a name field and nothing is reported:

parse("김민준 씨.")   # family 김, given 민준, suffix 씨.   <- ASCII period, works
parse("김민준 씨.")   # given 김, middle 민준, family 씨.  <- U+FF0E fullwidth
parse("김민준 씨。")   # given 김, middle 민준, family 씨。  <- U+3002 ideographic
parse("김민준 씨。")   # given 김, middle 민준, family 씨。   <- U+FF61 halfwidth ideographic

The fullwidth and ideographic stops are the ones a CJK writer is more likely to type than the ASCII one, since they are what a Japanese or Chinese IME produces by default.

Three different mechanisms, one candidate fix

1. Full stops. _lexicon._normalize does .strip("."), ASCII only, so _normalize("씨.") is "씨." and the vocabulary lookup misses before any predicate is consulted.

2. Halfwidth katakana. is_initial("ラ.") is True while is_initial("ラ.") is False. _SCRIPT_RANGES deliberately excludes U+FF65–FF9F ("legacy bank/CSV data uses it, but it is a separate normalization problem"), and _policy._NO_INITIALS silently inherits that exclusion — so the constant does not enforce its own stated rule on the halfwidth spelling of the same character. Nothing misroutes today, because every kana honorific is multi-character, but the inconsistency is now written into a documented rule.

3. NFD hangul. Decomposed Korean misses where decomposed Japanese does not:

parse(NFD("김민준, 씨."))    # title 씨., family 김민준        <- wrong
parse(NFD("田中さん, 様."))   # family 田中, suffix さん, 様.   <- right

Hangul decomposes into jamo (3+ characters, so no single-character shape matches); Han does not decompose at all. #272 deliberately kept segmentation matching raw, so this is a known consequence — but the ko/ja asymmetry is not recorded anywhere.

The shared decision

An NFKC fold for vocabulary lookup only would address all three — U+FF0E→., , and NFD→NFC by composition. That is a single change with one risk profile to evaluate, rather than three patches.

The anti-#100 invariant makes this delicate: token spans must index the original string exactly, so any normalization has to happen at lookup and must never rewrite token text. _normalize already works this way, which is why it is the natural home.

Found while reviewing #320 (PR #321). Not a regression — master agrees on every row above. #320's release note is scoped to the ASCII period so it does not imply otherwise.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions