From 03b1bd8f0efd50cc47b1d074624d4979ccaccb81 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 6 Aug 2026 23:11:06 -0700 Subject: [PATCH] Give Boundaries subheadings, shorter paragraphs, and examples The section ran 1444 words in 11 paragraphs with no subheadings, and one paragraph was 410 words on its own. It carried good rationale that was hard to scan and impossible to link to. Eight subheadings now, and the 410-word paragraph is three of them: honorifics coming off first, what commas and dots do and do not stop, and the credentials-run rule. The longest prose paragraph is 112 words. Six doctest blocks where the prose previously asserted behavior without showing it. The reach limit and the credentials rule get examples in particular, being the two rules hardest to believe unseen. All of them run under the default configuration: the Korean and Chinese honorific cases go through vocabulary rather than a segmenter, so nothing here needs namedivider, which is installed only in the ja-extra CI job. The segmenter contrast stays prose for that reason. Em-dashes went from 22 to 0. --- docs/usage.rst | 293 +++++++++++++++++++++++++++++++------------------ 1 file changed, 188 insertions(+), 105 deletions(-) diff --git a/docs/usage.rst b/docs/usage.rst index 781a4c2..43a7de8 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -313,137 +313,220 @@ Normalize both sides before comparing across encodings. Boundaries ~~~~~~~~~~~ -Several boundaries apply to all of the above. Romanized names ("Kim -Min-jun", "Yamada Taro") are Latin script and follow the ordinary -positional rules — order genuinely varies in romanized data, so -nothing script-based applies. A name written wholly in katakana stays -positional for the reason given above, pack or no pack: it is -predominantly a transcription, and a transcription is already in the -order it should be read in. A Han transcription written with a space -instead of the 间隔号 (威廉 莎士比亚) carries nothing to distinguish it -from a native two-token name and keeps the family-first reading — the -dot is the marker, and without it there is no signal. The same holds -for a transcription typed with the Japanese middle dot (威廉・莎士比亚): -each dot carries its own script's convention, the nakaguro's is -Japanese roster formatting rather than transcription, so only the -Chinese dot rescues the source order. And a comma disables the script -behaviors that decide where a name divides, on the reasoning -``name_order`` already follows: whoever wrote the comma has already -said where the family name ends. +Several boundaries apply to all of the above. + +When the script rules don't apply +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Romanized names ("Kim Min-jun", "Yamada Taro") are Latin script and +follow the ordinary positional rules. Order genuinely varies in +romanized data, so nothing script-based applies. A name written wholly +in katakana stays positional for the reason given above, pack or no +pack: it is predominantly a transcription, and a transcription is +already in the order it should be read in. + +A Han transcription written with a space instead of the 间隔号 +(威廉 莎士比亚) carries nothing to distinguish it from a native +two-token name, and keeps the family-first reading. The dot is the +marker, and without it there is no signal. The same holds for a +transcription typed with the Japanese middle dot (威廉・莎士比亚): each +dot carries its own script's convention, the nakaguro's is Japanese +roster formatting rather than transcription, so only the Chinese dot +rescues the source order. + +A comma disables the script behaviors that decide where a name +divides, on the reasoning ``name_order`` already follows: whoever +wrote the comma has already said where the family name ends. + +Honorifics come off first +^^^^^^^^^^^^^^^^^^^^^^^^^ Honorifics and degrees follow a CJK name, and both the spaced and the -glued forms are recognized as suffixes: ``王小明 先生`` reads family -王小明 with 先生 in ``suffix``, and so do glued ``田中さん``, -``山田太郎様`` and ``김민준씨``. The honorific is split off the end of -the last token of the name before the name is split or ordered, so -those rules see the name without it — which is why ``김민준씨`` still -divides into family 김 and given 민준, and why a configured Japanese -segmenter is handed 山田太郎 rather than 山田太郎様. Neither a comma nor -a 间隔号 switches the peel off. Both say where a name divides — the -comma that the writer has already given the family name, the dot that -the pieces are a transcription's syllable groups — and an honorific is -not part of the name in either reading. So ``김, 민준씨`` reads the -same as ``김 민준씨``. What a comma does instead is say which runs are -the name: the two around a family comma, an honorific being as often -glued to the given name as to the family. Nothing past those two is in -reach — ``김, 민준 지훈씨`` peels, ``김, 민준, 지훈씨`` does not — and -the second of those runs counts only where it is name text. A run that -is nothing but credentials is not, whatever the comma looked like, and -the same test that decides the comma structure decides this too: the -run is left alone and the honorific is found in the name. Being -credentials is necessary but not enough, because the test cannot help -counting the honorific itself — a glued honorific is a suffix word, -so it is part of what makes its own run read as credentials. The run -is passed over only when the part *before* the comma has an honorific -of its own to give up; otherwise passing it over would discard the -only one there is. So ``田中さん, V.`` and ``田中さん, Ph. D.`` -give さん up exactly as -``田中さん, PhD`` does — though where the credential itself lands is -the comma's business rather than the peel's, and still differs by -spelling (``title`` for ``PhD``, ``given`` for ``V.``, and ``Ph. D.`` -in ``suffix`` beside さん). What those marks do stop is -the *split*, which is a different question and still theirs to answer. -``田中さん, 太郎`` is unchanged, and not because of its comma: the -honorific there is not at the end of the name, 太郎 is. +glued forms are recognized as suffixes: + +.. doctest:: + + >>> parse("王小明 先生").suffix + '先生' + >>> glued = parse("김민준씨") + >>> glued.family, glued.given, glued.suffix + ('김', '민준', '씨') + +The honorific is split off the end of the last token of the name +before the name is split or ordered, so those rules see the name +without it. That is why ``김민준씨`` still divides into family 김 and +given 민준, and why a configured Japanese segmenter is handed 山田太郎 +rather than 山田太郎様. + +Commas and dots +^^^^^^^^^^^^^^^ + +Neither a comma nor a 间隔号 switches the peel off. Both say where a +name divides: the comma that the writer has already given the family +name, the dot that the pieces are a transcription's syllable groups. +An honorific is not part of the name in either reading. + +.. doctest:: + + >>> parse("김, 민준씨").given == parse("김 민준씨").given + True + +What a comma does instead is say which runs are the name: the two +around a family comma, an honorific being as often glued to the given +name as to the family. Nothing past those two is in reach. + +.. doctest:: + + >>> parse("김, 민준 지훈씨").suffix # second run, still in reach + '씨' + >>> parse("김, 민준, 지훈씨").suffix # a third run, left whole + '지훈씨' + +What those marks do stop is the *split*, which is a different question +and still theirs to answer. ``田中さん, 太郎`` is unchanged, and not +because of its comma: the honorific there is not at the end of the +name, 太郎 is. + +Credentials after a comma +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The second of those two runs counts only where it is name text. A run +that is nothing but credentials is not, whatever the comma looked +like, and the same test that decides the comma structure decides this +too: the run is left alone and the honorific is found in the name. + +Being credentials is necessary but not enough, because the test cannot +help counting the honorific itself. A glued honorific is a suffix +word, so it is part of what makes its own run read as credentials. The +run is passed over only when the part *before* the comma has an +honorific of its own to give up; otherwise passing it over would +discard the only one there is. + +.. doctest:: + + >>> credential = parse("田中さん, V.") + >>> credential.family, credential.suffix + ('田中', 'さん') + +So ``田中さん, V.`` and ``田中さん, Ph. D.`` give さん up exactly as +``田中さん, PhD`` does. Where the credential itself lands is the +comma's business rather than the peel's, and still differs by +spelling: ``title`` for ``PhD``, ``given`` for ``V.``, and ``Ph. D.`` +in ``suffix`` beside さん. + +Spacing, and where the name divides +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Where a segmenter divides the name, the two spellings part company. A spaced honorific is a token boundary the writer typed, and the -segmenter is asked only where an undivided name divides — so anything -standing beside the name calls it off, honorific or not: under the +segmenter is asked only where an undivided name divides, so anything +standing beside the name calls it off, honorific or not. Under the Japanese pack ``佐藤 氏`` keeps family 佐藤, where bare ``佐藤`` would -have been divided 佐 + 藤. That is a conservative reading rather than -a principled one; a spaced honorific cannot be told apart from a -spaced given name by position, and treating it as one keeps four real -surnames whole (``佐藤 氏``, ``田中 様``, ``鈴木 先生``, ``中村 教授``) -at the price of one division it declines to make (``山田太郎 様``). A -glued honorific carries no boundary at all — its writer drew none -anywhere — so ``田中さん`` divides the way bare ``田中`` does, into -family 田 and given 中, with さん in ``suffix``. Writing the honorific -spaced is therefore also how you ask for a family name to be kept -whole on this path, short of declining the pack. +have been divided 佐 + 藤. + +That is a conservative reading rather than a principled one. A spaced +honorific cannot be told apart from a spaced given name by position, +and treating it as one keeps four real surnames whole (``佐藤 氏``, +``田中 様``, ``鈴木 先生``, ``中村 教授``) at the price of one division +it declines to make (``山田太郎 様``). A glued honorific carries no +boundary at all, its writer having drawn none anywhere, so ``田中さん`` +divides the way bare ``田中`` does, into family 田 and given 中, with +さん in ``suffix``. Writing the honorific spaced is therefore also how +you ask for a family name to be kept whole on this path, short of +declining the pack. Where the VOCABULARY divides the name the two spellings never part company, because the peel hands the same remainder to the same surname -match either way: ``김민준 씨`` and ``김민준씨`` both give family 김, -given 민준 and suffix 씨, and under the Chinese pack ``王小明 先生`` -and ``王小明先生`` both give family 王, given 小明 and suffix 先生. -Spacing the honorific is no lever there, and for Korean data there is -no pack to decline either — hangul segmentation is on by default. +match either way: + +.. doctest:: + + >>> parse("김민준 씨").family, parse("김민준씨").family + ('김', '김') + +Under the Chinese pack ``王小明 先生`` and ``王小明先生`` both give +family 王, given 小明 and suffix 先生 for the same reason. Spacing the +honorific is no lever there, and for Korean data there is no pack to +decline either, hangul segmentation being on by default. + +Which honorifics peel when glued +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The glued reading is deliberately narrower than the spaced one. A spaced honorific sits behind a boundary its writer drew; a glued one has only itself to go on, so a word peels off the end of a name only -if it could never BE the end of a name. 씨, 님, 박사, 박사님, 선생님, -교수님, さん, さま, くん, ちゃん, 様, 先生, 教授, 女士 and 小姐 -qualify; 양, 군, 氏, 博士 and 殿 do not, because 김지양 is a given -name, 田中博士 is Tanaka Hiroshi as readily as Doctor Tanaka, and some -ninety Japanese surnames end in 殿 (鵜殿, 真殿). Those stay recognized -in their spaced form, where position settles what the glued form -leaves ambiguous. 君 is recognized in neither form, since 王君 is a -complete Chinese name — though its kana spelling くん peels. Exactly -one honorific peels off a token, and the entries are whole +if it could never BE the end of a name. + +씨, 님, 박사, 박사님, 선생님, 교수님, さん, さま, くん, ちゃん, 様, +先生, 教授, 女士 and 小姐 qualify. 양, 군, 氏, 博士 and 殿 do not, +because 김지양 is a given name, 田中博士 is Tanaka Hiroshi as readily +as Doctor Tanaka, and some ninety Japanese surnames end in 殿 (鵜殿, +真殿). Those stay recognized in their spaced form, where position +settles what the glued form leaves ambiguous. 君 is recognized in +neither form, since 王君 is a complete Chinese name, though its kana +spelling くん peels. + +Exactly one honorific peels off a token, and the entries are whole honorifics rather than parts: ``김민준박사님`` gives up 박사님 entire, not 님 with 박사 left behind. +When a division was a judgment call +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + A division the parser had to choose is reported rather than hidden. -When an unspaced name has more than one vocabulary-supported split — -``남궁민수`` is 남궁 + 민수 by the two-syllable surname but 남 + 궁민수 -by the single-syllable one — the longest surname wins and the parse -records the decision as an ``AmbiguityKind.SEGMENTATION``, described -under `When the parser had to guess`_; a name with only one possible -split reports nothing. A segmenter's answer is reported on the same -kind whenever its confidence falls below the stage's floor, naming the -division and the score in the report's ``detail``:: +When an unspaced name has more than one vocabulary-supported split, +the longest surname wins and the parse records the decision as an +``AmbiguityKind.SEGMENTATION``, described under `When the parser had +to guess`_. ``남궁민수`` is 남궁 + 민수 by the two-syllable surname but +남 + 궁민수 by the single-syllable one: + +.. doctest:: + + >>> parse("남궁민수").ambiguities + (Ambiguity('segmentation': '남궁'/'민수'),) + +A name with only one possible split reports nothing. + +A segmenter's answer is reported on the same kind whenever its +confidence falls below the stage's floor, naming the division and the +score in the report's ``detail``:: "'山田太郎' splits as '山田' + '太郎' on a segmenter answer scoring 0.44, under the 0.9 confidence floor" With namedivider that line separates its two kinds of answer. A -division it states as a rule — the kanji-to-kana boundary in 高橋みなみ -— scores 1.0 and reports nothing; a division read off kanji statistics -scores far below the floor and always reports. Read the report as a -statement about the *kind* of answer, not as a measure of how likely -this particular one is to be wrong. +division it states as a rule, such as the kanji-to-kana boundary in +高橋みなみ, scores 1.0 and reports nothing; a division read off kanji +statistics scores far below the floor and always reports. Read the +report as a statement about the *kind* of answer, not as a measure of +how likely this particular one is to be wrong. A lone two-character kanji name divides one character to each side, on namedivider's rule for that length. A name that short carries no -evidence of where its own boundary falls, and one character each way is -the presumption Japanese practice makes; that is an accepted -presumption, not a measurement, so a two-character token is the shape -to check first if a division looks wrong. - -A segmenter that answers outside the token it was given — a cut at or -past the end of the text — has violated the protocol, and the parse -says so rather than hiding it: it raises ``ValueError`` naming the +evidence of where its own boundary falls, and one character each way +is the presumption Japanese practice makes. That is an accepted +presumption rather than a measurement, so a two-character token is the +shape to check first if a division looks wrong. + +When a segmenter misbehaves +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A segmenter that answers outside the token it was given, with a cut at +or past the end of the text, has violated the protocol, and the parse +says so rather than hiding it. It raises ``ValueError`` naming the offending offset and the token's length, the way an answer of the wrong type raises ``TypeError``. Declining silently would leave an -off-by-one segmenter invisible; every answer it gave would vanish and -the name would merely look undivided. The shipped ``ja_segmenter()`` -does decline — returns ``None``, token left whole — for the cases that -are *not* protocol violations: text outside the Japanese repertoire, -text too short to divide, an answer that fails to reconstruct its -input, and a score outside [0, 1]. Exceptions are the one thing that -does *not* stay inside the parse: a segmenter is your code, so its -errors propagate rather than being absorbed as content errors. +off-by-one segmenter invisible: every answer it gave would vanish and +the name would merely look undivided. + +The shipped ``ja_segmenter()`` does decline, returning ``None`` and +leaving the token whole, for the cases that are *not* protocol +violations: text outside the Japanese repertoire, text too short to +divide, an answer that fails to reconstruct its input, and a score +outside [0, 1]. + +Exceptions are the one thing that does *not* stay inside the parse: a +segmenter is your code, so its errors propagate rather than being +absorbed as content errors. The command line takes the pack but not the segmenter: ``python -m nameparser --locale ja`` has no way to attach one, so it activates