Justified print breaks words at syllables; after paragraph lines are joined
with spaces those breaks survive as "de- fendant" — thousands of them in a
long document — and, when an emphasis span was split with the word, as
"Bap-** **tist".
Whether the hyphen itself belongs in the word cannot be decided locally
("de- fendant" is one word, "Third- Party" is a hyphenated compound), so the
document is used as its own dictionary. For each break:
1. fragments appear joined elsewhere ("defendant") -> join plain
2. appear hyphenated elsewhere ("six-month") -> keep the hyphen
3. capitalized continuation ("Hinds- Radix") -> keep the hyphen
4. continuation is to/and/or ("mid- to long-term") -> suspended
hyphen, leave untouched
5. both fragments are words the document uses
("commercial- type") -> keep the hyphen
6. default (syllable breaks dominate) -> join plain
The vocabulary is collected after scrubbing the break patterns themselves,
otherwise every broken word donates its own fragments ("evi", "dence") and
rule 5 misfires. Split emphasis spans rejoin inside the markers
("**Bap-** **tist**" -> "**Baptist**") when the markers match. Table rows
and fenced code blocks are left untouched.
Runs under the existing fix_hyphenation option (default on). On a 1,370-page
legal reporter this rejoins ~8,000 broken words with a single deliberate
survivor (a genuine suspended hyphen); word recall against a reference
extraction rises from 97.8% to 99.1%, and no "six-month" -> "sixmonth" class
errors are introduced (a failure mode common in extractors that strip
line-break hyphens unconditionally).
Regression-checked against a ~200-document corpus with semantic scoring
against an OCR baseline. Three in-repo fixture snapshots regenerated; each
diff inspected (syllable joins, kept compounds).
Adds 11 unit tests covering every rule, the vocabulary scrubbing, chained
breaks, mismatched emphasis markers, accented words, and the table/code
skips.
Abimael Martell
5ebb690170