Justified print breaks words at syllables; after paragraph lines are joined
with spaces those breaks survive as "de- fendant" — thousands of them in a
long document — and, when an emphasis span was split with the word, as
"Bap-** **tist".
Whether the hyphen belongs in the word cannot be decided locally ("de-
fendant" is one word, "Third- Party" is a hyphenated compound), so the
document is used as its own dictionary. For each break:
1. fragments appear joined elsewhere ("defendant") -> join plain
2. appear hyphenated elsewhere ("six-month"), or the
continuation is capitalized ("Hinds- Radix") -> keep the hyphen
3. both fragments are words the document uses and the
continuation has 4+ letters ("commercial- type") -> keep the hyphen
4. no evidence -> leave untouched
The policy contains zero hard-coded words: vocabulary evidence,
capitalization, and two length invariants. The 4-letter floor keeps
suspended hyphens intact in any language ("mid- and long-term", "klein- und
mittelgroß", "kuva- tai video") because conjunctions are near-universally
1-3 letters. Fragments over 40 combined characters are fused reading-order
noise and are never joined. The vocabulary is collected after scrubbing the
break pairs themselves and excludes fenced code blocks; table rows and code
blocks are never rewritten. Split emphasis spans rejoin inside their
markers. Runs under the existing fix_hyphenation option (default on).
On a 1,370-page justified legal reporter this rejoins ~8,000 broken words
(98.8% of breaks; evidence-less ones stay visibly intact); word recall
against a reference extraction rises from 97.8% to 99.1%. No "six-month" ->
"sixmonth" class errors, and no fused-column corruption by construction:
no rule joins without evidence.
Regression-checked against a ~200-document corpus with semantic scoring
against an OCR baseline: zero regressions. Three in-repo fixture snapshots
regenerated with each diff inspected. 17 unit tests cover every rule, the
vocabulary scrubbing and code-block exclusion, the length gates, chained
breaks, mismatched emphasis markers, accented and Cyrillic words, German
and Finnish suspended hyphens, and the table/code skips.
Abimael Martell
2543abe371