Shannon's entropy.pdf (1998 dvips 5.58 -> Distiller 3) exposed a failure
class affecting pre-2000 TeX PDFs:
- Base-14 fallback widths (new src/extractor/base14.rs): PDFs may omit
/Widths for non-embedded standard fonts; the spec requires the reader
to supply the metrics. Without them every item had width 0, breaking
space synthesis, sub/superscript detection, and table columns.
Tables generated from the Adobe Core 14 AFM data, resolved per code
through the font's Differences encoding.
- Type3 visual sizing: PK bitmap fonts use FontMatrix [1 0 0 -1 0 0]
with nominal sizes like 0.12pt; scale by FontBBox height x |matrix_y|
so drop caps and math glyphs get realistic sizes.
- Table detection: exclude script-attached small items (equation
sub/superscripts) from small-font candidates; reject tiny all-numeric
2-row grids (exponent clusters aligning as phantom tables).
- First-line-indent paragraph breaks for justified documents that
separate paragraphs only by indentation.
- Embedded drop-cap merge (two-line caps land as the second line's
first item, previously emitted mid-sentence).
- Heading robustness: classify by dominant alphanumeric font size,
ignore glyph-run lines when computing tiers, and stop rarity-promoting
display equations to headings.
On entropy.pdf: phantom-table pages 28 -> 8, glued sentences 95 -> 5,
correct sub/superscripts, drop cap restored, title/PART hierarchy fixed.
pdf-evals: 202/203 byte-identical, semantic score flat. Adds the paper's
first two pages as a snapshot fixture.
Abimael Martell
39be831f8f