pdf-inspector

mu-ref/pdf-inspector

Fork 0

Author	SHA1	Message	Date
Abimael Martell	7ba1966b54	layout: indent paragraph breaks, heading reclassification, table script filter Layout heuristics originally developed alongside the base-14 metrics fix and split out of it. These are NOT bug fixes — they are new heuristics, and they apply to every document regardless of vintage. - First-line-indent paragraph breaks: justified documents that separate paragraphs by indentation only (no vertical gap) previously came out as walls of text - Heading classification by dominant alphanumeric font size, so a leading drop cap or oversized math delimiter cannot reclassify a body line; tier computation filters glyph-run lines and requires uniform sizing - Table detection: exclude sub/superscripts attached at a script baseline offset from candidates; reject tiny all-numeric grids in the small-font pass - Embedded drop-cap merge for two-line caps that land as the second line's first item Corpus impact: ~98 of 184 eval documents on top of the font-metrics branch (full stack: 110 of 184; font metrics alone: 12). KNOWN REGRESSIONS — this is why it is split out and left as a draft: 1. Heading hierarchy collapse. In revista63 every heading moves h3 -> h1 because the new tier filters drop that document's larger tiers, so everything at body+2pt becomes top-level. 2 of 60 sampled documents lose heading depth this way. 2. The indent rule fires on non-prose. Its guard only asks whether the previous line ended ragged, which is true of URL lists, file listings, poems, and chat transcripts. In the mythos system card it splits a protein-sequence line and a URL list; the resulting short fragment is then promoted to a heading. Same mechanism yields the junk '#### Rr Fy' heading in 2103_07786. 3. 10 documents lose table rows with no replacement. The semantic composite is flat (+0.0011) but char-accuracy barely registers heading damage ('###' -> '#' is two characters), so that is not evidence these are harmless.	2026-08-03 16:16:13 -07:00
Abimael Martell	1228a2c2ca	fix(extractor): supply built-in metrics for non-embedded base-14 fonts PDFs may legally omit /Widths for non-embedded standard fonts (Times, Helvetica, Courier, Symbol, ZapfDingbats) — the spec requires the reader to supply the metrics. We returned None, so every glyph advanced 0 and each text item got width 0, silently breaking every gap-based heuristic downstream: space synthesis, sub/superscript detection, table column detection, heading merging. - src/extractor/base14.rs: Adobe Core-14 AFM width tables keyed by Unicode char, plus the standard Symbol/ZapfDingbats encoding vectors (their glyphs sit at byte positions unrelated to Latin text, so widths must resolve through the built-in encoding, not cp1252) - Width resolution order: Differences -> built-in encoding -> the same cp1252-style fallback the text decoder uses, so a code's advance always matches the character we emit for it - Type3 visual sizing: PK bitmap fonts (dvips) use FontMatrix [1 0 0 -1 0 0] with nominal sizes like 0.12pt; scale by FontBBox height x \|matrix_y\|. Applied in the page-stream and Form XObject paths. Indirect numeric array elements are resolved before use. Effect on Shannon's 'A Mathematical Theory of Communication' (1998 dvips/Distiller, the reported case): glued sentences 95 -> 5. Corpus impact: 12 of 184 eval documents, e.g. Data-Processing-Agreement recovers a paragraph that a phantom table had shredded into cells. Layout heuristics tuned on the same document (indent-based paragraph breaks, heading reclassification, table script filtering) are held back for a separate PR — they change ~98 further documents and need to be justified against the corpus, not against one PDF.	2026-08-03 16:15:04 -07:00

Author

SHA1

Message

Date

Abimael Martell

7ba1966b54

layout: indent paragraph breaks, heading reclassification, table script filter

Layout heuristics originally developed alongside the base-14 metrics fix
and split out of it. These are NOT bug fixes — they are new heuristics,
and they apply to every document regardless of vintage.

- First-line-indent paragraph breaks: justified documents that separate
  paragraphs by indentation only (no vertical gap) previously came out
  as walls of text
- Heading classification by dominant alphanumeric font size, so a
  leading drop cap or oversized math delimiter cannot reclassify a body
  line; tier computation filters glyph-run lines and requires uniform
  sizing
- Table detection: exclude sub/superscripts attached at a script
  baseline offset from candidates; reject tiny all-numeric grids in the
  small-font pass
- Embedded drop-cap merge for two-line caps that land as the second
  line's first item

Corpus impact: ~98 of 184 eval documents on top of the font-metrics
branch (full stack: 110 of 184; font metrics alone: 12).

KNOWN REGRESSIONS — this is why it is split out and left as a draft:

1. Heading hierarchy collapse. In revista63 every heading moves h3 -> h1
   because the new tier filters drop that document's larger tiers, so
   everything at body+2pt becomes top-level. 2 of 60 sampled documents
   lose heading depth this way.
2. The indent rule fires on non-prose. Its guard only asks whether the
   previous line ended ragged, which is true of URL lists, file
   listings, poems, and chat transcripts. In the mythos system card it
   splits a protein-sequence line and a URL list; the resulting short
   fragment is then promoted to a heading. Same mechanism yields the
   junk '#### Rr Fy' heading in 2103_07786.
3. 10 documents lose table rows with no replacement.

The semantic composite is flat (+0.0011) but char-accuracy barely
registers heading damage ('###' -> '#' is two characters), so that is
not evidence these are harmless.

2026-08-03 16:16:13 -07:00

Abimael Martell

1228a2c2ca

fix(extractor): supply built-in metrics for non-embedded base-14 fonts

PDFs may legally omit /Widths for non-embedded standard fonts (Times,
Helvetica, Courier, Symbol, ZapfDingbats) — the spec requires the reader
to supply the metrics. We returned None, so every glyph advanced 0 and
each text item got width 0, silently breaking every gap-based heuristic
downstream: space synthesis, sub/superscript detection, table column
detection, heading merging.

- src/extractor/base14.rs: Adobe Core-14 AFM width tables keyed by
  Unicode char, plus the standard Symbol/ZapfDingbats encoding vectors
  (their glyphs sit at byte positions unrelated to Latin text, so widths
  must resolve through the built-in encoding, not cp1252)
- Width resolution order: Differences -> built-in encoding -> the same
  cp1252-style fallback the text decoder uses, so a code's advance always
  matches the character we emit for it
- Type3 visual sizing: PK bitmap fonts (dvips) use FontMatrix
  [1 0 0 -1 0 0] with nominal sizes like 0.12pt; scale by FontBBox height
  x |matrix_y|. Applied in the page-stream and Form XObject paths.
  Indirect numeric array elements are resolved before use.

Effect on Shannon's 'A Mathematical Theory of Communication' (1998
dvips/Distiller, the reported case): glued sentences 95 -> 5. Corpus
impact: 12 of 184 eval documents, e.g. Data-Processing-Agreement
recovers a paragraph that a phantom table had shredded into cells.

Layout heuristics tuned on the same document (indent-based paragraph
breaks, heading reclassification, table script filtering) are held back
for a separate PR — they change ~98 further documents and need to be
justified against the corpus, not against one PDF.

2026-08-03 16:15:04 -07:00

Compare commits

Diff Content Not Available