Abimael MartellandClaude Opus 4.6 aa387503c3 fix(fonts): suppress garbage CID text on OCR-flagged pages (#9) ...

When Identity-H or Type3 fonts lack a ToUnicode CMap and the
CID-as-Unicode passthrough doesn't produce valid text, the raw CID
byte values appear as mojibake (random Latin Extended characters mixed
with C1 control codes).

The detector already flags these pages in pages_needing_ocr, but the
markdown pipeline still emitted the garbage. Now, for TextBased PDFs,
we check each OCR-flagged page's extracted text for CID garbage
(C1 control characters U+0080–U+009F at ≥5% density) and strip items
from pages that fail the check.

This is scoped to TextBased PDFs only — Mixed PDFs flag pages for OCR
due to template images, not font encoding issues.

Adds test fixture (shinagawa_identity_h.pdf) and integration test.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-23 10:35:45 -07:00

50 KiB

Raw History

View Raw