* fix(extractor): strip PDF comments that break lopdf content stream parsing
Some PDF generators (notably PD4ML used by school districts) embed
comments (% to end of line) in content streams. lopdf's Content::decode
parser fails to parse operators that follow comments, silently dropping
ET (end text) and Q (restore graphics state) operators. This caused
entire pages to produce 0 text items despite having valid text.
Fix: pre-process content streams to strip comments before parsing.
Comments inside string literals (parentheses) and hex strings are
preserved. The comment is replaced with a space to maintain token
separation.
Impact: fixes 13+ school district PDFs and similar PD4ML-generated
documents that were producing near-empty output (454 → 31,955 chars
for a 22-page school improvement plan).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(detect): flag sparse-extraction pages as needing OCR
When a TEXT-BASED PDF produces <50 chars/page average with <500 total
chars, flag all pages as needing OCR. This catches PDFs where the
extractable text is minimal (form templates, image-heavy layouts)
and the bulk of content requires OCR to access.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(detect): improve CID mojibake detection for Japanese/CJK PDFs
Extend is_cid_garbage to detect CID-as-Latin-1 mojibake: when ≥40%
of characters are high Latin-1 (U+00A0-00FF) and <33% are ASCII
letters, the text is likely CID values misinterpreted as Latin-1
characters (common in Japanese/CJK PDFs with broken ToUnicode CMaps).
Also add sparse-extraction OCR flagging: TEXT-BASED PDFs with
<50 chars/page and <500 total chars get all pages flagged for OCR.
Impact: Softbank Japanese PDFs now produce empty output with
pages_needing_ocr=all instead of mojibake garbage. Korean PDFs
with valid extraction (nexo-price-en) remain unaffected.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(detect): sparse extraction check only when markdown is generated
The sparse-extraction OCR check was triggering in Analyze mode where
markdown is not generated (md_len=0), causing false OCR flags on
every PDF processed via detect-pdf --analyze.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
