Merge CMap and Differences encoding at the byte level for single-byte
fonts, fixing garbled output when partial ToUnicode CMaps caused Latin-1
fallback to block the Differences path. Add Adobe-Korea1 CID-to-Unicode
predefined mapping for Identity-H CID fonts without ToUnicode streams,
and support TrueType cmap extraction via ttf-parser for embedded fonts.
Also handle suffixed glyph names (e.g. zero.tf, a.ss01) per Adobe spec.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates double-parsing of PDFs by using the lopdf document model exclusively
for ToUnicode CMap extraction. The old raw byte scanner parsed PDFs separately
from lopdf and only handled FlateDecode, while lopdf handles FlateDecode + LZW +
ASCII85. The new approach walks page fonts and Form XObject fonts via the document
API, yielding ~7x speedup on text-heavy PDFs and ~1.2x overall.
- Add FontCMaps::from_doc() with recursive Form XObject font walking
- Remove ~270 lines of raw byte scanning code (from_pdf_bytes, extract_stream_from_raw_pdf, etc.)
- Remove flate2 dependency (lopdf handles decompression internally)
- Remove dead CMap lookup fallback branches (by_name, get_with_obj, base_font_name)
- Remove unused font_base_names parameter from extract_text_from_operand
- Skip U+FFFD replacement characters in CMap decode (PDF notdef glyph markers)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --pages flag to insert <!-- Page N --> markers between pages
- Add --select-pages flag to process only specific pages (e.g. 1,3,5-10)
- Wire MarkdownOptions and page filter through process_pdf_with_config
- Fix fuzzy CMap matching that caused Cyrillic substitution on Latin text
- Fix whitespace Tj items not advancing text matrix (broke gap detection)
- Tighten single-char fragment join threshold from 0.25 to 0.20
- Gitignore debug/diagnostic binaries and remove tracked ones
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Image-only pages (e.g. scanned cover pages) were appended at the end of
output instead of appearing in page order. Now flushes tables/images for
intermediate pages with no text lines during page transitions.
Also collapses dot-leader runs (4+ dots) into " ... " for cleaner TOC
rendering, and adds debug_pages binary for inspecting per-page items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add pairwise cross-row column alignment check to find_table_regions_strict().
Real tables have fixed column X positions that repeat across rows (score ~1.0),
while paragraph text has varying word positions (score ~0.3-0.4). Regions with
average pairwise alignment score < 0.5 are now rejected.
Also remove the num_cols >= 5 content-check bypass for BodyFont mode, requiring
body-font tables to always have ≥30% data-like cell content.
Fixes false table detection across 9+ PDFs including AI_Cultural_Collections
(-730 false lines), 131212888 (-740), Data-Processing-Agreement (-108),
CEP_DN_Interest_Rates (-93). No regressions on legitimate tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Smart PDF type detection (text vs scanned) without full document load
- Text extraction using lopdf directly
- Markdown conversion with header/list/code detection
- CLI tools: detect-pdf, pdf2md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
