Subset fonts number GIDs by document encounter order, not sorted order.
The sequential remap scrambled character mappings, losing 2,532+ CJK
characters. Now promotes TrueType cmap fallback when it has more entries
than the primary ToUnicode CMap. Also adds CJK scoring to score_text()
as defense-in-depth.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add pdf.js binary CMap (bcmap) parser for Japan1/GB1/CNS1 CID fonts
- Support inline ToUnicode streams (not just object references)
- Add CMapDecisionCache for heuristic primary vs remapped CMap selection
- Build fallback CMaps from embedded font data and CIDSystemInfo
- Handle simple fonts without ToUnicode via embedded font cmap
- Add Symbol/Wingdings/ZapfDingbats font decoding fallback
- Support usecmap chaining in both text and binary CMap formats
- Scan XObject Forms for text operators in detector
- Change default detection strategy from EarlyExit to Sample(8)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Some PDF generators subset-embed fonts by renumbering GIDs sequentially
but fail to update the ToUnicode CMap, which still references original
GID values. Detect this mismatch (Identity-H, no CIDToGIDMap, W array
starts at CID ≤ 2 but CMap min CID > 2) and remap to sequential positions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge CMap and Differences encoding at the byte level for single-byte
fonts, fixing garbled output when partial ToUnicode CMaps caused Latin-1
fallback to block the Differences path. Add Adobe-Korea1 CID-to-Unicode
predefined mapping for Identity-H CID fonts without ToUnicode streams,
and support TrueType cmap extraction via ttf-parser for embedded fonts.
Also handle suffixed glyph names (e.g. zero.tf, a.ss01) per Adobe spec.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates double-parsing of PDFs by using the lopdf document model exclusively
for ToUnicode CMap extraction. The old raw byte scanner parsed PDFs separately
from lopdf and only handled FlateDecode, while lopdf handles FlateDecode + LZW +
ASCII85. The new approach walks page fonts and Form XObject fonts via the document
API, yielding ~7x speedup on text-heavy PDFs and ~1.2x overall.
- Add FontCMaps::from_doc() with recursive Form XObject font walking
- Remove ~270 lines of raw byte scanning code (from_pdf_bytes, extract_stream_from_raw_pdf, etc.)
- Remove flate2 dependency (lopdf handles decompression internally)
- Remove dead CMap lookup fallback branches (by_name, get_with_obj, base_font_name)
- Remove unused font_base_names parameter from extract_text_from_operand
- Skip U+FFFD replacement characters in CMap decode (PDF notdef glyph markers)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --pages flag to insert <!-- Page N --> markers between pages
- Add --select-pages flag to process only specific pages (e.g. 1,3,5-10)
- Wire MarkdownOptions and page filter through process_pdf_with_config
- Fix fuzzy CMap matching that caused Cyrillic substitution on Latin text
- Fix whitespace Tj items not advancing text matrix (broke gap detection)
- Tighten single-char fragment join threshold from 0.25 to 0.20
- Gitignore debug/diagnostic binaries and remove tracked ones
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lopdf's get_font_encoding() re-parses ToUnicode CMap streams via nom for
every Tj/TJ operator. Cache the encoding once per font during page setup.
Also add binary search for CMap range lookup and de-dup CMap extraction.
Reduces Arabic PDF processing from 45s to <1s (53x speedup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Abimael MartellandClaude Opus 4.6