- Remove `process_mode` from `MarkdownOptions` (it controlled the
pipeline, not markdown formatting)
- Remove dead `text` field from `PdfProcessResult`
- Add `PdfOptions` builder consolidating mode, detection, markdown,
and page filter configuration
- Add convenience functions: `detect_pdf()`, `detect_pdf_mem()`,
`process_pdf_with_options()`, `process_pdf_mem_with_options()`
- Eliminate double document parsing: load once, share between
detection and extraction via internal `pub(crate)` helpers
- Deprecate old `process_pdf_with_config*` functions (kept as shims)
- Update binaries to use new API
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add has_encoding_issues field to PdfProcessResult that detects garbled
text from broken ToUnicode CMaps (U+FFFD replacement characters or
systematic dollar-as-space substitution). Surfaced in JSON output so
clients can fall back to OCR for affected PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hand-rolled JSON escaping only handled \, ", and \n but missed
tabs, carriage returns, and other control characters (U+0000..U+001F),
producing invalid JSON that Python's json.loads would reject. Add a
proper json_escape() function covering all required escapes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces a ProcessMode enum that controls how far the PDF pipeline
runs, enabling fast document triage without paying extraction or
markdown conversion costs. Exposed via --detect-only and --analyze
flags in both pdf2md and detect-pdf CLIs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add LayoutComplexity struct to PdfProcessResult so callers can detect
when a PDF has complex layout (tables or multi-column text) and decide
whether to use the extracted markdown or fall back to OCR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --pages flag to insert <!-- Page N --> markers between pages
- Add --select-pages flag to process only specific pages (e.g. 1,3,5-10)
- Wire MarkdownOptions and page filter through process_pdf_with_config
- Fix fuzzy CMap matching that caused Cyrillic substitution on Latin text
- Fix whitespace Tj items not advancing text matrix (broke gap detection)
- Tighten single-char fragment join threshold from 0.25 to 0.20
- Gitignore debug/diagnostic binaries and remove tracked ones
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
For mixed PDFs, callers can now see exactly which pages need OCR instead
of re-analyzing the document. Phase 2 scan iterates all pages for Mixed
PDFs (caching sampled results), while TextBased gets empty and
Scanned/ImageBased gets all pages.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Image-only pages (e.g. scanned cover pages) were appended at the end of
output instead of appearing in page order. Now flushes tables/images for
intermediate pages with no text lines during page transitions.
Also collapses dot-leader runs (4+ dots) into " ... " for cleaner TOC
rendering, and adds debug_pages binary for inspecting per-page items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix Differences encoding path dropping standard characters when only
ligature bytes matched the sparse encoding map. Now combines Differences
entries with Latin-1 fallback for printable bytes instead of using
filter_map which silently dropped unmapped bytes.
Add expand_ligatures() to replace Unicode ligature characters (U+FB00-FB04)
with their ASCII components (ff, fi, fl, ffi, ffl) at all TextItem
creation sites. Add underscore-variant glyph names (f_f, f_f_i, f_f_l).
Eliminates 669 ligature corruptions across the eval suite.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Smart PDF type detection (text vs scanned) without full document load
- Text extraction using lopdf directly
- Markdown conversion with header/list/code detection
- CLI tools: detect-pdf, pdf2md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
