Strip leading/trailing digit sequences (page numbers) before frequency
comparison so headers like "Page 5" and "Page 6" are treated as identical.
Group TextLines at the same Y position into Y-bands and propagate removal
to all siblings when any member is stripped. Increase EDGE_LINE_COUNT from
4 to 5 to cover 5-row form column headers (e.g., IRS p1244).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bumps EDGE_LINE_COUNT from 3 to 4 to strip repeated form column headers
that sit just inside the page margin (4th-from-edge Y position).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add strip_repeated_lines() preprocessing step that removes running
headers and footers before markdown conversion. Uses multiple guard
rails to avoid false positives:
- Edge-line detection: only considers lines among the first/last 3
distinct Y positions on each page (not percentage-based margins)
- Y-position consistency: requires low variance across pages (stddev
< 5% of page span) to distinguish headers from scattered table content
- Position-aware removal: only strips instances at page edges, preserving
body content that happens to match a header/footer text
- Skips structural lines (headings, list items), short lines (<10 chars),
and decorative separators (repeated single characters)
- Frequency threshold: >= max(3, page_count * 30%) distinct pages
New strip_headers_footers option on MarkdownOptions (default: true).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove `process_mode` from `MarkdownOptions` (it controlled the
pipeline, not markdown formatting)
- Remove dead `text` field from `PdfProcessResult`
- Add `PdfOptions` builder consolidating mode, detection, markdown,
and page filter configuration
- Add convenience functions: `detect_pdf()`, `detect_pdf_mem()`,
`process_pdf_with_options()`, `process_pdf_mem_with_options()`
- Eliminate double document parsing: load once, share between
detection and extraction via internal `pub(crate)` helpers
- Deprecate old `process_pdf_with_config*` functions (kept as shims)
- Update binaries to use new API
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The R6 encryption fix has been merged upstream, so switch from the
fork back to the canonical repository.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
EM SPACE from PDF ActualText entries was lost by text.trim(), breaking
spacing after bullets and numbered list markers. Normalizing to ASCII
space lets should_join_items detect word boundaries naturally. NBSP
(U+00A0) is excluded as it's handled by coordinate-based spacing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Soft hyphens, zero-width spaces, BOM, ZWNJ/ZWJ, and word joiners now get
stripped in expand_ligatures(). Also call expand_ligatures() on the ActualText
code path which was the only TextItem creation site missing it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use abimaelmartell/lopdf firecrawl/fix-r6-encryption branch which
accepts /O and /U values longer than 48 bytes in V5/R6 encrypted PDFs.
This unblocks 4 encrypted PDFs in the bench suite that were previously
rejected with InvalidHashLength.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Map lopdf's Unimplemented("encrypted...") error to PdfError::Encrypted
instead of falling through to PdfError::Parse. Retry all Document::load
calls with an empty password when encryption is detected, so
owner-password-only PDFs can be opened. If the retry also fails, the
user now sees a clear "PDF is encrypted" message.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unit test verifying short sub-header rows (e.g. month names) are not
merged as continuation rows in table formatting. Snapshot test for
2013_app2.pdf pinning the full row-stripe table output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When PDF tables use full-width alternating row shading (row-stripe rects),
the normal rect grid detection fails because all rects share the same X/width,
collapsing to ~1 column. Add a fallback that uses rect Y-edges for rows and
text X-position clustering for columns, with a lower 15pt threshold to
separate narrow columns like row numbers and dates.
Also fix continuation-row merging in table formatting to not merge short
single-cell rows (≤5 chars) that are section sub-headers (e.g. month names).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The chest-wall PDF uses $ as separator but also has trailing $ after
spaces, giving only 13.7% letter-dollar-letter ratio (594 of 4332).
Add absolute count threshold (>20) alongside the ratio check so both
concentrated and dispersed substitution patterns are caught.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add has_encoding_issues field to PdfProcessResult that detects garbled
text from broken ToUnicode CMaps (U+FFFD replacement characters or
systematic dollar-as-space substitution). Surfaced in JSON output so
clients can fall back to OCR for affected PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Symbol/Wingdings fonts often map codes to PUA (U+F000-F0FF) via ToUnicode
CMaps, producing invisible/tofu characters in markdown output. Add
clean_symbol_pua() post-processing to extract_text_from_operand() that
converts these to standard Unicode: bullets to U+2022, checkmark to U+2713,
and ASCII/Latin-1 range by stripping the F000 offset.
Eliminates 1,405 PUA characters across 25+ eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add pdf.js binary CMap (bcmap) parser for Japan1/GB1/CNS1 CID fonts
- Support inline ToUnicode streams (not just object references)
- Add CMapDecisionCache for heuristic primary vs remapped CMap selection
- Build fallback CMaps from embedded font data and CIDSystemInfo
- Handle simple fonts without ToUnicode via embedded font cmap
- Add Symbol/Wingdings/ZapfDingbats font decoding fallback
- Support usecmap chaining in both text and binary CMap formats
- Scan XObject Forms for text operators in detector
- Change default detection strategy from EarlyExit to Sample(8)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Some PDF generators subset-embed fonts by renumbering GIDs sequentially
but fail to update the ToUnicode CMap, which still references original
GID values. Detect this mismatch (Identity-H, no CIDToGIDMap, W array
starts at CID ≤ 2 but CMap min CID > 2) and remap to sequential positions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hand-rolled JSON escaping only handled \, ", and \n but missed
tabs, carriage returns, and other control characters (U+0000..U+001F),
producing invalid JSON that Python's json.loads would reject. Add a
proper json_escape() function covering all required escapes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces a ProcessMode enum that controls how far the PDF pipeline
runs, enabling fast document triage without paying extraction or
markdown conversion costs. Exposed via --detect-only and --analyze
flags in both pdf2md and detect-pdf CLIs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add LayoutComplexity struct to PdfProcessResult so callers can detect
when a PDF has complex layout (tables or multi-column text) and decide
whether to use the extracted markdown or fall back to OCR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a PDF page has a graph directly above a table (e.g. Figure 2 above
Table II in real-estate-pricing.pdf), the heuristic table detector would
merge graph axis labels and legend items into the table, producing a
corrupted result.
Add rect hint regions: when a small cluster of cell-border rects (4-6)
fails full grid validation (e.g. only row borders, no column dividers),
extract their Y bounding box as a "hint region". The heuristic detector
then runs separately on items inside vs outside hint regions, preventing
unrelated content from being merged into tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rects spanning multiple grid rows (e.g. classification labels) now have
their text consolidated into the first sub-row via propagate_merged_cells().
Also prevent continuation-row merge from folding body data into the header.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 5 stripped public-domain PDF fixtures and golden markdown snapshots
for CI regression testing. Fix non-deterministic output caused by
HashMap iteration order in font stats, table heuristics, and rect
clustering by adding deterministic tie-breaking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The snap_edges tolerance of 2.0 was too tight for PDFs where header and
body cell boundaries differ by ~3pt due to cell padding. This created
phantom empty columns that caused valid tables to be rejected. Increase
snap and cell coverage tolerances from 2.0/3.0 to 6.0. Also add debug
logging to rect-based table detection pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge CMap and Differences encoding at the byte level for single-byte
fonts, fixing garbled output when partial ToUnicode CMaps caused Latin-1
fallback to block the Differences path. Add Adobe-Korea1 CID-to-Unicode
predefined mapping for Identity-H CID fonts without ToUnicode streams,
and support TrueType cmap extraction via ttf-parser for embedded fonts.
Also handle suffixed glyph names (e.g. zero.tf, a.ss01) per Adobe spec.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates double-parsing of PDFs by using the lopdf document model exclusively
for ToUnicode CMap extraction. The old raw byte scanner parsed PDFs separately
from lopdf and only handled FlateDecode, while lopdf handles FlateDecode + LZW +
ASCII85. The new approach walks page fonts and Form XObject fonts via the document
API, yielding ~7x speedup on text-heavy PDFs and ~1.2x overall.
- Add FontCMaps::from_doc() with recursive Form XObject font walking
- Remove ~270 lines of raw byte scanning code (from_pdf_bytes, extract_stream_from_raw_pdf, etc.)
- Remove flate2 dependency (lopdf handles decompression internally)
- Remove dead CMap lookup fallback branches (by_name, get_with_obj, base_font_name)
- Remove unused font_base_names parameter from extract_text_from_operand
- Skip U+FFFD replacement characters in CMap decode (PDF notdef glyph markers)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detect newspaper-style two-column layouts (dense independent text
flows) and emit columns sequentially instead of Y-interleaving them.
This fixes academic papers where column text was being merged across
columns at the same Y position.
Key changes:
- Add is_newspaper_layout() heuristic (≥15 lines/col, >50% Y-collision)
- Add split_column_stragglers() to separate core column clusters from
header remnants and per-word items that landed in column buckets
- Filter wide items (>60% page width) from column detection histogram
so full-width text doesn't fill the gutter and prevent detection
- Emit: above spanning → core columns sequentially → column stragglers
sequentially → below spanning items
- Fix markdown paragraph break to detect backward Y jumps (y_gap.abs())
so sequential columns on the same page get proper paragraph breaks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detect RTL scripts (Hebrew, Arabic, Syriac, etc.) by Unicode ranges and
sort line items by X descending when a line is predominantly RTL. Fix
gap calculation in should_join_items() to be direction-neutral. Extract
WMode from Type0 font dictionaries for vertical text foundation. Add
missing Korean Hangul ranges to is_cjk_char().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like privacy notices have 6-7 distinct visual table sections per
page, each with its own set of cell rects. Previously all rects were
merged into a single giant sparse grid that failed validation. Now
spatially connected rects are clustered via union-find before running
grid detection independently on each cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add U+FB05/FB06 (ſt/st→st) to ligature table matching PyMuPDF's full set
- Refactor expand_ligatures to single-pass match loop (clippy fix)
- Detect dot-leader lines (TOC entries) and prevent list continuation
merging, which was joining sub-entries onto parent numbered items
- Force newline breaks between consecutive dot-leader lines in paragraphs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace section-based column merge (which emitted all left-column lines
then all right-column lines) with Y-interleaved merge that combines
lines at the same Y position from different columns into a single line.
This fixes tabular layouts (like court dockets) where rows span across
columns — previously hearing descriptions were detached from their
entries and line breaks between rows were lost.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fillable PDF forms store typed values in the AcroForm dictionary, not in
the page content stream. This adds extraction of form field names and
values, emitting them as TextItems positioned at the field's Rect so they
flow naturally into the markdown pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
effective_font_size() was computed from the text matrix alone, ignoring
CTM scaling. In PDFs with a small CTM scale (e.g. 0.24×) and large Tm
values (e.g. 58), font_size was inflated (58pt instead of ~14pt), causing
the merge threshold to bridge inter-column gaps and merge two-column
text into single lines. Now compute the combined matrix (text_matrix × CTM)
before calling effective_font_size() at all 6 call sites.
Also adds PdfRect extraction from `re` operators for future table-grid
detection, and related plumbing changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Emit ActualText items at EMC (end of marked content) instead of at the
first Tj/TJ inside the BDC region. This computes the width from the
text matrix delta across the entire BDC..EMC span, so ligature
replacements like "fi" get proper widths and merge correctly with
adjacent text items (e.g., "Defi nitions" → "Definitions").
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Handle text rendering mode (Tr=3) to skip invisible OCR overlay text
- Support BDC/EMC marked content with ActualText for tagged PDFs
- Merge adjacent single-char TextItems into words at extraction layer
- Skip image XObjects instead of emitting [Image: ...] placeholders
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
