- Add pdf.js binary CMap (bcmap) parser for Japan1/GB1/CNS1 CID fonts
- Support inline ToUnicode streams (not just object references)
- Add CMapDecisionCache for heuristic primary vs remapped CMap selection
- Build fallback CMaps from embedded font data and CIDSystemInfo
- Handle simple fonts without ToUnicode via embedded font cmap
- Add Symbol/Wingdings/ZapfDingbats font decoding fallback
- Support usecmap chaining in both text and binary CMap formats
- Scan XObject Forms for text operators in detector
- Change default detection strategy from EarlyExit to Sample(8)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Some PDF generators subset-embed fonts by renumbering GIDs sequentially
but fail to update the ToUnicode CMap, which still references original
GID values. Detect this mismatch (Identity-H, no CIDToGIDMap, W array
starts at CID ≤ 2 but CMap min CID > 2) and remap to sequential positions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hand-rolled JSON escaping only handled \, ", and \n but missed
tabs, carriage returns, and other control characters (U+0000..U+001F),
producing invalid JSON that Python's json.loads would reject. Add a
proper json_escape() function covering all required escapes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces a ProcessMode enum that controls how far the PDF pipeline
runs, enabling fast document triage without paying extraction or
markdown conversion costs. Exposed via --detect-only and --analyze
flags in both pdf2md and detect-pdf CLIs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add LayoutComplexity struct to PdfProcessResult so callers can detect
when a PDF has complex layout (tables or multi-column text) and decide
whether to use the extracted markdown or fall back to OCR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a PDF page has a graph directly above a table (e.g. Figure 2 above
Table II in real-estate-pricing.pdf), the heuristic table detector would
merge graph axis labels and legend items into the table, producing a
corrupted result.
Add rect hint regions: when a small cluster of cell-border rects (4-6)
fails full grid validation (e.g. only row borders, no column dividers),
extract their Y bounding box as a "hint region". The heuristic detector
then runs separately on items inside vs outside hint regions, preventing
unrelated content from being merged into tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rects spanning multiple grid rows (e.g. classification labels) now have
their text consolidated into the first sub-row via propagate_merged_cells().
Also prevent continuation-row merge from folding body data into the header.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 5 stripped public-domain PDF fixtures and golden markdown snapshots
for CI regression testing. Fix non-deterministic output caused by
HashMap iteration order in font stats, table heuristics, and rect
clustering by adding deterministic tie-breaking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The snap_edges tolerance of 2.0 was too tight for PDFs where header and
body cell boundaries differ by ~3pt due to cell padding. This created
phantom empty columns that caused valid tables to be rejected. Increase
snap and cell coverage tolerances from 2.0/3.0 to 6.0. Also add debug
logging to rect-based table detection pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge CMap and Differences encoding at the byte level for single-byte
fonts, fixing garbled output when partial ToUnicode CMaps caused Latin-1
fallback to block the Differences path. Add Adobe-Korea1 CID-to-Unicode
predefined mapping for Identity-H CID fonts without ToUnicode streams,
and support TrueType cmap extraction via ttf-parser for embedded fonts.
Also handle suffixed glyph names (e.g. zero.tf, a.ss01) per Adobe spec.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates double-parsing of PDFs by using the lopdf document model exclusively
for ToUnicode CMap extraction. The old raw byte scanner parsed PDFs separately
from lopdf and only handled FlateDecode, while lopdf handles FlateDecode + LZW +
ASCII85. The new approach walks page fonts and Form XObject fonts via the document
API, yielding ~7x speedup on text-heavy PDFs and ~1.2x overall.
- Add FontCMaps::from_doc() with recursive Form XObject font walking
- Remove ~270 lines of raw byte scanning code (from_pdf_bytes, extract_stream_from_raw_pdf, etc.)
- Remove flate2 dependency (lopdf handles decompression internally)
- Remove dead CMap lookup fallback branches (by_name, get_with_obj, base_font_name)
- Remove unused font_base_names parameter from extract_text_from_operand
- Skip U+FFFD replacement characters in CMap decode (PDF notdef glyph markers)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detect newspaper-style two-column layouts (dense independent text
flows) and emit columns sequentially instead of Y-interleaving them.
This fixes academic papers where column text was being merged across
columns at the same Y position.
Key changes:
- Add is_newspaper_layout() heuristic (≥15 lines/col, >50% Y-collision)
- Add split_column_stragglers() to separate core column clusters from
header remnants and per-word items that landed in column buckets
- Filter wide items (>60% page width) from column detection histogram
so full-width text doesn't fill the gutter and prevent detection
- Emit: above spanning → core columns sequentially → column stragglers
sequentially → below spanning items
- Fix markdown paragraph break to detect backward Y jumps (y_gap.abs())
so sequential columns on the same page get proper paragraph breaks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detect RTL scripts (Hebrew, Arabic, Syriac, etc.) by Unicode ranges and
sort line items by X descending when a line is predominantly RTL. Fix
gap calculation in should_join_items() to be direction-neutral. Extract
WMode from Type0 font dictionaries for vertical text foundation. Add
missing Korean Hangul ranges to is_cjk_char().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like privacy notices have 6-7 distinct visual table sections per
page, each with its own set of cell rects. Previously all rects were
merged into a single giant sparse grid that failed validation. Now
spatially connected rects are clustered via union-find before running
grid detection independently on each cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add U+FB05/FB06 (ſt/st→st) to ligature table matching PyMuPDF's full set
- Refactor expand_ligatures to single-pass match loop (clippy fix)
- Detect dot-leader lines (TOC entries) and prevent list continuation
merging, which was joining sub-entries onto parent numbered items
- Force newline breaks between consecutive dot-leader lines in paragraphs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace section-based column merge (which emitted all left-column lines
then all right-column lines) with Y-interleaved merge that combines
lines at the same Y position from different columns into a single line.
This fixes tabular layouts (like court dockets) where rows span across
columns — previously hearing descriptions were detached from their
entries and line breaks between rows were lost.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fillable PDF forms store typed values in the AcroForm dictionary, not in
the page content stream. This adds extraction of form field names and
values, emitting them as TextItems positioned at the field's Rect so they
flow naturally into the markdown pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
effective_font_size() was computed from the text matrix alone, ignoring
CTM scaling. In PDFs with a small CTM scale (e.g. 0.24×) and large Tm
values (e.g. 58), font_size was inflated (58pt instead of ~14pt), causing
the merge threshold to bridge inter-column gaps and merge two-column
text into single lines. Now compute the combined matrix (text_matrix × CTM)
before calling effective_font_size() at all 6 call sites.
Also adds PdfRect extraction from `re` operators for future table-grid
detection, and related plumbing changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Emit ActualText items at EMC (end of marked content) instead of at the
first Tj/TJ inside the BDC region. This computes the width from the
text matrix delta across the entire BDC..EMC span, so ligature
replacements like "fi" get proper widths and merge correctly with
adjacent text items (e.g., "Defi nitions" → "Definitions").
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Handle text rendering mode (Tr=3) to skip invisible OCR overlay text
- Support BDC/EMC marked content with ActualText for tagged PDFs
- Merge adjacent single-char TextItems into words at extraction layer
- Skip image XObjects instead of emitting [Image: ...] placeholders
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like SEC filings render each glyph as a separate text operator with
intra-word gaps ≈ 0 and word gaps ≈ 0.15× font_size. The previous default
threshold (0.15) was exactly at the boundary, causing words to merge
(e.g. "ofOperations", "endedJune", "furnishedas").
Reduce single-char-to-single-char alphabetic threshold from 0.15 to 0.10,
cleanly separating word boundaries from intra-word kerning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --pages flag to insert <!-- Page N --> markers between pages
- Add --select-pages flag to process only specific pages (e.g. 1,3,5-10)
- Wire MarkdownOptions and page filter through process_pdf_with_config
- Fix fuzzy CMap matching that caused Cyrillic substitution on Latin text
- Fix whitespace Tj items not advancing text matrix (broke gap detection)
- Tighten single-char fragment join threshold from 0.25 to 0.20
- Gitignore debug/diagnostic binaries and remove tracked ones
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the blanket multi-column page guard (which blocked ALL table
detection on two-column pages) with content-based validation that
rejects paragraph text falsely detected as tables:
- Detect word-break hyphens at cell boundaries (paragraph signal)
- Reject high empty-cell ratio with many rows (sparse grid = text)
- Detect letter-spaced text (wide character spacing, not data)
- Reject long sentence fragments (avg cell >40 chars + many >60)
- Recover body-font header rows for small-font tables
- Add X-range filtering to body-font table region detection
olmOCR-bench: table_tests 16.4%→18.7% (+24), overall 31.6%→32.2%
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TJ arrays that span multiple table columns are now split into separate
TextItems at gaps >4x the space threshold, enabling the table detector
to assign text to correct columns. Also relaxes find_first_table_row()
to only skip rows where the majority of cells look form-like, preserving
real table headers that happen to contain colons.
olmOCR-bench table_tests: 10.7% → 16.4% (+53% relative, +58 tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Type3 fonts with FontMatrix [1,0,0,-1,0,0] (units_scale=1.0) had
broken word spacing: intra-word kerns of -4000 exceeded the clamped
threshold of 200, producing "Op en-Domain", "comp etition", etc.
Two fixes:
- Compute space_width fallback after units_scale is known; for
non-standard scales, estimate from average glyph width (~45%)
instead of the hardcoded 250 (calibrated for 1/1000-unit fonts)
- Remove upper clamp on space threshold (200 was too low for fonts
where TJ values naturally range into thousands)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace hardcoded max_pages_to_sample with a ScanStrategy enum that
supports EarlyExit (default), Full, Sample(n), and Pages(vec) modes.
Add process_pdf_with_config and process_pdf_mem_with_config to the
public API. Update README with usage examples and strategy docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change max_pages_to_sample from 5 to u32::MAX so every page is analyzed.
This prevents misclassifying Mixed PDFs as TextBased when scanned pages
fall outside the old 5-page sample window.
Add early-exit: stop scanning as soon as a non-text page is found, since
the PDF can't be purely TextBased. A 492-page mixed PDF exits after 2
pages instead of scanning all 492.
Also add title and confidence fields to PdfProcessResult for downstream
consumers (NAPI wrapper, feature-flag gating).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lopdf's get_font_encoding() re-parses ToUnicode CMap streams via nom for
every Tj/TJ operator. Cache the encoding once per font during page setup.
Also add binary search for CMap range lookup and de-dup CMap extraction.
Reduces Arabic PDF processing from 45s to <1s (53x speedup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Raise NOISE_FRACTION 0.05→0.15 so width bleed into gutter bins doesn't
prevent column detection
- Lower table bypass threshold 150→120pt to protect 3-column layouts
- Make has_template_image a sufficient condition for OCR routing
- Add unit tests for 2/3-column detection, width bleed tolerance, and
single-column regression guard
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
For mixed PDFs, callers can now see exactly which pages need OCR instead
of re-analyzing the document. Phase 2 scan iterates all pages for Mixed
PDFs (caching sampled results), while TextBased gets empty and
Scanned/ImageBased gets all pages.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-column text layouts (e.g., creditor matrices with 3 address columns)
were incorrectly claimed by body-font table detection. Now runs column
detection before table detection and skips tables on pages where all
detected columns are wider than 150pt, letting group_into_lines handle
the column-by-column reading order correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Td/TD operators were adding tx/ty offsets directly to the text line
matrix position without multiplying by the matrix scale factors. Per the
PDF spec, Td translates in text space: e_new = tx*a + ty*c + e. When the
text matrix contains font scaling (e.g., Tm [12,0,0,12,x,y] instead of
using Tf for size), all Td offsets were ~12x too small, causing items
from different visual lines to cluster at nearly identical Y coordinates
and items on the same line to have wrong X positions.
This fix:
- Scales Td/TD tx,ty by line_matrix[0..3] per PDF spec
- Adds TL operator tracking for proper text leading
- Fixes T* and ' operators to use TL with matrix scaling
- Fixes XObject form Td handler with same scaling
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Some PDF generators (e.g., 3B2 Total Publishing) use character code 0x41
('A') painted with white fill color (1 g) as invisible spacing. Our
extractor was emitting these as literal "A" characters, producing
pervasive "AA" artifacts throughout the output.
Track fill color state (g/rg/k operators) with save/restore via q/Q,
and skip text items when fill is white (>0.95 gray, or equivalent in
RGB/CMYK). Text matrix advancement is preserved so subsequent visible
text positions correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Validate files against %PDF- magic before parsing, returning a
machine-readable NotAPdf error with a hint about the actual file type
(HTML, XML, JSON, PNG, JPEG, ZIP, plain text). Improves From<lopdf::Error>
with structured matching for IO, encryption, and structural errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
