effective_font_size() was computed from the text matrix alone, ignoring
CTM scaling. In PDFs with a small CTM scale (e.g. 0.24×) and large Tm
values (e.g. 58), font_size was inflated (58pt instead of ~14pt), causing
the merge threshold to bridge inter-column gaps and merge two-column
text into single lines. Now compute the combined matrix (text_matrix × CTM)
before calling effective_font_size() at all 6 call sites.
Also adds PdfRect extraction from `re` operators for future table-grid
detection, and related plumbing changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Emit ActualText items at EMC (end of marked content) instead of at the
first Tj/TJ inside the BDC region. This computes the width from the
text matrix delta across the entire BDC..EMC span, so ligature
replacements like "fi" get proper widths and merge correctly with
adjacent text items (e.g., "Defi nitions" → "Definitions").
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Handle text rendering mode (Tr=3) to skip invisible OCR overlay text
- Support BDC/EMC marked content with ActualText for tagged PDFs
- Merge adjacent single-char TextItems into words at extraction layer
- Skip image XObjects instead of emitting [Image: ...] placeholders
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like SEC filings render each glyph as a separate text operator with
intra-word gaps ≈ 0 and word gaps ≈ 0.15× font_size. The previous default
threshold (0.15) was exactly at the boundary, causing words to merge
(e.g. "ofOperations", "endedJune", "furnishedas").
Reduce single-char-to-single-char alphabetic threshold from 0.15 to 0.10,
cleanly separating word boundaries from intra-word kerning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --pages flag to insert <!-- Page N --> markers between pages
- Add --select-pages flag to process only specific pages (e.g. 1,3,5-10)
- Wire MarkdownOptions and page filter through process_pdf_with_config
- Fix fuzzy CMap matching that caused Cyrillic substitution on Latin text
- Fix whitespace Tj items not advancing text matrix (broke gap detection)
- Tighten single-char fragment join threshold from 0.25 to 0.20
- Gitignore debug/diagnostic binaries and remove tracked ones
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the blanket multi-column page guard (which blocked ALL table
detection on two-column pages) with content-based validation that
rejects paragraph text falsely detected as tables:
- Detect word-break hyphens at cell boundaries (paragraph signal)
- Reject high empty-cell ratio with many rows (sparse grid = text)
- Detect letter-spaced text (wide character spacing, not data)
- Reject long sentence fragments (avg cell >40 chars + many >60)
- Recover body-font header rows for small-font tables
- Add X-range filtering to body-font table region detection
olmOCR-bench: table_tests 16.4%→18.7% (+24), overall 31.6%→32.2%
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TJ arrays that span multiple table columns are now split into separate
TextItems at gaps >4x the space threshold, enabling the table detector
to assign text to correct columns. Also relaxes find_first_table_row()
to only skip rows where the majority of cells look form-like, preserving
real table headers that happen to contain colons.
olmOCR-bench table_tests: 10.7% → 16.4% (+53% relative, +58 tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Type3 fonts with FontMatrix [1,0,0,-1,0,0] (units_scale=1.0) had
broken word spacing: intra-word kerns of -4000 exceeded the clamped
threshold of 200, producing "Op en-Domain", "comp etition", etc.
Two fixes:
- Compute space_width fallback after units_scale is known; for
non-standard scales, estimate from average glyph width (~45%)
instead of the hardcoded 250 (calibrated for 1/1000-unit fonts)
- Remove upper clamp on space threshold (200 was too low for fonts
where TJ values naturally range into thousands)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace hardcoded max_pages_to_sample with a ScanStrategy enum that
supports EarlyExit (default), Full, Sample(n), and Pages(vec) modes.
Add process_pdf_with_config and process_pdf_mem_with_config to the
public API. Update README with usage examples and strategy docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change max_pages_to_sample from 5 to u32::MAX so every page is analyzed.
This prevents misclassifying Mixed PDFs as TextBased when scanned pages
fall outside the old 5-page sample window.
Add early-exit: stop scanning as soon as a non-text page is found, since
the PDF can't be purely TextBased. A 492-page mixed PDF exits after 2
pages instead of scanning all 492.
Also add title and confidence fields to PdfProcessResult for downstream
consumers (NAPI wrapper, feature-flag gating).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lopdf's get_font_encoding() re-parses ToUnicode CMap streams via nom for
every Tj/TJ operator. Cache the encoding once per font during page setup.
Also add binary search for CMap range lookup and de-dup CMap extraction.
Reduces Arabic PDF processing from 45s to <1s (53x speedup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Raise NOISE_FRACTION 0.05→0.15 so width bleed into gutter bins doesn't
prevent column detection
- Lower table bypass threshold 150→120pt to protect 3-column layouts
- Make has_template_image a sufficient condition for OCR routing
- Add unit tests for 2/3-column detection, width bleed tolerance, and
single-column regression guard
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
For mixed PDFs, callers can now see exactly which pages need OCR instead
of re-analyzing the document. Phase 2 scan iterates all pages for Mixed
PDFs (caching sampled results), while TextBased gets empty and
Scanned/ImageBased gets all pages.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-column text layouts (e.g., creditor matrices with 3 address columns)
were incorrectly claimed by body-font table detection. Now runs column
detection before table detection and skips tables on pages where all
detected columns are wider than 150pt, letting group_into_lines handle
the column-by-column reading order correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Td/TD operators were adding tx/ty offsets directly to the text line
matrix position without multiplying by the matrix scale factors. Per the
PDF spec, Td translates in text space: e_new = tx*a + ty*c + e. When the
text matrix contains font scaling (e.g., Tm [12,0,0,12,x,y] instead of
using Tf for size), all Td offsets were ~12x too small, causing items
from different visual lines to cluster at nearly identical Y coordinates
and items on the same line to have wrong X positions.
This fix:
- Scales Td/TD tx,ty by line_matrix[0..3] per PDF spec
- Adds TL operator tracking for proper text leading
- Fixes T* and ' operators to use TL with matrix scaling
- Fixes XObject form Td handler with same scaling
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Some PDF generators (e.g., 3B2 Total Publishing) use character code 0x41
('A') painted with white fill color (1 g) as invisible spacing. Our
extractor was emitting these as literal "A" characters, producing
pervasive "AA" artifacts throughout the output.
Track fill color state (g/rg/k operators) with save/restore via q/Q,
and skip text items when fill is white (>0.95 gray, or equivalent in
RGB/CMYK). Text matrix advancement is preserved so subsequent visible
text positions correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Validate files against %PDF- magic before parsing, returning a
machine-readable NotAPdf error with a hint about the actual file type
(HTML, XML, JSON, PNG, JPEG, ZIP, plain text). Improves From<lopdf::Error>
with structured matching for IO, encryption, and structural errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dense financial tables (balance sheets, income statements) emit each data
row as 2 TextItems — a label and a single wide item containing all column
values. The table detector requires 3+ X-position clusters per row, so
these 2-item rows were missed. Pre-expand qualifying wide numeric items
into individual sub-items before detection runs, then map indices back.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The CID word-separation heuristic was applying to ALL fonts based solely on
gap size, causing spurious splits in non-CID fonts ("elevat ing") and missed
spaces in CID fonts ("commitmenttoimproving"). Add is_cid_font() helper to
detect CID fonts by resource name prefix (C2_*, C0_*) and only apply the
heuristic to actual CID fonts. Remove the fragile char-count guards that
were proxying for font awareness.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace center-point column assignment with maximum horizontal overlap
to prevent gutter mis-assignment. Add large-gap guards in should_join_items
to block cross-column word merging. Widen spacing threshold (0.15→0.18)
only for lowercase→lowercase junctions to fix mid-word space injection
in CID fonts while preserving word boundaries in all-caps headings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Image-only pages (e.g. scanned cover pages) were appended at the end of
output instead of appearing in page order. Now flushes tables/images for
intermediate pages with no text lines during page transitions.
Also collapses dot-leader runs (4+ dots) into " ... " for cleaner TOC
rendering, and adds debug_pages binary for inspecting per-page items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace O(pages*n) per-page item filtering with a single O(n) grouping
pass using a HashMap. Also replace O(n) global index lookups per table
item with O(1) pre-computed mappings.
Doc 9713 (832 pages): 49.2s → 5.6s (8.8x speedup)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When consecutive pages each have exactly one table with the same column
count, treat them as a single table spanning multiple pages. Strips the
redundant header+separator rows from continuation pages and appends
their data rows to the first page's table.
Closed-Business-Accounts PDF now produces one 2080-row table instead of
19 separate tables each with their own header.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raise BodyFont max_rows from 100 to 200 so tables with ~115 rows
(like the Closed Business Accounts PDF) pass validation. Add date
pattern recognition (MM/DD/YYYY, YYYY-MM-DD) to looks_like_table_data()
so date columns count toward the 30% data threshold.
Fix a bug where pages whose text is entirely consumed by tables
produced empty output — the post-loop cleanup only checked
current_page (stuck at 0), missing all table content. Now iterates
all pages with uninserted tables/images.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The symmetric single-char check (prev==1 OR curr==1) was too aggressive
for PDFs using per-glyph positioning (each letter as a separate item),
causing word spaces to disappear ("EffectiveDate" instead of
"Effective Date").
Now distinguishes three cases:
- Asymmetric (one single-char, other multi-char): generous 0.25 threshold
for fragment rejoining ("b"+"illion", "C"+"ultural")
- Both single-char numeric: generous 0.25 threshold for number continuity
("1"+"0"+"0" within per-glyph numbers)
- Both single-char alphabetic: normal 0.15 threshold to preserve word
spaces in per-glyph Latin text
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two targeted threshold adjustments in the accurate-width path of
should_join_items():
1. Numeric continuity (0.3x threshold): When adjacent items form a
number sequence (digits, commas, periods, percent signs), use a
generous threshold. Fixes splits like "34,20 8" → "34,208" and
"+13. 0 %" → "+13.0%".
2. Single-character fragments (0.25x threshold): Single-char items from
per-glyph positioning are almost never standalone words. Fixes splits
like "b illion", "C ultural", "togeth er".
The general 0.15x threshold is preserved for multi-character items to
avoid regressions in normal word spacing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three changes to improve heading quality:
1. Raise heading threshold from 1.1x to 1.2x base font size, reducing
false positives where slightly larger body text was promoted to headers.
2. Add word count guard (max 15 words) to skip heading detection for
long body paragraphs that happen to use a larger font.
3. Add merge_heading_lines() preprocessing that joins consecutive lines
at the same heading level on the same page (e.g., "About Glenair,
the Mission-Critical" + "Interconnect Company" → single heading).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The fallback estimated-width path in should_join_items() used Latin
case-based thresholds (uppercase/lowercase transitions) to decide word
boundaries. For CJK text emitted as per-glyph items, these heuristics
incorrectly inserted spaces within words (e.g., "ス テ ップ" instead of
"ステップ"). Now detects CJK characters and uses a generous join
threshold, matching the behavior already applied in the accurate-width path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Row merging occurred in uniform-spacing PDFs (e.g., 210603_ROOFING_BIDRESP)
because the fixed 10.0pt cluster threshold matched the exact line spacing,
causing adjacent rows to merge in pairs. Now derives threshold from 0.8×
median font size, fixes >= comparison, and raises SmallFont max_rows to 200.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Store column lines per-column instead of in a flat Vec, and use section-based
merge (spanning items define vertical zones) so each column is read top-to-bottom
before moving to the next column.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Preserve leading whitespace in text_with_formatting() to fix ~80% of
missing-space issues (e.g. "holdermeans" -> "holder means")
- Tighten word boundary gap threshold from 0.05 to 0.01 and add
word-count heuristic to distinguish CID word-level operators from
Type1 line-level operators, fixing spurious spaces (e.g. "t emporary")
- Replace fixed paragraph threshold (1.8x base_size) with dynamic
median-based computation for double-spaced documents
- Remove --- page break markers between pages
- Fix page number y-threshold for US Letter (720pt vs 800pt)
Eval: +1.4% mean word_sim, zero char_sim regressions across 63 PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CJK languages (Chinese, Japanese, Korean) don't use spaces between
words. The multi-char word-boundary detection was inserting unwanted
spaces in Japanese text (e.g., "である 履行義務" instead of
"である履行義務"). Skip the heuristic when either item boundary
involves a CJK character.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-character words from separate Tj operators (CID/Identity-H fonts)
were being joined without spaces when positioned back-to-back (gap≈0).
Detect multi-char items touching at near-zero gap and insert spaces,
while preserving single-char per-glyph joining for running headers.
Also strip null bytes and control characters in expand_ligatures() to
clean Latin-1 fallback artifacts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix Differences encoding path dropping standard characters when only
ligature bytes matched the sparse encoding map. Now combines Differences
entries with Latin-1 fallback for printable bytes instead of using
filter_map which silently dropped unmapped bytes.
Add expand_ligatures() to replace Unicode ligature characters (U+FB00-FB04)
with their ASCII components (ff, fi, fl, ffi, ffl) at all TextItem
creation sites. Add underscore-variant glyph names (f_f, f_f_i, f_f_l).
Eliminates 669 ligature corruptions across the eval suite.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add pairwise cross-row column alignment check to find_table_regions_strict().
Real tables have fixed column X positions that repeat across rows (score ~1.0),
while paragraph text has varying word positions (score ~0.3-0.4). Regions with
average pairwise alignment score < 0.5 are now rejected.
Also remove the num_cols >= 5 content-check bypass for BodyFont mode, requiring
body-font tables to always have ≥30% data-like cell content.
Fixes false table detection across 9+ PDFs including AI_Cultural_Collections
(-730 false lines), 131212888 (-740), Data-Processing-Agreement (-108),
CEP_DN_Interest_Rates (-93). No regressions on legitimate tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tables at body font size were invisible because detect_tables() gated on
font_size <= base * 0.90. Add a second pass with stricter structural
criteria (3+ columns, 70% alignment, 3+ rows with 3+ X-clusters) to
detect body-font tables without false-positiving on paragraphs. Also
raise the row limit from 30 to 50/100 to support large data tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous approach used a 30% page-width gap threshold on sorted X positions,
which never detected real column gutters (typically 2-5% of page width). This
caused text from adjacent columns to be interleaved on the same line (e.g.
"Brookfieldchief", "areclinging").
The new approach builds an occupancy histogram across the page width, finds empty
valleys (gutters), and validates them with vertical consistency checks. Also adds
spanning-item detection so full-width headers/titles are handled correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ItemType enum to distinguish Text, Image, and Link items
- Extract XObject images from page resources with position/dimensions
- Parse Link annotations to extract hyperlinks with URLs
- Add include_images and include_links options to MarkdownOptions
- Fix line grouping to better detect new lines vs same-line items
- Add samples/ and scripts/ to .gitignore
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
