Some PDF generators (e.g., 3B2 Total Publishing) use character code 0x41
('A') painted with white fill color (1 g) as invisible spacing. Our
extractor was emitting these as literal "A" characters, producing
pervasive "AA" artifacts throughout the output.
Track fill color state (g/rg/k operators) with save/restore via q/Q,
and skip text items when fill is white (>0.95 gray, or equivalent in
RGB/CMYK). Text matrix advancement is preserved so subsequent visible
text positions correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Validate files against %PDF- magic before parsing, returning a
machine-readable NotAPdf error with a hint about the actual file type
(HTML, XML, JSON, PNG, JPEG, ZIP, plain text). Improves From<lopdf::Error>
with structured matching for IO, encryption, and structural errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dense financial tables (balance sheets, income statements) emit each data
row as 2 TextItems — a label and a single wide item containing all column
values. The table detector requires 3+ X-position clusters per row, so
these 2-item rows were missed. Pre-expand qualifying wide numeric items
into individual sub-items before detection runs, then map indices back.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The CID word-separation heuristic was applying to ALL fonts based solely on
gap size, causing spurious splits in non-CID fonts ("elevat ing") and missed
spaces in CID fonts ("commitmenttoimproving"). Add is_cid_font() helper to
detect CID fonts by resource name prefix (C2_*, C0_*) and only apply the
heuristic to actual CID fonts. Remove the fragile char-count guards that
were proxying for font awareness.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace center-point column assignment with maximum horizontal overlap
to prevent gutter mis-assignment. Add large-gap guards in should_join_items
to block cross-column word merging. Widen spacing threshold (0.15→0.18)
only for lowercase→lowercase junctions to fix mid-word space injection
in CID fonts while preserving word boundaries in all-caps headings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Image-only pages (e.g. scanned cover pages) were appended at the end of
output instead of appearing in page order. Now flushes tables/images for
intermediate pages with no text lines during page transitions.
Also collapses dot-leader runs (4+ dots) into " ... " for cleaner TOC
rendering, and adds debug_pages binary for inspecting per-page items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace O(pages*n) per-page item filtering with a single O(n) grouping
pass using a HashMap. Also replace O(n) global index lookups per table
item with O(1) pre-computed mappings.
Doc 9713 (832 pages): 49.2s → 5.6s (8.8x speedup)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When consecutive pages each have exactly one table with the same column
count, treat them as a single table spanning multiple pages. Strips the
redundant header+separator rows from continuation pages and appends
their data rows to the first page's table.
Closed-Business-Accounts PDF now produces one 2080-row table instead of
19 separate tables each with their own header.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raise BodyFont max_rows from 100 to 200 so tables with ~115 rows
(like the Closed Business Accounts PDF) pass validation. Add date
pattern recognition (MM/DD/YYYY, YYYY-MM-DD) to looks_like_table_data()
so date columns count toward the 30% data threshold.
Fix a bug where pages whose text is entirely consumed by tables
produced empty output — the post-loop cleanup only checked
current_page (stuck at 0), missing all table content. Now iterates
all pages with uninserted tables/images.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The symmetric single-char check (prev==1 OR curr==1) was too aggressive
for PDFs using per-glyph positioning (each letter as a separate item),
causing word spaces to disappear ("EffectiveDate" instead of
"Effective Date").
Now distinguishes three cases:
- Asymmetric (one single-char, other multi-char): generous 0.25 threshold
for fragment rejoining ("b"+"illion", "C"+"ultural")
- Both single-char numeric: generous 0.25 threshold for number continuity
("1"+"0"+"0" within per-glyph numbers)
- Both single-char alphabetic: normal 0.15 threshold to preserve word
spaces in per-glyph Latin text
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two targeted threshold adjustments in the accurate-width path of
should_join_items():
1. Numeric continuity (0.3x threshold): When adjacent items form a
number sequence (digits, commas, periods, percent signs), use a
generous threshold. Fixes splits like "34,20 8" → "34,208" and
"+13. 0 %" → "+13.0%".
2. Single-character fragments (0.25x threshold): Single-char items from
per-glyph positioning are almost never standalone words. Fixes splits
like "b illion", "C ultural", "togeth er".
The general 0.15x threshold is preserved for multi-character items to
avoid regressions in normal word spacing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three changes to improve heading quality:
1. Raise heading threshold from 1.1x to 1.2x base font size, reducing
false positives where slightly larger body text was promoted to headers.
2. Add word count guard (max 15 words) to skip heading detection for
long body paragraphs that happen to use a larger font.
3. Add merge_heading_lines() preprocessing that joins consecutive lines
at the same heading level on the same page (e.g., "About Glenair,
the Mission-Critical" + "Interconnect Company" → single heading).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The fallback estimated-width path in should_join_items() used Latin
case-based thresholds (uppercase/lowercase transitions) to decide word
boundaries. For CJK text emitted as per-glyph items, these heuristics
incorrectly inserted spaces within words (e.g., "ス テ ップ" instead of
"ステップ"). Now detects CJK characters and uses a generous join
threshold, matching the behavior already applied in the accurate-width path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Row merging occurred in uniform-spacing PDFs (e.g., 210603_ROOFING_BIDRESP)
because the fixed 10.0pt cluster threshold matched the exact line spacing,
causing adjacent rows to merge in pairs. Now derives threshold from 0.8×
median font size, fixes >= comparison, and raises SmallFont max_rows to 200.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Store column lines per-column instead of in a flat Vec, and use section-based
merge (spanning items define vertical zones) so each column is read top-to-bottom
before moving to the next column.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Preserve leading whitespace in text_with_formatting() to fix ~80% of
missing-space issues (e.g. "holdermeans" -> "holder means")
- Tighten word boundary gap threshold from 0.05 to 0.01 and add
word-count heuristic to distinguish CID word-level operators from
Type1 line-level operators, fixing spurious spaces (e.g. "t emporary")
- Replace fixed paragraph threshold (1.8x base_size) with dynamic
median-based computation for double-spaced documents
- Remove --- page break markers between pages
- Fix page number y-threshold for US Letter (720pt vs 800pt)
Eval: +1.4% mean word_sim, zero char_sim regressions across 63 PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CJK languages (Chinese, Japanese, Korean) don't use spaces between
words. The multi-char word-boundary detection was inserting unwanted
spaces in Japanese text (e.g., "である 履行義務" instead of
"である履行義務"). Skip the heuristic when either item boundary
involves a CJK character.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-character words from separate Tj operators (CID/Identity-H fonts)
were being joined without spaces when positioned back-to-back (gap≈0).
Detect multi-char items touching at near-zero gap and insert spaces,
while preserving single-char per-glyph joining for running headers.
Also strip null bytes and control characters in expand_ligatures() to
clean Latin-1 fallback artifacts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix Differences encoding path dropping standard characters when only
ligature bytes matched the sparse encoding map. Now combines Differences
entries with Latin-1 fallback for printable bytes instead of using
filter_map which silently dropped unmapped bytes.
Add expand_ligatures() to replace Unicode ligature characters (U+FB00-FB04)
with their ASCII components (ff, fi, fl, ffi, ffl) at all TextItem
creation sites. Add underscore-variant glyph names (f_f, f_f_i, f_f_l).
Eliminates 669 ligature corruptions across the eval suite.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add pairwise cross-row column alignment check to find_table_regions_strict().
Real tables have fixed column X positions that repeat across rows (score ~1.0),
while paragraph text has varying word positions (score ~0.3-0.4). Regions with
average pairwise alignment score < 0.5 are now rejected.
Also remove the num_cols >= 5 content-check bypass for BodyFont mode, requiring
body-font tables to always have ≥30% data-like cell content.
Fixes false table detection across 9+ PDFs including AI_Cultural_Collections
(-730 false lines), 131212888 (-740), Data-Processing-Agreement (-108),
CEP_DN_Interest_Rates (-93). No regressions on legitimate tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tables at body font size were invisible because detect_tables() gated on
font_size <= base * 0.90. Add a second pass with stricter structural
criteria (3+ columns, 70% alignment, 3+ rows with 3+ X-clusters) to
detect body-font tables without false-positiving on paragraphs. Also
raise the row limit from 30 to 50/100 to support large data tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous approach used a 30% page-width gap threshold on sorted X positions,
which never detected real column gutters (typically 2-5% of page width). This
caused text from adjacent columns to be interleaved on the same line (e.g.
"Brookfieldchief", "areclinging").
The new approach builds an occupancy histogram across the page width, finds empty
valleys (gutters), and validates them with vertical consistency checks. Also adds
spanning-item detection so full-width headers/titles are handled correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ItemType enum to distinguish Text, Image, and Link items
- Extract XObject images from page resources with position/dimensions
- Parse Link annotations to extract hyperlinks with URLs
- Add include_images and include_links options to MarkdownOptions
- Fix line grouping to better detect new lines vs same-line items
- Add samples/ and scripts/ to .gitignore
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. Word Fragment Joining (extractor.rs:63-94)
Added is_word_continuation() to detect when text items are fragments of the same word and should be joined without spaces:
2. Caption Detection (markdown.rs:621-658)
Added is_caption_line() to detect figures, tables, and source citations:
- Ensures captions are on their own line with paragraph breaks
3. Paragraph Threshold Adjustment
Changed from base_size * 2.0 to base_size * 1.8 for better paragraph detection.
Paragraphs now flow as continuous text, with only actual paragraph
breaks (large Y gaps) creating newlines between them.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
