When consecutive pages each have exactly one table with the same column
count, treat them as a single table spanning multiple pages. Strips the
redundant header+separator rows from continuation pages and appends
their data rows to the first page's table.
Closed-Business-Accounts PDF now produces one 2080-row table instead of
19 separate tables each with their own header.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raise BodyFont max_rows from 100 to 200 so tables with ~115 rows
(like the Closed Business Accounts PDF) pass validation. Add date
pattern recognition (MM/DD/YYYY, YYYY-MM-DD) to looks_like_table_data()
so date columns count toward the 30% data threshold.
Fix a bug where pages whose text is entirely consumed by tables
produced empty output — the post-loop cleanup only checked
current_page (stuck at 0), missing all table content. Now iterates
all pages with uninserted tables/images.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The symmetric single-char check (prev==1 OR curr==1) was too aggressive
for PDFs using per-glyph positioning (each letter as a separate item),
causing word spaces to disappear ("EffectiveDate" instead of
"Effective Date").
Now distinguishes three cases:
- Asymmetric (one single-char, other multi-char): generous 0.25 threshold
for fragment rejoining ("b"+"illion", "C"+"ultural")
- Both single-char numeric: generous 0.25 threshold for number continuity
("1"+"0"+"0" within per-glyph numbers)
- Both single-char alphabetic: normal 0.15 threshold to preserve word
spaces in per-glyph Latin text
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two targeted threshold adjustments in the accurate-width path of
should_join_items():
1. Numeric continuity (0.3x threshold): When adjacent items form a
number sequence (digits, commas, periods, percent signs), use a
generous threshold. Fixes splits like "34,20 8" → "34,208" and
"+13. 0 %" → "+13.0%".
2. Single-character fragments (0.25x threshold): Single-char items from
per-glyph positioning are almost never standalone words. Fixes splits
like "b illion", "C ultural", "togeth er".
The general 0.15x threshold is preserved for multi-character items to
avoid regressions in normal word spacing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three changes to improve heading quality:
1. Raise heading threshold from 1.1x to 1.2x base font size, reducing
false positives where slightly larger body text was promoted to headers.
2. Add word count guard (max 15 words) to skip heading detection for
long body paragraphs that happen to use a larger font.
3. Add merge_heading_lines() preprocessing that joins consecutive lines
at the same heading level on the same page (e.g., "About Glenair,
the Mission-Critical" + "Interconnect Company" → single heading).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The fallback estimated-width path in should_join_items() used Latin
case-based thresholds (uppercase/lowercase transitions) to decide word
boundaries. For CJK text emitted as per-glyph items, these heuristics
incorrectly inserted spaces within words (e.g., "ス テ ップ" instead of
"ステップ"). Now detects CJK characters and uses a generous join
threshold, matching the behavior already applied in the accurate-width path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Row merging occurred in uniform-spacing PDFs (e.g., 210603_ROOFING_BIDRESP)
because the fixed 10.0pt cluster threshold matched the exact line spacing,
causing adjacent rows to merge in pairs. Now derives threshold from 0.8×
median font size, fixes >= comparison, and raises SmallFont max_rows to 200.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Store column lines per-column instead of in a flat Vec, and use section-based
merge (spanning items define vertical zones) so each column is read top-to-bottom
before moving to the next column.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Preserve leading whitespace in text_with_formatting() to fix ~80% of
missing-space issues (e.g. "holdermeans" -> "holder means")
- Tighten word boundary gap threshold from 0.05 to 0.01 and add
word-count heuristic to distinguish CID word-level operators from
Type1 line-level operators, fixing spurious spaces (e.g. "t emporary")
- Replace fixed paragraph threshold (1.8x base_size) with dynamic
median-based computation for double-spaced documents
- Remove --- page break markers between pages
- Fix page number y-threshold for US Letter (720pt vs 800pt)
Eval: +1.4% mean word_sim, zero char_sim regressions across 63 PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CJK languages (Chinese, Japanese, Korean) don't use spaces between
words. The multi-char word-boundary detection was inserting unwanted
spaces in Japanese text (e.g., "である 履行義務" instead of
"である履行義務"). Skip the heuristic when either item boundary
involves a CJK character.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-character words from separate Tj operators (CID/Identity-H fonts)
were being joined without spaces when positioned back-to-back (gap≈0).
Detect multi-char items touching at near-zero gap and insert spaces,
while preserving single-char per-glyph joining for running headers.
Also strip null bytes and control characters in expand_ligatures() to
clean Latin-1 fallback artifacts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix Differences encoding path dropping standard characters when only
ligature bytes matched the sparse encoding map. Now combines Differences
entries with Latin-1 fallback for printable bytes instead of using
filter_map which silently dropped unmapped bytes.
Add expand_ligatures() to replace Unicode ligature characters (U+FB00-FB04)
with their ASCII components (ff, fi, fl, ffi, ffl) at all TextItem
creation sites. Add underscore-variant glyph names (f_f, f_f_i, f_f_l).
Eliminates 669 ligature corruptions across the eval suite.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add pairwise cross-row column alignment check to find_table_regions_strict().
Real tables have fixed column X positions that repeat across rows (score ~1.0),
while paragraph text has varying word positions (score ~0.3-0.4). Regions with
average pairwise alignment score < 0.5 are now rejected.
Also remove the num_cols >= 5 content-check bypass for BodyFont mode, requiring
body-font tables to always have ≥30% data-like cell content.
Fixes false table detection across 9+ PDFs including AI_Cultural_Collections
(-730 false lines), 131212888 (-740), Data-Processing-Agreement (-108),
CEP_DN_Interest_Rates (-93). No regressions on legitimate tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tables at body font size were invisible because detect_tables() gated on
font_size <= base * 0.90. Add a second pass with stricter structural
criteria (3+ columns, 70% alignment, 3+ rows with 3+ X-clusters) to
detect body-font tables without false-positiving on paragraphs. Also
raise the row limit from 30 to 50/100 to support large data tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous approach used a 30% page-width gap threshold on sorted X positions,
which never detected real column gutters (typically 2-5% of page width). This
caused text from adjacent columns to be interleaved on the same line (e.g.
"Brookfieldchief", "areclinging").
The new approach builds an occupancy histogram across the page width, finds empty
valleys (gutters), and validates them with vertical consistency checks. Also adds
spanning-item detection so full-width headers/titles are handled correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ItemType enum to distinguish Text, Image, and Link items
- Extract XObject images from page resources with position/dimensions
- Parse Link annotations to extract hyperlinks with URLs
- Add include_images and include_links options to MarkdownOptions
- Fix line grouping to better detect new lines vs same-line items
- Add samples/ and scripts/ to .gitignore
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. Word Fragment Joining (extractor.rs:63-94)
Added is_word_continuation() to detect when text items are fragments of the same word and should be joined without spaces:
2. Caption Detection (markdown.rs:621-658)
Added is_caption_line() to detect figures, tables, and source citations:
- Ensures captions are on their own line with paragraph breaks
3. Paragraph Threshold Adjustment
Changed from base_size * 2.0 to base_size * 1.8 for better paragraph detection.
Paragraphs now flow as continuous text, with only actual paragraph
breaks (large Y gaps) creating newlines between them.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Drop caps are large decorative first letters that span multiple lines.
Due to PDF coordinate sorting, they often appear after the text they
belong to. This change:
- Detects drop caps: single uppercase char with font size >= 2.5x base
- Finds the paragraph start (first lowercase-starting line after header)
- Merges the drop cap with that line
Example: "G" + "lenair brings..." -> "Glenair brings..."
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Major changes:
- Switch from lopdf.extract_text() to position-aware extraction
- Text is now sorted by visual reading order (top→bottom, left→right)
- Fixed font size calculation to account for text matrix scaling
- Headers are now detected based on font size ratios
- Skip very short text (≤3 chars) for header detection to avoid drop caps
This significantly improves output quality for PDFs with complex layouts.
Before: Title appeared at end, no structure detected
After: Correct reading order, headers marked with # syntax
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The previous code used `trimmed.len() >= 2` to check if there were at
least 2 characters, but `len()` returns byte count, not character count.
For multi-byte UTF-8 characters (e.g., "é" which is 2 bytes), this check
would pass but `chars().nth(1)` would return None, causing a panic.
Fixed by using iterator pattern matching to safely extract the first
two characters.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
