Fix Differences encoding path dropping standard characters when only
ligature bytes matched the sparse encoding map. Now combines Differences
entries with Latin-1 fallback for printable bytes instead of using
filter_map which silently dropped unmapped bytes.
Add expand_ligatures() to replace Unicode ligature characters (U+FB00-FB04)
with their ASCII components (ff, fi, fl, ffi, ffl) at all TextItem
creation sites. Add underscore-variant glyph names (f_f, f_f_i, f_f_l).
Eliminates 669 ligature corruptions across the eval suite.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add pairwise cross-row column alignment check to find_table_regions_strict().
Real tables have fixed column X positions that repeat across rows (score ~1.0),
while paragraph text has varying word positions (score ~0.3-0.4). Regions with
average pairwise alignment score < 0.5 are now rejected.
Also remove the num_cols >= 5 content-check bypass for BodyFont mode, requiring
body-font tables to always have ≥30% data-like cell content.
Fixes false table detection across 9+ PDFs including AI_Cultural_Collections
(-730 false lines), 131212888 (-740), Data-Processing-Agreement (-108),
CEP_DN_Interest_Rates (-93). No regressions on legitimate tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tables at body font size were invisible because detect_tables() gated on
font_size <= base * 0.90. Add a second pass with stricter structural
criteria (3+ columns, 70% alignment, 3+ rows with 3+ X-clusters) to
detect body-font tables without false-positiving on paragraphs. Also
raise the row limit from 30 to 50/100 to support large data tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous approach used a 30% page-width gap threshold on sorted X positions,
which never detected real column gutters (typically 2-5% of page width). This
caused text from adjacent columns to be interleaved on the same line (e.g.
"Brookfieldchief", "areclinging").
The new approach builds an occupancy histogram across the page width, finds empty
valleys (gutters), and validates them with vertical consistency checks. Also adds
spanning-item detection so full-width headers/titles are handled correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ItemType enum to distinguish Text, Image, and Link items
- Extract XObject images from page resources with position/dimensions
- Parse Link annotations to extract hyperlinks with URLs
- Add include_images and include_links options to MarkdownOptions
- Fix line grouping to better detect new lines vs same-line items
- Add samples/ and scripts/ to .gitignore
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. Word Fragment Joining (extractor.rs:63-94)
Added is_word_continuation() to detect when text items are fragments of the same word and should be joined without spaces:
2. Caption Detection (markdown.rs:621-658)
Added is_caption_line() to detect figures, tables, and source citations:
- Ensures captions are on their own line with paragraph breaks
3. Paragraph Threshold Adjustment
Changed from base_size * 2.0 to base_size * 1.8 for better paragraph detection.
Paragraphs now flow as continuous text, with only actual paragraph
breaks (large Y gaps) creating newlines between them.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Drop caps are large decorative first letters that span multiple lines.
Due to PDF coordinate sorting, they often appear after the text they
belong to. This change:
- Detects drop caps: single uppercase char with font size >= 2.5x base
- Finds the paragraph start (first lowercase-starting line after header)
- Merges the drop cap with that line
Example: "G" + "lenair brings..." -> "Glenair brings..."
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Major changes:
- Switch from lopdf.extract_text() to position-aware extraction
- Text is now sorted by visual reading order (top→bottom, left→right)
- Fixed font size calculation to account for text matrix scaling
- Headers are now detected based on font size ratios
- Skip very short text (≤3 chars) for header detection to avoid drop caps
This significantly improves output quality for PDFs with complex layouts.
Before: Title appeared at end, no structure detected
After: Correct reading order, headers marked with # syntax
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The previous code used `trimmed.len() >= 2` to check if there were at
least 2 characters, but `len()` returns byte count, not character count.
For multi-byte UTF-8 characters (e.g., "é" which is 2 bytes), this check
would pass but `chars().nth(1)` would return None, causing a panic.
Fixed by using iterator pattern matching to safely extract the first
two characters.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Smart PDF type detection (text vs scanned) without full document load
- Text extraction using lopdf directly
- Markdown conversion with header/list/code detection
- CLI tools: detect-pdf, pdf2md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
