The CID word-separation heuristic was applying to ALL fonts based solely on
gap size, causing spurious splits in non-CID fonts ("elevat ing") and missed
spaces in CID fonts ("commitmenttoimproving"). Add is_cid_font() helper to
detect CID fonts by resource name prefix (C2_*, C0_*) and only apply the
heuristic to actual CID fonts. Remove the fragile char-count guards that
were proxying for font awareness.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace center-point column assignment with maximum horizontal overlap
to prevent gutter mis-assignment. Add large-gap guards in should_join_items
to block cross-column word merging. Widen spacing threshold (0.15→0.18)
only for lowercase→lowercase junctions to fix mid-word space injection
in CID fonts while preserving word boundaries in all-caps headings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The symmetric single-char check (prev==1 OR curr==1) was too aggressive
for PDFs using per-glyph positioning (each letter as a separate item),
causing word spaces to disappear ("EffectiveDate" instead of
"Effective Date").
Now distinguishes three cases:
- Asymmetric (one single-char, other multi-char): generous 0.25 threshold
for fragment rejoining ("b"+"illion", "C"+"ultural")
- Both single-char numeric: generous 0.25 threshold for number continuity
("1"+"0"+"0" within per-glyph numbers)
- Both single-char alphabetic: normal 0.15 threshold to preserve word
spaces in per-glyph Latin text
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two targeted threshold adjustments in the accurate-width path of
should_join_items():
1. Numeric continuity (0.3x threshold): When adjacent items form a
number sequence (digits, commas, periods, percent signs), use a
generous threshold. Fixes splits like "34,20 8" → "34,208" and
"+13. 0 %" → "+13.0%".
2. Single-character fragments (0.25x threshold): Single-char items from
per-glyph positioning are almost never standalone words. Fixes splits
like "b illion", "C ultural", "togeth er".
The general 0.15x threshold is preserved for multi-character items to
avoid regressions in normal word spacing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The fallback estimated-width path in should_join_items() used Latin
case-based thresholds (uppercase/lowercase transitions) to decide word
boundaries. For CJK text emitted as per-glyph items, these heuristics
incorrectly inserted spaces within words (e.g., "ス テ ップ" instead of
"ステップ"). Now detects CJK characters and uses a generous join
threshold, matching the behavior already applied in the accurate-width path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Store column lines per-column instead of in a flat Vec, and use section-based
merge (spanning items define vertical zones) so each column is read top-to-bottom
before moving to the next column.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Preserve leading whitespace in text_with_formatting() to fix ~80% of
missing-space issues (e.g. "holdermeans" -> "holder means")
- Tighten word boundary gap threshold from 0.05 to 0.01 and add
word-count heuristic to distinguish CID word-level operators from
Type1 line-level operators, fixing spurious spaces (e.g. "t emporary")
- Replace fixed paragraph threshold (1.8x base_size) with dynamic
median-based computation for double-spaced documents
- Remove --- page break markers between pages
- Fix page number y-threshold for US Letter (720pt vs 800pt)
Eval: +1.4% mean word_sim, zero char_sim regressions across 63 PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CJK languages (Chinese, Japanese, Korean) don't use spaces between
words. The multi-char word-boundary detection was inserting unwanted
spaces in Japanese text (e.g., "である 履行義務" instead of
"である履行義務"). Skip the heuristic when either item boundary
involves a CJK character.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-character words from separate Tj operators (CID/Identity-H fonts)
were being joined without spaces when positioned back-to-back (gap≈0).
Detect multi-char items touching at near-zero gap and insert spaces,
while preserving single-char per-glyph joining for running headers.
Also strip null bytes and control characters in expand_ligatures() to
clean Latin-1 fallback artifacts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix Differences encoding path dropping standard characters when only
ligature bytes matched the sparse encoding map. Now combines Differences
entries with Latin-1 fallback for printable bytes instead of using
filter_map which silently dropped unmapped bytes.
Add expand_ligatures() to replace Unicode ligature characters (U+FB00-FB04)
with their ASCII components (ff, fi, fl, ffi, ffl) at all TextItem
creation sites. Add underscore-variant glyph names (f_f, f_f_i, f_f_l).
Eliminates 669 ligature corruptions across the eval suite.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous approach used a 30% page-width gap threshold on sorted X positions,
which never detected real column gutters (typically 2-5% of page width). This
caused text from adjacent columns to be interleaved on the same line (e.g.
"Brookfieldchief", "areclinging").
The new approach builds an occupancy histogram across the page width, finds empty
valleys (gutters), and validates them with vertical consistency checks. Also adds
spanning-item detection so full-width headers/titles are handled correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ItemType enum to distinguish Text, Image, and Link items
- Extract XObject images from page resources with position/dimensions
- Parse Link annotations to extract hyperlinks with URLs
- Add include_images and include_links options to MarkdownOptions
- Fix line grouping to better detect new lines vs same-line items
- Add samples/ and scripts/ to .gitignore
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. Word Fragment Joining (extractor.rs:63-94)
Added is_word_continuation() to detect when text items are fragments of the same word and should be joined without spaces:
2. Caption Detection (markdown.rs:621-658)
Added is_caption_line() to detect figures, tables, and source citations:
- Ensures captions are on their own line with paragraph breaks
3. Paragraph Threshold Adjustment
Changed from base_size * 2.0 to base_size * 1.8 for better paragraph detection.
Major changes:
- Switch from lopdf.extract_text() to position-aware extraction
- Text is now sorted by visual reading order (top→bottom, left→right)
- Fixed font size calculation to account for text matrix scaling
- Headers are now detected based on font size ratios
- Skip very short text (≤3 chars) for header detection to avoid drop caps
This significantly improves output quality for PDFs with complex layouts.
Before: Title appeared at end, no structure detected
After: Correct reading order, headers marked with # syntax
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Smart PDF type detection (text vs scanned) without full document load
- Text extraction using lopdf directly
- Markdown conversion with header/list/code detection
- CLI tools: detect-pdf, pdf2md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
