Image-only pages (e.g. scanned cover pages) were appended at the end of
output instead of appearing in page order. Now flushes tables/images for
intermediate pages with no text lines during page transitions.
Also collapses dot-leader runs (4+ dots) into " ... " for cleaner TOC
rendering, and adds debug_pages binary for inspecting per-page items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace O(pages*n) per-page item filtering with a single O(n) grouping
pass using a HashMap. Also replace O(n) global index lookups per table
item with O(1) pre-computed mappings.
Doc 9713 (832 pages): 49.2s → 5.6s (8.8x speedup)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When consecutive pages each have exactly one table with the same column
count, treat them as a single table spanning multiple pages. Strips the
redundant header+separator rows from continuation pages and appends
their data rows to the first page's table.
Closed-Business-Accounts PDF now produces one 2080-row table instead of
19 separate tables each with their own header.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raise BodyFont max_rows from 100 to 200 so tables with ~115 rows
(like the Closed Business Accounts PDF) pass validation. Add date
pattern recognition (MM/DD/YYYY, YYYY-MM-DD) to looks_like_table_data()
so date columns count toward the 30% data threshold.
Fix a bug where pages whose text is entirely consumed by tables
produced empty output — the post-loop cleanup only checked
current_page (stuck at 0), missing all table content. Now iterates
all pages with uninserted tables/images.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three changes to improve heading quality:
1. Raise heading threshold from 1.1x to 1.2x base font size, reducing
false positives where slightly larger body text was promoted to headers.
2. Add word count guard (max 15 words) to skip heading detection for
long body paragraphs that happen to use a larger font.
3. Add merge_heading_lines() preprocessing that joins consecutive lines
at the same heading level on the same page (e.g., "About Glenair,
the Mission-Critical" + "Interconnect Company" → single heading).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Preserve leading whitespace in text_with_formatting() to fix ~80% of
missing-space issues (e.g. "holdermeans" -> "holder means")
- Tighten word boundary gap threshold from 0.05 to 0.01 and add
word-count heuristic to distinguish CID word-level operators from
Type1 line-level operators, fixing spurious spaces (e.g. "t emporary")
- Replace fixed paragraph threshold (1.8x base_size) with dynamic
median-based computation for double-spaced documents
- Remove --- page break markers between pages
- Fix page number y-threshold for US Letter (720pt vs 800pt)
Eval: +1.4% mean word_sim, zero char_sim regressions across 63 PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ItemType enum to distinguish Text, Image, and Link items
- Extract XObject images from page resources with position/dimensions
- Parse Link annotations to extract hyperlinks with URLs
- Add include_images and include_links options to MarkdownOptions
- Fix line grouping to better detect new lines vs same-line items
- Add samples/ and scripts/ to .gitignore
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. Word Fragment Joining (extractor.rs:63-94)
Added is_word_continuation() to detect when text items are fragments of the same word and should be joined without spaces:
2. Caption Detection (markdown.rs:621-658)
Added is_caption_line() to detect figures, tables, and source citations:
- Ensures captions are on their own line with paragraph breaks
3. Paragraph Threshold Adjustment
Changed from base_size * 2.0 to base_size * 1.8 for better paragraph detection.
Paragraphs now flow as continuous text, with only actual paragraph
breaks (large Y gaps) creating newlines between them.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Drop caps are large decorative first letters that span multiple lines.
Due to PDF coordinate sorting, they often appear after the text they
belong to. This change:
- Detects drop caps: single uppercase char with font size >= 2.5x base
- Finds the paragraph start (first lowercase-starting line after header)
- Merges the drop cap with that line
Example: "G" + "lenair brings..." -> "Glenair brings..."
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Major changes:
- Switch from lopdf.extract_text() to position-aware extraction
- Text is now sorted by visual reading order (top→bottom, left→right)
- Fixed font size calculation to account for text matrix scaling
- Headers are now detected based on font size ratios
- Skip very short text (≤3 chars) for header detection to avoid drop caps
This significantly improves output quality for PDFs with complex layouts.
Before: Title appeared at end, no structure detected
After: Correct reading order, headers marked with # syntax
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The previous code used `trimmed.len() >= 2` to check if there were at
least 2 characters, but `len()` returns byte count, not character count.
For multi-byte UTF-8 characters (e.g., "é" which is 2 bytes), this check
would pass but `chars().nth(1)` would return None, causing a panic.
Fixed by using iterator pattern matching to safely extract the first
two characters.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Smart PDF type detection (text vs scanned) without full document load
- Text extraction using lopdf directly
- Markdown conversion with header/list/code detection
- CLI tools: detect-pdf, pdf2md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
