Detect newspaper-style two-column layouts (dense independent text
flows) and emit columns sequentially instead of Y-interleaving them.
This fixes academic papers where column text was being merged across
columns at the same Y position.
Key changes:
- Add is_newspaper_layout() heuristic (≥15 lines/col, >50% Y-collision)
- Add split_column_stragglers() to separate core column clusters from
header remnants and per-word items that landed in column buckets
- Filter wide items (>60% page width) from column detection histogram
so full-width text doesn't fill the gutter and prevent detection
- Emit: above spanning → core columns sequentially → column stragglers
sequentially → below spanning items
- Fix markdown paragraph break to detect backward Y jumps (y_gap.abs())
so sequential columns on the same page get proper paragraph breaks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add U+FB05/FB06 (ſt/st→st) to ligature table matching PyMuPDF's full set
- Refactor expand_ligatures to single-pass match loop (clippy fix)
- Detect dot-leader lines (TOC entries) and prevent list continuation
merging, which was joining sub-entries onto parent numbered items
- Force newline breaks between consecutive dot-leader lines in paragraphs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fillable PDF forms store typed values in the AcroForm dictionary, not in
the page content stream. This adds extraction of form field names and
values, emitting them as TextItems positioned at the field's Rect so they
flow naturally into the markdown pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
effective_font_size() was computed from the text matrix alone, ignoring
CTM scaling. In PDFs with a small CTM scale (e.g. 0.24×) and large Tm
values (e.g. 58), font_size was inflated (58pt instead of ~14pt), causing
the merge threshold to bridge inter-column gaps and merge two-column
text into single lines. Now compute the combined matrix (text_matrix × CTM)
before calling effective_font_size() at all 6 call sites.
Also adds PdfRect extraction from `re` operators for future table-grid
detection, and related plumbing changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --pages flag to insert <!-- Page N --> markers between pages
- Add --select-pages flag to process only specific pages (e.g. 1,3,5-10)
- Wire MarkdownOptions and page filter through process_pdf_with_config
- Fix fuzzy CMap matching that caused Cyrillic substitution on Latin text
- Fix whitespace Tj items not advancing text matrix (broke gap detection)
- Tighten single-char fragment join threshold from 0.25 to 0.20
- Gitignore debug/diagnostic binaries and remove tracked ones
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the blanket multi-column page guard (which blocked ALL table
detection on two-column pages) with content-based validation that
rejects paragraph text falsely detected as tables:
- Detect word-break hyphens at cell boundaries (paragraph signal)
- Reject high empty-cell ratio with many rows (sparse grid = text)
- Detect letter-spaced text (wide character spacing, not data)
- Reject long sentence fragments (avg cell >40 chars + many >60)
- Recover body-font header rows for small-font tables
- Add X-range filtering to body-font table region detection
olmOCR-bench: table_tests 16.4%→18.7% (+24), overall 31.6%→32.2%
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Raise NOISE_FRACTION 0.05→0.15 so width bleed into gutter bins doesn't
prevent column detection
- Lower table bypass threshold 150→120pt to protect 3-column layouts
- Make has_template_image a sufficient condition for OCR routing
- Add unit tests for 2/3-column detection, width bleed tolerance, and
single-column regression guard
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-column text layouts (e.g., creditor matrices with 3 address columns)
were incorrectly claimed by body-font table detection. Now runs column
detection before table detection and skips tables on pages where all
detected columns are wider than 150pt, letting group_into_lines handle
the column-by-column reading order correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Image-only pages (e.g. scanned cover pages) were appended at the end of
output instead of appearing in page order. Now flushes tables/images for
intermediate pages with no text lines during page transitions.
Also collapses dot-leader runs (4+ dots) into " ... " for cleaner TOC
rendering, and adds debug_pages binary for inspecting per-page items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace O(pages*n) per-page item filtering with a single O(n) grouping
pass using a HashMap. Also replace O(n) global index lookups per table
item with O(1) pre-computed mappings.
Doc 9713 (832 pages): 49.2s → 5.6s (8.8x speedup)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When consecutive pages each have exactly one table with the same column
count, treat them as a single table spanning multiple pages. Strips the
redundant header+separator rows from continuation pages and appends
their data rows to the first page's table.
Closed-Business-Accounts PDF now produces one 2080-row table instead of
19 separate tables each with their own header.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raise BodyFont max_rows from 100 to 200 so tables with ~115 rows
(like the Closed Business Accounts PDF) pass validation. Add date
pattern recognition (MM/DD/YYYY, YYYY-MM-DD) to looks_like_table_data()
so date columns count toward the 30% data threshold.
Fix a bug where pages whose text is entirely consumed by tables
produced empty output — the post-loop cleanup only checked
current_page (stuck at 0), missing all table content. Now iterates
all pages with uninserted tables/images.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three changes to improve heading quality:
1. Raise heading threshold from 1.1x to 1.2x base font size, reducing
false positives where slightly larger body text was promoted to headers.
2. Add word count guard (max 15 words) to skip heading detection for
long body paragraphs that happen to use a larger font.
3. Add merge_heading_lines() preprocessing that joins consecutive lines
at the same heading level on the same page (e.g., "About Glenair,
the Mission-Critical" + "Interconnect Company" → single heading).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Preserve leading whitespace in text_with_formatting() to fix ~80% of
missing-space issues (e.g. "holdermeans" -> "holder means")
- Tighten word boundary gap threshold from 0.05 to 0.01 and add
word-count heuristic to distinguish CID word-level operators from
Type1 line-level operators, fixing spurious spaces (e.g. "t emporary")
- Replace fixed paragraph threshold (1.8x base_size) with dynamic
median-based computation for double-spaced documents
- Remove --- page break markers between pages
- Fix page number y-threshold for US Letter (720pt vs 800pt)
Eval: +1.4% mean word_sim, zero char_sim regressions across 63 PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ItemType enum to distinguish Text, Image, and Link items
- Extract XObject images from page resources with position/dimensions
- Parse Link annotations to extract hyperlinks with URLs
- Add include_images and include_links options to MarkdownOptions
- Fix line grouping to better detect new lines vs same-line items
- Add samples/ and scripts/ to .gitignore
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. Word Fragment Joining (extractor.rs:63-94)
Added is_word_continuation() to detect when text items are fragments of the same word and should be joined without spaces:
2. Caption Detection (markdown.rs:621-658)
Added is_caption_line() to detect figures, tables, and source citations:
- Ensures captions are on their own line with paragraph breaks
3. Paragraph Threshold Adjustment
Changed from base_size * 2.0 to base_size * 1.8 for better paragraph detection.
Paragraphs now flow as continuous text, with only actual paragraph
breaks (large Y gaps) creating newlines between them.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Drop caps are large decorative first letters that span multiple lines.
Due to PDF coordinate sorting, they often appear after the text they
belong to. This change:
- Detects drop caps: single uppercase char with font size >= 2.5x base
- Finds the paragraph start (first lowercase-starting line after header)
- Merges the drop cap with that line
Example: "G" + "lenair brings..." -> "Glenair brings..."
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Major changes:
- Switch from lopdf.extract_text() to position-aware extraction
- Text is now sorted by visual reading order (top→bottom, left→right)
- Fixed font size calculation to account for text matrix scaling
- Headers are now detected based on font size ratios
- Skip very short text (≤3 chars) for header detection to avoid drop caps
This significantly improves output quality for PDFs with complex layouts.
Before: Title appeared at end, no structure detected
After: Correct reading order, headers marked with # syntax
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The previous code used `trimmed.len() >= 2` to check if there were at
least 2 characters, but `len()` returns byte count, not character count.
For multi-byte UTF-8 characters (e.g., "é" which is 2 bytes), this check
would pass but `chars().nth(1)` would return None, causing a panic.
Fixed by using iterator pattern matching to safely extract the first
two characters.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Smart PDF type detection (text vs scanned) without full document load
- Text extraction using lopdf directly
- Markdown conversion with header/list/code detection
- CLI tools: detect-pdf, pdf2md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
