* feat(tables): improve heuristic detection for borderless wrapped-cell tables
Three changes to the body-font heuristic detector:
1. Adaptive Y-gap in find_table_regions_strict: use median qualifying-row
spacing × 3 instead of fixed 25pt. Tables with wrapped cells have
larger gaps between qualifying rows (those with 3+ X-clusters).
2. Y-only region filtering: use full X range when collecting region items.
The strict X bounds from qualifying rows excluded continuation lines
in wrapped cells, starving find_column_boundaries of items.
3. Merged-band retry: when split_side_by_side splits a page into bands
but no band produces a table, retry heuristic detection on all items
merged as a single band.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): gap-histogram column detection for small tables + lower avg_cells
Two changes to fix PDF 045 (borderless table with narrow "No." column):
1. Extend gap-histogram column threshold to small tables: when the gap
between within-column jitter and between-column spacing is >10pt
(unambiguous bimodal signal), use the detected threshold even with
fewer than 500 items. Previously only triggered for dense tables.
2. Lower BodyFont avg_cells_per_row minimum from 2.5 to 2.0 to handle
tables with wrapped multi-line cells where continuation lines have
only 1 filled cell.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): trim empty outer columns + relax partial H-line validation
- Rect detection: trim empty first/last columns instead of rejecting
the whole table. Rect edges often extend beyond text boundaries.
- Line detection: accept tables with 6+ partial horizontal lines
(>15% width) when <3 full-spanning lines exist. Handles tables
with column-level separators instead of full-width rules.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): cell-rect fallback for tables with variable-width backgrounds
When rect clustering produces a grid that fails validation (empty
interior columns from variable-width cell backgrounds), fall through
to a new strategy: use rect Y-edges for row boundaries and text
X-position clustering for columns. This handles tables like the
opendataloader-bench 088-090 comparison tables where each cell has
its own background rect at different widths.
Also widen failed-cluster hint width cap for large clusters (≥30 rects)
to allow page-spanning table regions.
TEDS score on opendataloader-bench: 0.300 → 0.353.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): relax vertical line spanning validation for partial borders
Accept tables with 4+ partial vertical lines (>10% table height) when
fewer than 2 span >30%. Handles tables like opendataloader-bench 053
with column-level vertical separators that don't extend the full height.
TEDS: 0.353 → 0.377 on opendataloader-bench.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): lower cell-rect density threshold + add validation logging
- Lower cell-rect density minimum from 25% to 15% to accept sparser
tables with decorative backgrounds (fixes 147).
- Add debug logging to all heuristic validation paths for diagnosability.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): relax body-font validations for text-only and 2-column tables
Four fixes closing 71% of the TEDS gap vs opendataloader:
1. Validation 7 (table-like content): bypass numeric content requirement
for tables with 3+ columns that passed all structural checks. Text-only
tables (category lists, program descriptions) are legitimate.
2. Qualifying row threshold: lower from 3+ to 2+ X-clusters per row.
Enables 2-column body-font table detection (fixes 166).
3. Row-stripe max cell length: raise from 500 to 2000 for 3+ column
tables. Tables with paragraph descriptions in one column are valid
(fixes 121).
4. Row-stripe empty-column trimming: apply the same outer-column trim
as grid detection (fixes 121 column-0 rejection).
TEDS: 0.377 → 0.438 on opendataloader-bench (gap: -0.056 vs odl).
TEDS=0 docs: 14 → 9.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): enable 2-column body-font tables + lower all minimums
- Lower BodyFont minimum columns from 3 to 2 in detect_table_in_region
- Lower BodyFont minimum rows from 3 to 2
- Lower avg_cells_per_row minimum from 2.0 to 1.5 (handles wrapped cells
in 2-column tables)
- Apply empty-outer-column trimming to row-stripe detection (not just grid)
TEDS: 0.438 → 0.468 on opendataloader-bench (gap: -0.027 vs odl).
TEDS=0 docs: 9 → 8. 86% of original gap closed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): text-based row fallback + fix 120 flow-chart and 188 leaderboard
Three changes that push TEDS past opendataloader:
1. Cell-rect Y-edge fallback: when rects have too few Y-edges for row
structure, derive rows from text Y-position clustering within the
rect bounding box. Fixes flow-chart tables (120) and column-header-
only rects (188).
2. Lower cell-rect minimum from 20 to 6 rects to catch smaller tables.
3. Relax validation 1 (first-column presence) from 50% to 25% of rows.
Tables with wrapped model names have continuation lines without first
column content.
TEDS: 0.468 → 0.508 on opendataloader-bench.
Now BEATS opendataloader (0.508 vs 0.494, gap=+0.014).
TEDS=0 docs: 8 → 5.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): wrapped-cell continuation row merging
Merge rows that have fewer filled cells than the header row into the
previous row. Handles wrapped multi-line cells where text overflow
creates extra rows (e.g., "Direct" + "communications" → "Direct
communications").
Conditions: fewer filled cells than header, more than previous row had,
not a data row (numeric), not a short subheader label.
TEDS: 0.508 → 0.522 on opendataloader-bench (now +0.028 vs odl).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): tighten cell-rect validation + fix continuation-row merging
Cell-rect false positives:
- Raise density threshold back to 25% (from 15%)
- Add max cell length check (500 chars) to reject paragraph content
- Reject disproportionate grids (>20 rows, <4 cols)
Continuation-row merging:
- Wide tables (5+ cols): only merge rows with ≤50% header cells
- Narrow tables (2-4 cols): merge rows with fewer cells than header
- Prevents merging normal data rows in large tables (6_KE_Chart)
while keeping wrapped-cell merging for narrow tables (178)
TEDS: 0.498 on opendataloader-bench (still +0.004 vs odl).
pdf-evals: 191/192 passed, 0 regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Struct-tree tables with incomplete page tagging (e.g., only 22 of 50+
rows tagged on a page) would claim items and block rect detection,
leaving unclaimed items as loose text. Now require struct-tree tables
to capture ≥50% of band items before using them; incomplete trees
fall through to geometry-based detection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skip rect-detected tables that overlap with items already claimed by
struct-tree detection. Previously both strategies emitted separate
tables for the same content, doubling the output on tagged PDFs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a PDF has a well-formed structure tree with /Table > /TR > /TD|TH
elements linked to MCIDs, build tables directly from the semantic
hierarchy. Runs as highest-priority detection (step 0) before rect-based,
line-based, and heuristic strategies.
- Add StructTree::extract_tables() to walk the tree and collect table
descriptors with row/cell/MCID info
- Add detect_tables_from_struct_tree() to match MCIDs to TextItems
- Reject tables with <30% MCID cell coverage (stale structure trees)
- Update 2013-app2 snapshot (struct-tree gives valid but different
column ordering)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Origin-anchored full-page rects (x<5, y<5, h>20× median) are clipping
paths or page fills that bridge separate table regions into one cluster,
corrupting row-stripe detection. Exclude them from union-find adjacency
while keeping them available for hint generation and fallback paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Newsletter-style PDFs have decorative background rects (sidebar,
header, section bands) that pass row-stripe detection as false tables.
Reject when any cell exceeds 500 chars — real alternating-row data
tables have short cell content; layout backgrounds produce paragraph-
length "cells".
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(layout): relative valley column detection for justified text
Add fallback column detection using relative valley analysis for PDFs with
justified text where item widths extend past gutter boundaries. The absolute
valley detector fails on these layouts because gutter bins are at ~40% of
peak (well above the 15% noise threshold).
The relative valley detector smooths the histogram with a 5-bin moving
average, finds local minima where contrast < 0.60 of surrounding peaks,
and validates with peak balance >= 0.40. Limited to single best valley
(max 2 columns) and requires >= 100 items per page.
Tested on IRS Publication 17 (2002), a 289-page 2-column justified text
document: column detection went from ~40 pages to 165 pages.
190 passed, 0 regressions across 191 eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(layout): tighten relative valley thresholds to reduce false positives
Reduce PEAK_WINDOW from 40 to 25 bins (50pt) so valleys are only validated
against nearby peaks, not distant ones. Add MIN_PEAK_HEIGHT of 20 (smoothed)
to reject sparse pages where histogram peaks are too low to indicate dense
two-column text.
Previous thresholds caused 13 regressions across the eval suite by splitting
tables, TOCs, checklists, and forms. Now: 188 passed, 0 regressions (2 minor
metadata-only diffs on IRS P17 and 9978293).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(layout): skip relative valley detection on pages with tables
Table column gaps in the histogram look identical to text column gutters
but the table pipeline already handles reading order for those pages.
Pass page_has_table flag through detect_columns to suppress the relative
valley fallback on pages where tables were detected.
This eliminates all remaining regressions from relative valley detection:
190 passed, 0 regressions across 191 eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(layout): prose density validation for relative valley detection
Add columns_have_prose() to validate relative valley column splits.
Checks that both sides of a proposed split contain paragraph-like
content (fill ratio >= 40%, avg items/line <= 3.5) before committing
to a column split. Combined with the table-page guard, this prevents
false column splits on financial statements, forms, and tabular
layouts where long labels or dot leaders fill the column width.
Also tightens find_relative_valleys() thresholds (PEAK_WINDOW 40->25,
MIN_PEAK_HEIGHT 5->20) to reduce false positive valley candidates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Dense multi-column newspapers (WSJ, NYT) have extractable text but
produce poor output due to complex interleaved article layouts that
defeat column-based reading order. Detect these by counting Tf (font
change) operators alongside existing Tj/TJ counts, then flagging
TextBased PDFs where most sampled pages show high text density
(>=1500 ops), moderate font switches (>=50 Tf), and a low Tf/Tj
ratio (<0.15) — the ratio distinguishes newspapers from richly-styled
legal/business docs that have high Tf counts due to per-character
styling.
Calibrated against 108 TextBased PDFs in pdf-evals with zero false
positives. WSJ 50-page newspaper correctly flagged (5/8 sampled pages
match).
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The is_table_of_contents heuristic falsely rejected wide statistical
tables (e.g. ERP appendix tables) where the first column has year
labels with dot leaders ("1973..........") and other columns have
small numeric values. Add column-aware analysis: if dots are confined
to ≤1 column and ≥3 columns contain numbers, it's a data table.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(text): handle Tc/Tw character and word spacing in text width computation
PDFs using Tc (character spacing) and Tw (word spacing) operators for text
justification had words incorrectly split across TextItems. The computed
advance width didn't account for these spacing parameters, causing spurious
spaces mid-word (e.g. "deve lopers" instead of "developers").
- Add Tc/Tw operator handling and graphics state save/restore
- Incorporate char_spacing and word_spacing into compute_string_width_ts
- Add adaptive merge threshold: tighter for lowercase→lowercase junctions,
wider before joining punctuation
- Add unit tests for Tc/Tw width computation and merge behavior
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(fonts): add large Tc width computation test
Verifies that large character spacing values are applied in full
without any artificial cap, matching PDF spec behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(text): guard against Tc/Tw-inflated widths in merge and join paths
Two targeted fixes to prevent character-spacing (Tc) and word-spacing
(Tw) inflation from causing data quality regressions:
1. should_join_items: reject large negative gaps (< -font_size) that
arise when Tc/Tw inflate item widths past adjacent items. Fixes
FY_2015 merged numbers (e.g. "239.696.0" → "239.69 6.0").
2. merge_text_items: cap effective width for gap computation when Tw
inflates space-containing items beyond 0.85× font_size per char.
Prevents column-level gaps from collapsing into merge range,
recovering table detection for Baldwin-Edwards and similar PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
PDFs that embed landscape content in portrait pages via a rotated text
matrix (e.g. [0, b, -b, 0, tx, ty] for 90° CCW) produced garbled output
because the layout engine assumed x=horizontal, y=vertical.
Track the dominant text direction from combined matrices during extraction.
When ≥67% of text operators are rotated, swap x↔y coordinates (with
y-negation for correct reading order) for all text items, rects, and lines.
Also estimate text widths from char count × font size since scale_x ≈ 0
for rotated text.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When Identity-H or Type3 fonts lack a ToUnicode CMap and the
CID-as-Unicode passthrough doesn't produce valid text, the raw CID
byte values appear as mojibake (random Latin Extended characters mixed
with C1 control codes).
The detector already flags these pages in pages_needing_ocr, but the
markdown pipeline still emitted the garbage. Now, for TextBased PDFs,
we check each OCR-flagged page's extracted text for CID garbage
(C1 control characters U+0080–U+009F at ≥5% density) and strip items
from pages that fail the check.
This is scoped to TextBased PDFs only — Mixed PDFs flag pages for OCR
due to template images, not font encoding issues.
Adds test fixture (shinagawa_identity_h.pdf) and integration test.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(fonts): CID-as-Unicode passthrough and subscript/superscript merging
Add smart CID-as-Unicode passthrough for Identity-H fonts without
ToUnicode maps. Uses /W array median CID heuristic to distinguish
Unicode-CID PDFs (Chromium-generated) from GID-based subsets.
Add merge_subscript_items() pass that merges small-font items (<75%
of dominant font size, ≤4 chars, tightly adjacent) into parent items.
Fixes chemical formulas (NH3, H2O, KClO3), footnote references, and
subscript notation (vf, Hfg, m3/kg) that were previously orphaned
as separate text items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(subscripts): restrict merge to purely numeric text only
Tighten subscript merging to only merge items containing ASCII digits
(0-9). This avoids false positives with ordinal indicators (º), letter
subscripts (sol, vf), and small bullet characters (▶) that caused
table restructuring regressions.
Numeric-only keeps the primary wins: chemical formulas (NH3, H2O),
footnote references, and unit notation (m2, m3).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(subscripts): restrict merge to parent text ending with a letter
Only merge numeric subscripts when the parent item's text ends with
an alphabetic character. Prevents false merges like "33" + "1" in
fractions (33 1/3%), table credit numbers after spaces, and footnote
refs after punctuation (land.1 → land. 1). Chemical formulas (NH3,
H2O, KClO3) still merge correctly since parent ends with a letter.
Reduces pdf-eval regressions from 13 to 2 (both are correct reversions
of over-aggressive footnote merging from the prior commit).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Points to firecrawl/lopdf@edf6279 which fixes decompressed_content()
for streams without a /Filter entry. Our xobjects.rs workaround remains
as a defensive fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lopdf's decompressed_content() fails on Form XObjects without a /Filter
entry (uncompressed streams). This caused all body text to be lost in PDFs
generated by pdfrw and similar tools that wrap page content in uncompressed
Form XObjects (e.g. Cambridge University Press excerpts).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lopdf's decompressed_content() fails with DictKey("Filter") when a
ToUnicode stream has no /Filter entry (uncompressed raw text). Now
falls back to the raw stream.content when decompression fails.
Fixes Identity-H fonts with valid uncompressed ToUnicode CMaps
producing empty text (e.g. neoenergia tabela PDF).
Type3 fonts render each glyph as a custom drawing/bitmap. Without a
ToUnicode CMap the character codes can't be mapped to Unicode, so
extracted text is garbage. Pages using only Type3 fonts are now
excluded from pages_with_text and added to pages_needing_ocr.
Fixes Korean/CJK PDFs using Type3 fonts (e.g. D2Coding) being
classified as TextBased when no usable text can be extracted.
Document titles, column headers, and newsletter mastheads that repeat
on every page were being fully stripped by strip_repeated_lines. Now
the first page's occurrence is preserved so the content appears once.
Fixes missing titles like "VOICE OF SOUTH MARION" in tax certificate
PDFs and column headers in IRS forms.
When every page uses fonts with unresolvable gid-encoded glyphs,
the extracted text is unreliable garbage. Suppress the markdown
output so consumers know to use OCR instead.
Only triggers for gid-encoded fonts specifically, not for other
OCR signals (Identity-H without ToUnicode, template images) where
the text layer may still be partially useful.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(detect): flag Identity-H fonts without ToUnicode for OCR
Cyrillic (and other non-Latin) PDFs with Type0/Identity-H encoded fonts
and no ToUnicode CMap produce garbage text from direct extraction. Two
fixes:
1. Detector: new `page_has_identity_h_no_tounicode` check adds affected
pages to `pages_needing_ocr` regardless of PDF classification.
2. Extraction: extend garbage-text safety net to TextBased PDFs — when
extracted text is <50% alphanumeric, drop the markdown, set
`has_encoding_issues`, and flag all pages for OCR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: replace integration tests with synthetic unit tests
Remove PDF fixture dependencies from detector and lib tests. Use
in-memory lopdf documents to test Identity-H/ToUnicode detection logic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When rect-based, line-based, and heuristic detection all fail to find
tables on a page, try building a table directly from the layout
engine's column boundaries. Handles borderless tabular layouts like
exam/reference grids where columns are defined purely by text
alignment.
Includes header-row column refinement: when a detected column
contains multiple header items, it gets split at the gap between
them to recover the correct number of columns.
Guards against false positives: requires ≥4 columns, ≤40 rows,
>50% multi-column rows, short cells (avg ≤40 chars), no prose
content, no dominant single column, and no structural elements
(≥6 rects or ≥4 lines) on the page.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fonts using raw glyph ID names (gidNNNNN) in their Differences
encoding cannot be decoded to Unicode without the original font's
cmap table. Detect this pattern during font parsing and add
affected pages to pages_needing_ocr so downstream consumers
know to use OCR instead.
Fixes text extraction on PDFs like Tezukuri_Food-Menu.pdf where
the main body font (AcuminVariableConcept) uses gid-encoded
glyphs — even PyMuPDF and ODL fail on these.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two changes to improve balance sheet / financial table detection:
1. split_side_by_side: Don't split when one side is text labels
and the other is numeric data at matching Y positions. This
prevents splitting a single label+number table into two
independent regions.
2. try_add_label_column: After detecting a numeric-only table,
look for unclaimed text items to the left at matching Y
positions and prepend them as column 0 (row labels).
Tested on IN_Annual_Report_2017 balance sheet which now produces
proper 3-column tables (Label|2016|2017) instead of separated
number tables and paragraph text.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When rect clusters have valid bounding boxes but insufficient grid
structure (e.g. 2x2 edges from outer borders), emit their bounding
box as a RectHintRegion so the heuristic detector can be scoped to
the table area. Requires reasonable dimensions (100-600pt height,
≤500pt width) and ≥6 text items inside the region.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Screenshot PDFs (e.g., Chrome "Save as PDF") embed images inside
tiling Pattern resources rather than as direct XObject images.
Now traverses Pattern resources during image detection, correctly
classifying these as Mixed with OCR recommended.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The third-pass change to extract embedded cmaps for TrueType fonts with
WinAnsiEncoding caused apostrophes and other characters to be dropped
in subsetted fonts where the cmap doesn't cover all glyphs. The declared
encoding is authoritative for these fonts. The OCR text extraction fix
(invisible Tr=3 text) doesn't need this cmap override.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OCR text layers and some PDF producers emit trailing spaces on each
text item, which combine with gap-based joining to produce double
spaces ("Vice President"). Now collapses runs of 2+ spaces to single
space within lines, preserving leading indentation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scanned PDFs with OCR text layers use rendering mode 3 (invisible text)
positioned behind page images. Previously we skipped all Tr=3 text.
Now for Mixed/template PDFs, if normal extraction produces garbage or
empty output, we retry with invisible text included. This unlocks text
from OCR-generated PDFs without requiring external OCR.
Also adds Windows Unicode BMP (3,1) subtable support to the TrueType
cmap fallback, and allows TrueType fonts with explicit encoding to
extract their embedded cmap (OCR fonts often lie about encoding).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scanned PDFs with JBIG2/tiled image strips were misclassified as
TextBased because no individual image tile exceeded the template
threshold. Now checks aggregate image area per page (≥2M pixels).
Also adds is_garbage_text() check: if a Mixed/template PDF's extracted
text is predominantly non-alphanumeric (<50%), upgrade to Scanned so
callers use proper OCR instead of the garbage text layer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Multi-item lines (titles, section headers) that span across detected
columns were being split into individual column buckets, corrupting
newspaper detection and reading order. Add identify_spanning_lines()
that groups items by Y-proximity and marks lines wider than 1.3x the
widest column that have no gap at a detected gutter boundary.
Uses column-aware thresholds and gutter-interval gap detection for
precision — only fires on pages with detected columns, preserves
original item order via boolean mask.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a heading wraps across multiple visual lines, the structure tree
creates separate heading nodes per line, producing broken output like
`## Snow` / `## Lake`. Extend merge_heading_lines to consider struct-tree
roles (H1-H6) in addition to font-size heuristics. A 20-word combined
limit prevents merging body text mis-tagged as headings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: tagged PDF structure tree support for semantic markdown generation
Parse /StructTreeRoot from tagged PDFs and use semantic roles (H1-H6, P,
LI, BlockQuote, Code, Caption) to improve markdown output. Structure tree
headings add to font-size heuristics without suppressing them. Coverage
threshold (≥50%) ensures only properly tagged PDFs activate this path.
Phase 1: Parse structure tree with role maps, MCID collection, flattening
Phase 2: Capture MCIDs from BMC/BDC operators, tag TextItems
Phase 3: Structure-aware markdown generation in convert loop
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: accumulate consecutive code lines into single fenced block
Per-line code fencing produced broken markdown for multi-line code
blocks (separate ``` open/close per line). Unify struct-tree Code
role and font-based monospace detection into a single is_code_line
check with in_code_block state for proper accumulation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add tagged PDF fixture with Firecrawl docs content
Synthetic 7-page PDF with rich structure tree exercising H1, H2, H3,
P, Code, LI, Caption, TH, TD roles. Generated via fpdf2 script.
Integration test verifies struct tree parsing and code fence output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: remove python PDF generator script from repo
Keep the generated fixture PDF but don't track the generator script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle malformed bare-name struct types in tagged PDFs
Some PDF generators (e.g. fpdf2) write /S Code instead of /S /Code
in structure elements. lopdf silently drops these objects since bare
tokens are invalid PDF syntax. Add a pre-processor that scans for
known bare struct type names and prepends / before loading.
Unifies path and memory loading through the same fix pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: update lopdf dependency to main branch
The firecrawl/zlib-checksum-encrypted branch was merged and deleted.
Point to main which includes all previously merged fixes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: switch lopdf to upstream repo pinned at 845cd3d
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Narrow annotation columns beside wide body columns were being
Y-interleaved with body text, producing garbled reading order.
Add sidebar detection with width ratio, line balance, and sparse
density guards to correctly trigger sequential column reading.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use character-width ratios instead of font_size for Canva-style PDFs:
- Single-char prev: gap/prev.width < 1.25
- Multi→single: gap/avg_prev_char_width < 1.25
- Multi→multi: page-level threshold (gap/font_size)
Also adds second Canva detection path for per-character rendering
without embedded spaces (>50% single-char items).
Improves ebgt7isj04ophcq word accuracy from ~3% to 64%.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Canva-generated PDFs render text character-by-character with CSS-style
letter-spacing (~0.5-0.9× font_size). The hardcoded 0.10 threshold
caused every character to get a space inserted ("K a r i b i b").
Detect Canva pages via fix_letterspaced_items (≥50% items match "a b c"
pattern), compute an IQR-based threshold (median × 1.55) on the gap
distribution BEFORE space removal, then propagate per-page thresholds
through PageThresholds → group_into_lines_with_thresholds → TextLine
→ should_join_items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Drop column-width padding from table output. Saves ~37% bytes on
table-heavy PDFs. Markdown renders identically — padding was purely
cosmetic. Optimizes token efficiency for AI agent consumers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Arabic PDFs using ToUnicode maps to presentation forms (U+FB50-FDFF,
U+FE70-FEFE) produced unreadable output. Apply NFKC normalization to
convert back to base Arabic (U+0600-06FF), then reverse character order
within items that had presentation forms (visual→logical order).
Improves Arabic PDF word overlap with Mistral from ~4.5% to ~78.6%.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add bimodal gap analysis to find_column_boundaries for PDFs with many
narrow columns (e.g. 24-column train schedules at 26pt spacing). When
a clear gap between within-column jitter and between-column spacing is
detected, switch to edge-based clustering with a lower threshold.
Also raise heuristic max columns from 15 to 25, and relax column
consistency validation for very wide tables (>15 cols).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tables with very wide cells (e.g. 230+ chars from merged schedule columns)
caused massive whitespace padding in every row. Capping alignment at 40
characters reduces output size significantly (franklin: 77KB→24KB,
Closed-Business: 571KB→502KB) without affecting content.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Subset fonts number GIDs by document encounter order, not sorted order.
The sequential remap scrambled character mappings, losing 2,532+ CJK
characters. Now promotes TrueType cmap fallback when it has more entries
than the primary ToUnicode CMap. Also adds CJK scoring to score_text()
as defense-in-depth.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Points to firecrawl/lopdf branch that falls back to raw deflate when
zlib adler32 checksum is corrupted after RC4 decryption. Fixes 5
encrypted BrokerCheck PDFs that were misclassified as scanned.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rect-guided tables:
- Add try_build_rect_guided_table() to build tables from rect cluster
X positions as column boundaries, bypassing heuristic detection
- Split merged multi-number TextItems (e.g. "10 11 12...31") into
individual day-column cells using column-advancing boundary assignment
- Interpolate missing column boundaries for holiday/non-work days
that lack colored rects
- Strip tilde-leader noise from cells (legend text bleeding)
- Filter legend text beyond table area via max-X threshold
- Pass cluster_rects through RectHintRegion for downstream use
Extractor improvements:
- Deduplicate clip-path and fill-path rects before using as table hints
- Add font width fallback for missing glyph metrics via average width
- Improve column detection scoring to prefer balanced gutters
- Handle side-by-side layouts with hint-region derived split points
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 48 new tests (140 → 188 total) covering previously untested private
functions with synthetic data. No PDF fixtures needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Images nested inside Form XObjects (e.g. HuttoISDWorkPerks.pdf) were
invisible to the detector. Rewrote analyze_page_images to recursively
follow Form XObjects. Also added pages_with_vector_text tracking so
PDFs with vector-outlined text (e.g. qsop_21-0055.pdf) are correctly
classified as IMAGE-BASED with OCR recommended.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The retry mechanism (excluding page-background rects) was firing for any
grid rejection, including "column 0 is completely empty". This caused a
regression on HTM_02-01_Part_A where the retry accepted a bad 12x9 grid
instead of falling through to heuristic detection.
Introduce GridResult enum so the caller only retries on FewNonEmptyRows
(the specific symptom of propagate_merged_cells collapse), not on
structural failures like empty columns or low fill ratio.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs with full-page background fills (e.g. 6_KE_Chart_5-5-17.pdf) had
their table grids rejected because propagate_merged_cells collapsed all
text into row 0. Add a retry mechanism: normal detection first, then
if it fails and page-origin background rects are present (>=95% width,
>=90% height, near origin), retry with those rects excluded from X-edge
extraction and propagate_merged_cells.
The retry path uses strict validation (50% non-empty rows, 40% content
density, max 200 chars per cell) to avoid false positives on pages where
paragraph text would be incorrectly captured in the grid.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pages using nested Form XObject chains (page → wrapper Form → text Form)
were returning 0 text items because extract_form_xobject_text() didn't
handle the Do operator. Add Do/q/Q/cm handling with recursive descent
(max depth 5) and proper CTM tracking through the nesting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pages with images AND very few text operators (<10) are image pages
with overlay text (headers/footers), not real text pages. This fixes
image-heavy PDFs like newspaper ads being misclassified as TextBased.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The firecrawl/fix-cmap-optional-slash branch caused a regression on
the Italian PDF (garbled CMap-encoded text). The EPEVER fix was a
wrong diagnosis — the PDF extracts correctly on upstream main.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Count path construction/painting operators (m, l, c, h, f, S, re, etc.)
during the existing content stream scan. Pages where path ops vastly
outnumber text ops (>=1000 path ops AND >200x text ops) contain
vector-outlined glyphs that need OCR instead of text extraction.
This fixes EPEVER-DataSheet-XTRA-N-G3-Series-3.pdf page 2 which has
41,263 path ops but only 33 Tj ops — now correctly classified as Mixed
with page 2 flagged for OCR.
Also removes the check_page_decode_health phase (Phase 1.5) which
required parsing content streams into operations — the path-based
detection is both simpler and more effective.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switches to a new lopdf branch that includes the CMap parser fix for
PDFs with uncompressed ToUnicode streams that omit the leading slash
on metadata keys (CIDSystemInfo, CMapName, CMapType).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like fare1011.pdf draw table row stripes and borders using
m/l/h/f* filled paths instead of `re` operators. Extract axis-aligned
rectangles from these filled subpaths as a third-priority rect source
(after `re` rects and clip-path rects).
Key changes:
- Track completed subpaths in `h` handler for f/f* rect extraction
- Update W/W* handler to read from pending_subpaths when pending_lines
is empty (after h clears them)
- Add row-stripe fallback in detect_rects when clustering produces no
large clusters (non-overlapping row stripes), with ≥15 rects and
≥10 rows required to prevent decorative fill false positives
- Revert body-font upper bound to 1.05x (no longer needed at 1.10x)
fare1011.pdf: 12-col heuristic table → 14-col rect-based table with
all fare values correctly captured.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract axis-aligned rectangles from W/W* clip operators in content
streams — many PDFs define table cells as clipping paths instead of
stroked rects. Add merged-cluster fallback in detect_tables_from_rects()
that merges all cluster rects when per-cluster detection fails or only
produces narrow false-positives (≤3 columns). Uses rect Y-edges for
rows and text X-clustering for columns.
Also updates lopdf to firecrawl fork (fix-leading-whitespace branch).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Thread a shared HashSet<u8> through scan_content_for_text_operators and
scan_xobjects_in_resources instead of returning a count and re-scanning
with a separate collect_unique_chars_from_content function.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like wildfire_waiver.pdf render each line as a separate image XObject
with only bullet characters as actual text operators. Add image-dominance
heuristic (image_count > 10 && image_count > text_ops * 3) and unique text
character check (>= 5 unique non-whitespace chars) so these pages are no
longer misclassified as text-based.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only skip text with rendering mode 3 (truly invisible). Removes
fill color tracking that was incorrectly hiding white-on-dark text
in presentations and brochures.
- Lower gap threshold from 40pt to 30pt to catch narrower inter-table gaps
- Constrain candidates to middle 60% of page X range
- Use center-position balance check instead of left-edge counting
- Add balanced-cluster dedup (50pt) to distinguish side-by-side tables
from single wide tables with multiple column gaps
- Improve filter_rects_to_band with 70% containment check to exclude
page-wide background stripes
Split pages with two independent table regions (e.g. D'Addario tension
chart) into separate bands before running rect/line/heuristic detection.
The split function finds X-position gaps ≥40pt, validates that <2% of
items cross the split point, and requires ≥30% balance on each side.
Also adds heuristic table detection fallback in compute_layout_complexity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs that set fill color via `sc`/`scn` (named color space operators)
instead of `g`/`rg`/`k` were incorrectly keeping fill_is_white=true
from earlier page background fills, causing all text to be skipped.
Infer color model from operand count: 3=RGB, 4=CMYK, other=non-white.
Fixes extraction of tension_chart PDF (D'Addario string specifications)
which previously output only glyph names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add width-based outlier filter to remove page-spanning clipping paths
without losing row-stripe background rects
- Deduplicate sub-rects (cell-internal decorations) to prevent spurious
Y-edge splits, with height constraint to preserve row-stripe patterns
- Raise column limit from 12 to 25 for statistical lookup tables (MWU,
chi-square)
- Skip propagate_merged_cells for wide tables (>10 cols) where spanning
rects are background fills, not true merged cells
- Add numeric cell check to continuation-row heuristic so short text
labels (e.g. "Liquid", "Vapour") are still merged while numeric data
rows are kept separate
- Raise row-stripe content density threshold from 25% to 40% to reject
false-positive tables from alternating-shade prose sections
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Many IRS forms and government PDFs draw table gridlines using path
operators (m/l/S) instead of rectangle (re) operators. This adds
line-based table detection to capture these tables.
- Add PdfLine type for line segments from path operators
- Capture m/l/h/S/s/B/b/f/n path operators in content_stream.rs
- Thread Vec<PdfLine> through extraction pipeline
- New detect_lines.rs: classify lines, snap to grid, validate and
assign items with extensive false-positive filters
- Integrate in markdown pipeline: rects first, then lines as fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip leading/trailing digit sequences (page numbers) before frequency
comparison so headers like "Page 5" and "Page 6" are treated as identical.
Group TextLines at the same Y position into Y-bands and propagate removal
to all siblings when any member is stripped. Increase EDGE_LINE_COUNT from
4 to 5 to cover 5-row form column headers (e.g., IRS p1244).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bumps EDGE_LINE_COUNT from 3 to 4 to strip repeated form column headers
that sit just inside the page margin (4th-from-edge Y position).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add strip_repeated_lines() preprocessing step that removes running
headers and footers before markdown conversion. Uses multiple guard
rails to avoid false positives:
- Edge-line detection: only considers lines among the first/last 3
distinct Y positions on each page (not percentage-based margins)
- Y-position consistency: requires low variance across pages (stddev
< 5% of page span) to distinguish headers from scattered table content
- Position-aware removal: only strips instances at page edges, preserving
body content that happens to match a header/footer text
- Skips structural lines (headings, list items), short lines (<10 chars),
and decorative separators (repeated single characters)
- Frequency threshold: >= max(3, page_count * 30%) distinct pages
New strip_headers_footers option on MarkdownOptions (default: true).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove `process_mode` from `MarkdownOptions` (it controlled the
pipeline, not markdown formatting)
- Remove dead `text` field from `PdfProcessResult`
- Add `PdfOptions` builder consolidating mode, detection, markdown,
and page filter configuration
- Add convenience functions: `detect_pdf()`, `detect_pdf_mem()`,
`process_pdf_with_options()`, `process_pdf_mem_with_options()`
- Eliminate double document parsing: load once, share between
detection and extraction via internal `pub(crate)` helpers
- Deprecate old `process_pdf_with_config*` functions (kept as shims)
- Update binaries to use new API
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The R6 encryption fix has been merged upstream, so switch from the
fork back to the canonical repository.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
EM SPACE from PDF ActualText entries was lost by text.trim(), breaking
spacing after bullets and numbered list markers. Normalizing to ASCII
space lets should_join_items detect word boundaries naturally. NBSP
(U+00A0) is excluded as it's handled by coordinate-based spacing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Soft hyphens, zero-width spaces, BOM, ZWNJ/ZWJ, and word joiners now get
stripped in expand_ligatures(). Also call expand_ligatures() on the ActualText
code path which was the only TextItem creation site missing it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use abimaelmartell/lopdf firecrawl/fix-r6-encryption branch which
accepts /O and /U values longer than 48 bytes in V5/R6 encrypted PDFs.
This unblocks 4 encrypted PDFs in the bench suite that were previously
rejected with InvalidHashLength.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Map lopdf's Unimplemented("encrypted...") error to PdfError::Encrypted
instead of falling through to PdfError::Parse. Retry all Document::load
calls with an empty password when encryption is detected, so
owner-password-only PDFs can be opened. If the retry also fails, the
user now sees a clear "PDF is encrypted" message.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unit test verifying short sub-header rows (e.g. month names) are not
merged as continuation rows in table formatting. Snapshot test for
2013_app2.pdf pinning the full row-stripe table output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When PDF tables use full-width alternating row shading (row-stripe rects),
the normal rect grid detection fails because all rects share the same X/width,
collapsing to ~1 column. Add a fallback that uses rect Y-edges for rows and
text X-position clustering for columns, with a lower 15pt threshold to
separate narrow columns like row numbers and dates.
Also fix continuation-row merging in table formatting to not merge short
single-cell rows (≤5 chars) that are section sub-headers (e.g. month names).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The chest-wall PDF uses $ as separator but also has trailing $ after
spaces, giving only 13.7% letter-dollar-letter ratio (594 of 4332).
Add absolute count threshold (>20) alongside the ratio check so both
concentrated and dispersed substitution patterns are caught.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add has_encoding_issues field to PdfProcessResult that detects garbled
text from broken ToUnicode CMaps (U+FFFD replacement characters or
systematic dollar-as-space substitution). Surfaced in JSON output so
clients can fall back to OCR for affected PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Symbol/Wingdings fonts often map codes to PUA (U+F000-F0FF) via ToUnicode
CMaps, producing invisible/tofu characters in markdown output. Add
clean_symbol_pua() post-processing to extract_text_from_operand() that
converts these to standard Unicode: bullets to U+2022, checkmark to U+2713,
and ASCII/Latin-1 range by stripping the F000 offset.
Eliminates 1,405 PUA characters across 25+ eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add pdf.js binary CMap (bcmap) parser for Japan1/GB1/CNS1 CID fonts
- Support inline ToUnicode streams (not just object references)
- Add CMapDecisionCache for heuristic primary vs remapped CMap selection
- Build fallback CMaps from embedded font data and CIDSystemInfo
- Handle simple fonts without ToUnicode via embedded font cmap
- Add Symbol/Wingdings/ZapfDingbats font decoding fallback
- Support usecmap chaining in both text and binary CMap formats
- Scan XObject Forms for text operators in detector
- Change default detection strategy from EarlyExit to Sample(8)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Some PDF generators subset-embed fonts by renumbering GIDs sequentially
but fail to update the ToUnicode CMap, which still references original
GID values. Detect this mismatch (Identity-H, no CIDToGIDMap, W array
starts at CID ≤ 2 but CMap min CID > 2) and remap to sequential positions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hand-rolled JSON escaping only handled \, ", and \n but missed
tabs, carriage returns, and other control characters (U+0000..U+001F),
producing invalid JSON that Python's json.loads would reject. Add a
proper json_escape() function covering all required escapes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces a ProcessMode enum that controls how far the PDF pipeline
runs, enabling fast document triage without paying extraction or
markdown conversion costs. Exposed via --detect-only and --analyze
flags in both pdf2md and detect-pdf CLIs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add LayoutComplexity struct to PdfProcessResult so callers can detect
when a PDF has complex layout (tables or multi-column text) and decide
whether to use the extracted markdown or fall back to OCR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a PDF page has a graph directly above a table (e.g. Figure 2 above
Table II in real-estate-pricing.pdf), the heuristic table detector would
merge graph axis labels and legend items into the table, producing a
corrupted result.
Add rect hint regions: when a small cluster of cell-border rects (4-6)
fails full grid validation (e.g. only row borders, no column dividers),
extract their Y bounding box as a "hint region". The heuristic detector
then runs separately on items inside vs outside hint regions, preventing
unrelated content from being merged into tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rects spanning multiple grid rows (e.g. classification labels) now have
their text consolidated into the first sub-row via propagate_merged_cells().
Also prevent continuation-row merge from folding body data into the header.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 5 stripped public-domain PDF fixtures and golden markdown snapshots
for CI regression testing. Fix non-deterministic output caused by
HashMap iteration order in font stats, table heuristics, and rect
clustering by adding deterministic tie-breaking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
