When a heading wraps across multiple visual lines, the structure tree
creates separate heading nodes per line, producing broken output like
`## Snow` / `## Lake`. Extend merge_heading_lines to consider struct-tree
roles (H1-H6) in addition to font-size heuristics. A 20-word combined
limit prevents merging body text mis-tagged as headings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: tagged PDF structure tree support for semantic markdown generation
Parse /StructTreeRoot from tagged PDFs and use semantic roles (H1-H6, P,
LI, BlockQuote, Code, Caption) to improve markdown output. Structure tree
headings add to font-size heuristics without suppressing them. Coverage
threshold (≥50%) ensures only properly tagged PDFs activate this path.
Phase 1: Parse structure tree with role maps, MCID collection, flattening
Phase 2: Capture MCIDs from BMC/BDC operators, tag TextItems
Phase 3: Structure-aware markdown generation in convert loop
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: accumulate consecutive code lines into single fenced block
Per-line code fencing produced broken markdown for multi-line code
blocks (separate ``` open/close per line). Unify struct-tree Code
role and font-based monospace detection into a single is_code_line
check with in_code_block state for proper accumulation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add tagged PDF fixture with Firecrawl docs content
Synthetic 7-page PDF with rich structure tree exercising H1, H2, H3,
P, Code, LI, Caption, TH, TD roles. Generated via fpdf2 script.
Integration test verifies struct tree parsing and code fence output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: remove python PDF generator script from repo
Keep the generated fixture PDF but don't track the generator script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle malformed bare-name struct types in tagged PDFs
Some PDF generators (e.g. fpdf2) write /S Code instead of /S /Code
in structure elements. lopdf silently drops these objects since bare
tokens are invalid PDF syntax. Add a pre-processor that scans for
known bare struct type names and prepends / before loading.
Unifies path and memory loading through the same fix pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: update lopdf dependency to main branch
The firecrawl/zlib-checksum-encrypted branch was merged and deleted.
Point to main which includes all previously merged fixes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: switch lopdf to upstream repo pinned at 845cd3d
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Narrow annotation columns beside wide body columns were being
Y-interleaved with body text, producing garbled reading order.
Add sidebar detection with width ratio, line balance, and sparse
density guards to correctly trigger sequential column reading.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use character-width ratios instead of font_size for Canva-style PDFs:
- Single-char prev: gap/prev.width < 1.25
- Multi→single: gap/avg_prev_char_width < 1.25
- Multi→multi: page-level threshold (gap/font_size)
Also adds second Canva detection path for per-character rendering
without embedded spaces (>50% single-char items).
Improves ebgt7isj04ophcq word accuracy from ~3% to 64%.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Canva-generated PDFs render text character-by-character with CSS-style
letter-spacing (~0.5-0.9× font_size). The hardcoded 0.10 threshold
caused every character to get a space inserted ("K a r i b i b").
Detect Canva pages via fix_letterspaced_items (≥50% items match "a b c"
pattern), compute an IQR-based threshold (median × 1.55) on the gap
distribution BEFORE space removal, then propagate per-page thresholds
through PageThresholds → group_into_lines_with_thresholds → TextLine
→ should_join_items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Drop column-width padding from table output. Saves ~37% bytes on
table-heavy PDFs. Markdown renders identically — padding was purely
cosmetic. Optimizes token efficiency for AI agent consumers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Arabic PDFs using ToUnicode maps to presentation forms (U+FB50-FDFF,
U+FE70-FEFE) produced unreadable output. Apply NFKC normalization to
convert back to base Arabic (U+0600-06FF), then reverse character order
within items that had presentation forms (visual→logical order).
Improves Arabic PDF word overlap with Mistral from ~4.5% to ~78.6%.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add bimodal gap analysis to find_column_boundaries for PDFs with many
narrow columns (e.g. 24-column train schedules at 26pt spacing). When
a clear gap between within-column jitter and between-column spacing is
detected, switch to edge-based clustering with a lower threshold.
Also raise heuristic max columns from 15 to 25, and relax column
consistency validation for very wide tables (>15 cols).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tables with very wide cells (e.g. 230+ chars from merged schedule columns)
caused massive whitespace padding in every row. Capping alignment at 40
characters reduces output size significantly (franklin: 77KB→24KB,
Closed-Business: 571KB→502KB) without affecting content.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Subset fonts number GIDs by document encounter order, not sorted order.
The sequential remap scrambled character mappings, losing 2,532+ CJK
characters. Now promotes TrueType cmap fallback when it has more entries
than the primary ToUnicode CMap. Also adds CJK scoring to score_text()
as defense-in-depth.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Points to firecrawl/lopdf branch that falls back to raw deflate when
zlib adler32 checksum is corrupted after RC4 decryption. Fixes 5
encrypted BrokerCheck PDFs that were misclassified as scanned.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rect-guided tables:
- Add try_build_rect_guided_table() to build tables from rect cluster
X positions as column boundaries, bypassing heuristic detection
- Split merged multi-number TextItems (e.g. "10 11 12...31") into
individual day-column cells using column-advancing boundary assignment
- Interpolate missing column boundaries for holiday/non-work days
that lack colored rects
- Strip tilde-leader noise from cells (legend text bleeding)
- Filter legend text beyond table area via max-X threshold
- Pass cluster_rects through RectHintRegion for downstream use
Extractor improvements:
- Deduplicate clip-path and fill-path rects before using as table hints
- Add font width fallback for missing glyph metrics via average width
- Improve column detection scoring to prefer balanced gutters
- Handle side-by-side layouts with hint-region derived split points
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 48 new tests (140 → 188 total) covering previously untested private
functions with synthetic data. No PDF fixtures needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Images nested inside Form XObjects (e.g. HuttoISDWorkPerks.pdf) were
invisible to the detector. Rewrote analyze_page_images to recursively
follow Form XObjects. Also added pages_with_vector_text tracking so
PDFs with vector-outlined text (e.g. qsop_21-0055.pdf) are correctly
classified as IMAGE-BASED with OCR recommended.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The retry mechanism (excluding page-background rects) was firing for any
grid rejection, including "column 0 is completely empty". This caused a
regression on HTM_02-01_Part_A where the retry accepted a bad 12x9 grid
instead of falling through to heuristic detection.
Introduce GridResult enum so the caller only retries on FewNonEmptyRows
(the specific symptom of propagate_merged_cells collapse), not on
structural failures like empty columns or low fill ratio.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs with full-page background fills (e.g. 6_KE_Chart_5-5-17.pdf) had
their table grids rejected because propagate_merged_cells collapsed all
text into row 0. Add a retry mechanism: normal detection first, then
if it fails and page-origin background rects are present (>=95% width,
>=90% height, near origin), retry with those rects excluded from X-edge
extraction and propagate_merged_cells.
The retry path uses strict validation (50% non-empty rows, 40% content
density, max 200 chars per cell) to avoid false positives on pages where
paragraph text would be incorrectly captured in the grid.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pages using nested Form XObject chains (page → wrapper Form → text Form)
were returning 0 text items because extract_form_xobject_text() didn't
handle the Do operator. Add Do/q/Q/cm handling with recursive descent
(max depth 5) and proper CTM tracking through the nesting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pages with images AND very few text operators (<10) are image pages
with overlay text (headers/footers), not real text pages. This fixes
image-heavy PDFs like newspaper ads being misclassified as TextBased.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The firecrawl/fix-cmap-optional-slash branch caused a regression on
the Italian PDF (garbled CMap-encoded text). The EPEVER fix was a
wrong diagnosis — the PDF extracts correctly on upstream main.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Count path construction/painting operators (m, l, c, h, f, S, re, etc.)
during the existing content stream scan. Pages where path ops vastly
outnumber text ops (>=1000 path ops AND >200x text ops) contain
vector-outlined glyphs that need OCR instead of text extraction.
This fixes EPEVER-DataSheet-XTRA-N-G3-Series-3.pdf page 2 which has
41,263 path ops but only 33 Tj ops — now correctly classified as Mixed
with page 2 flagged for OCR.
Also removes the check_page_decode_health phase (Phase 1.5) which
required parsing content streams into operations — the path-based
detection is both simpler and more effective.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switches to a new lopdf branch that includes the CMap parser fix for
PDFs with uncompressed ToUnicode streams that omit the leading slash
on metadata keys (CIDSystemInfo, CMapName, CMapType).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like fare1011.pdf draw table row stripes and borders using
m/l/h/f* filled paths instead of `re` operators. Extract axis-aligned
rectangles from these filled subpaths as a third-priority rect source
(after `re` rects and clip-path rects).
Key changes:
- Track completed subpaths in `h` handler for f/f* rect extraction
- Update W/W* handler to read from pending_subpaths when pending_lines
is empty (after h clears them)
- Add row-stripe fallback in detect_rects when clustering produces no
large clusters (non-overlapping row stripes), with ≥15 rects and
≥10 rows required to prevent decorative fill false positives
- Revert body-font upper bound to 1.05x (no longer needed at 1.10x)
fare1011.pdf: 12-col heuristic table → 14-col rect-based table with
all fare values correctly captured.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract axis-aligned rectangles from W/W* clip operators in content
streams — many PDFs define table cells as clipping paths instead of
stroked rects. Add merged-cluster fallback in detect_tables_from_rects()
that merges all cluster rects when per-cluster detection fails or only
produces narrow false-positives (≤3 columns). Uses rect Y-edges for
rows and text X-clustering for columns.
Also updates lopdf to firecrawl fork (fix-leading-whitespace branch).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Thread a shared HashSet<u8> through scan_content_for_text_operators and
scan_xobjects_in_resources instead of returning a count and re-scanning
with a separate collect_unique_chars_from_content function.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like wildfire_waiver.pdf render each line as a separate image XObject
with only bullet characters as actual text operators. Add image-dominance
heuristic (image_count > 10 && image_count > text_ops * 3) and unique text
character check (>= 5 unique non-whitespace chars) so these pages are no
longer misclassified as text-based.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only skip text with rendering mode 3 (truly invisible). Removes
fill color tracking that was incorrectly hiding white-on-dark text
in presentations and brochures.
- Lower gap threshold from 40pt to 30pt to catch narrower inter-table gaps
- Constrain candidates to middle 60% of page X range
- Use center-position balance check instead of left-edge counting
- Add balanced-cluster dedup (50pt) to distinguish side-by-side tables
from single wide tables with multiple column gaps
- Improve filter_rects_to_band with 70% containment check to exclude
page-wide background stripes
Split pages with two independent table regions (e.g. D'Addario tension
chart) into separate bands before running rect/line/heuristic detection.
The split function finds X-position gaps ≥40pt, validates that <2% of
items cross the split point, and requires ≥30% balance on each side.
Also adds heuristic table detection fallback in compute_layout_complexity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs that set fill color via `sc`/`scn` (named color space operators)
instead of `g`/`rg`/`k` were incorrectly keeping fill_is_white=true
from earlier page background fills, causing all text to be skipped.
Infer color model from operand count: 3=RGB, 4=CMYK, other=non-white.
Fixes extraction of tension_chart PDF (D'Addario string specifications)
which previously output only glyph names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add width-based outlier filter to remove page-spanning clipping paths
without losing row-stripe background rects
- Deduplicate sub-rects (cell-internal decorations) to prevent spurious
Y-edge splits, with height constraint to preserve row-stripe patterns
- Raise column limit from 12 to 25 for statistical lookup tables (MWU,
chi-square)
- Skip propagate_merged_cells for wide tables (>10 cols) where spanning
rects are background fills, not true merged cells
- Add numeric cell check to continuation-row heuristic so short text
labels (e.g. "Liquid", "Vapour") are still merged while numeric data
rows are kept separate
- Raise row-stripe content density threshold from 25% to 40% to reject
false-positive tables from alternating-shade prose sections
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Many IRS forms and government PDFs draw table gridlines using path
operators (m/l/S) instead of rectangle (re) operators. This adds
line-based table detection to capture these tables.
- Add PdfLine type for line segments from path operators
- Capture m/l/h/S/s/B/b/f/n path operators in content_stream.rs
- Thread Vec<PdfLine> through extraction pipeline
- New detect_lines.rs: classify lines, snap to grid, validate and
assign items with extensive false-positive filters
- Integrate in markdown pipeline: rects first, then lines as fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip leading/trailing digit sequences (page numbers) before frequency
comparison so headers like "Page 5" and "Page 6" are treated as identical.
Group TextLines at the same Y position into Y-bands and propagate removal
to all siblings when any member is stripped. Increase EDGE_LINE_COUNT from
4 to 5 to cover 5-row form column headers (e.g., IRS p1244).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bumps EDGE_LINE_COUNT from 3 to 4 to strip repeated form column headers
that sit just inside the page margin (4th-from-edge Y position).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add strip_repeated_lines() preprocessing step that removes running
headers and footers before markdown conversion. Uses multiple guard
rails to avoid false positives:
- Edge-line detection: only considers lines among the first/last 3
distinct Y positions on each page (not percentage-based margins)
- Y-position consistency: requires low variance across pages (stddev
< 5% of page span) to distinguish headers from scattered table content
- Position-aware removal: only strips instances at page edges, preserving
body content that happens to match a header/footer text
- Skips structural lines (headings, list items), short lines (<10 chars),
and decorative separators (repeated single characters)
- Frequency threshold: >= max(3, page_count * 30%) distinct pages
New strip_headers_footers option on MarkdownOptions (default: true).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove `process_mode` from `MarkdownOptions` (it controlled the
pipeline, not markdown formatting)
- Remove dead `text` field from `PdfProcessResult`
- Add `PdfOptions` builder consolidating mode, detection, markdown,
and page filter configuration
- Add convenience functions: `detect_pdf()`, `detect_pdf_mem()`,
`process_pdf_with_options()`, `process_pdf_mem_with_options()`
- Eliminate double document parsing: load once, share between
detection and extraction via internal `pub(crate)` helpers
- Deprecate old `process_pdf_with_config*` functions (kept as shims)
- Update binaries to use new API
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The R6 encryption fix has been merged upstream, so switch from the
fork back to the canonical repository.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
EM SPACE from PDF ActualText entries was lost by text.trim(), breaking
spacing after bullets and numbered list markers. Normalizing to ASCII
space lets should_join_items detect word boundaries naturally. NBSP
(U+00A0) is excluded as it's handled by coordinate-based spacing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Soft hyphens, zero-width spaces, BOM, ZWNJ/ZWJ, and word joiners now get
stripped in expand_ligatures(). Also call expand_ligatures() on the ActualText
code path which was the only TextItem creation site missing it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use abimaelmartell/lopdf firecrawl/fix-r6-encryption branch which
accepts /O and /U values longer than 48 bytes in V5/R6 encrypted PDFs.
This unblocks 4 encrypted PDFs in the bench suite that were previously
rejected with InvalidHashLength.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Map lopdf's Unimplemented("encrypted...") error to PdfError::Encrypted
instead of falling through to PdfError::Parse. Retry all Document::load
calls with an empty password when encryption is detected, so
owner-password-only PDFs can be opened. If the retry also fails, the
user now sees a clear "PDF is encrypted" message.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unit test verifying short sub-header rows (e.g. month names) are not
merged as continuation rows in table formatting. Snapshot test for
2013_app2.pdf pinning the full row-stripe table output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When PDF tables use full-width alternating row shading (row-stripe rects),
the normal rect grid detection fails because all rects share the same X/width,
collapsing to ~1 column. Add a fallback that uses rect Y-edges for rows and
text X-position clustering for columns, with a lower 15pt threshold to
separate narrow columns like row numbers and dates.
Also fix continuation-row merging in table formatting to not merge short
single-cell rows (≤5 chars) that are section sub-headers (e.g. month names).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The chest-wall PDF uses $ as separator but also has trailing $ after
spaces, giving only 13.7% letter-dollar-letter ratio (594 of 4332).
Add absolute count threshold (>20) alongside the ratio check so both
concentrated and dispersed substitution patterns are caught.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
