Add bimodal gap analysis to find_column_boundaries for PDFs with many
narrow columns (e.g. 24-column train schedules at 26pt spacing). When
a clear gap between within-column jitter and between-column spacing is
detected, switch to edge-based clustering with a lower threshold.
Also raise heuristic max columns from 15 to 25, and relax column
consistency validation for very wide tables (>15 cols).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tables with very wide cells (e.g. 230+ chars from merged schedule columns)
caused massive whitespace padding in every row. Capping alignment at 40
characters reduces output size significantly (franklin: 77KB→24KB,
Closed-Business: 571KB→502KB) without affecting content.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Subset fonts number GIDs by document encounter order, not sorted order.
The sequential remap scrambled character mappings, losing 2,532+ CJK
characters. Now promotes TrueType cmap fallback when it has more entries
than the primary ToUnicode CMap. Also adds CJK scoring to score_text()
as defense-in-depth.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rect-guided tables:
- Add try_build_rect_guided_table() to build tables from rect cluster
X positions as column boundaries, bypassing heuristic detection
- Split merged multi-number TextItems (e.g. "10 11 12...31") into
individual day-column cells using column-advancing boundary assignment
- Interpolate missing column boundaries for holiday/non-work days
that lack colored rects
- Strip tilde-leader noise from cells (legend text bleeding)
- Filter legend text beyond table area via max-X threshold
- Pass cluster_rects through RectHintRegion for downstream use
Extractor improvements:
- Deduplicate clip-path and fill-path rects before using as table hints
- Add font width fallback for missing glyph metrics via average width
- Improve column detection scoring to prefer balanced gutters
- Handle side-by-side layouts with hint-region derived split points
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 48 new tests (140 → 188 total) covering previously untested private
functions with synthetic data. No PDF fixtures needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Images nested inside Form XObjects (e.g. HuttoISDWorkPerks.pdf) were
invisible to the detector. Rewrote analyze_page_images to recursively
follow Form XObjects. Also added pages_with_vector_text tracking so
PDFs with vector-outlined text (e.g. qsop_21-0055.pdf) are correctly
classified as IMAGE-BASED with OCR recommended.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The retry mechanism (excluding page-background rects) was firing for any
grid rejection, including "column 0 is completely empty". This caused a
regression on HTM_02-01_Part_A where the retry accepted a bad 12x9 grid
instead of falling through to heuristic detection.
Introduce GridResult enum so the caller only retries on FewNonEmptyRows
(the specific symptom of propagate_merged_cells collapse), not on
structural failures like empty columns or low fill ratio.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs with full-page background fills (e.g. 6_KE_Chart_5-5-17.pdf) had
their table grids rejected because propagate_merged_cells collapsed all
text into row 0. Add a retry mechanism: normal detection first, then
if it fails and page-origin background rects are present (>=95% width,
>=90% height, near origin), retry with those rects excluded from X-edge
extraction and propagate_merged_cells.
The retry path uses strict validation (50% non-empty rows, 40% content
density, max 200 chars per cell) to avoid false positives on pages where
paragraph text would be incorrectly captured in the grid.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pages using nested Form XObject chains (page → wrapper Form → text Form)
were returning 0 text items because extract_form_xobject_text() didn't
handle the Do operator. Add Do/q/Q/cm handling with recursive descent
(max depth 5) and proper CTM tracking through the nesting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pages with images AND very few text operators (<10) are image pages
with overlay text (headers/footers), not real text pages. This fixes
image-heavy PDFs like newspaper ads being misclassified as TextBased.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Count path construction/painting operators (m, l, c, h, f, S, re, etc.)
during the existing content stream scan. Pages where path ops vastly
outnumber text ops (>=1000 path ops AND >200x text ops) contain
vector-outlined glyphs that need OCR instead of text extraction.
This fixes EPEVER-DataSheet-XTRA-N-G3-Series-3.pdf page 2 which has
41,263 path ops but only 33 Tj ops — now correctly classified as Mixed
with page 2 flagged for OCR.
Also removes the check_page_decode_health phase (Phase 1.5) which
required parsing content streams into operations — the path-based
detection is both simpler and more effective.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like fare1011.pdf draw table row stripes and borders using
m/l/h/f* filled paths instead of `re` operators. Extract axis-aligned
rectangles from these filled subpaths as a third-priority rect source
(after `re` rects and clip-path rects).
Key changes:
- Track completed subpaths in `h` handler for f/f* rect extraction
- Update W/W* handler to read from pending_subpaths when pending_lines
is empty (after h clears them)
- Add row-stripe fallback in detect_rects when clustering produces no
large clusters (non-overlapping row stripes), with ≥15 rects and
≥10 rows required to prevent decorative fill false positives
- Revert body-font upper bound to 1.05x (no longer needed at 1.10x)
fare1011.pdf: 12-col heuristic table → 14-col rect-based table with
all fare values correctly captured.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract axis-aligned rectangles from W/W* clip operators in content
streams — many PDFs define table cells as clipping paths instead of
stroked rects. Add merged-cluster fallback in detect_tables_from_rects()
that merges all cluster rects when per-cluster detection fails or only
produces narrow false-positives (≤3 columns). Uses rect Y-edges for
rows and text X-clustering for columns.
Also updates lopdf to firecrawl fork (fix-leading-whitespace branch).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Thread a shared HashSet<u8> through scan_content_for_text_operators and
scan_xobjects_in_resources instead of returning a count and re-scanning
with a separate collect_unique_chars_from_content function.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like wildfire_waiver.pdf render each line as a separate image XObject
with only bullet characters as actual text operators. Add image-dominance
heuristic (image_count > 10 && image_count > text_ops * 3) and unique text
character check (>= 5 unique non-whitespace chars) so these pages are no
longer misclassified as text-based.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only skip text with rendering mode 3 (truly invisible). Removes
fill color tracking that was incorrectly hiding white-on-dark text
in presentations and brochures.
- Lower gap threshold from 40pt to 30pt to catch narrower inter-table gaps
- Constrain candidates to middle 60% of page X range
- Use center-position balance check instead of left-edge counting
- Add balanced-cluster dedup (50pt) to distinguish side-by-side tables
from single wide tables with multiple column gaps
- Improve filter_rects_to_band with 70% containment check to exclude
page-wide background stripes
Split pages with two independent table regions (e.g. D'Addario tension
chart) into separate bands before running rect/line/heuristic detection.
The split function finds X-position gaps ≥40pt, validates that <2% of
items cross the split point, and requires ≥30% balance on each side.
Also adds heuristic table detection fallback in compute_layout_complexity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs that set fill color via `sc`/`scn` (named color space operators)
instead of `g`/`rg`/`k` were incorrectly keeping fill_is_white=true
from earlier page background fills, causing all text to be skipped.
Infer color model from operand count: 3=RGB, 4=CMYK, other=non-white.
Fixes extraction of tension_chart PDF (D'Addario string specifications)
which previously output only glyph names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add width-based outlier filter to remove page-spanning clipping paths
without losing row-stripe background rects
- Deduplicate sub-rects (cell-internal decorations) to prevent spurious
Y-edge splits, with height constraint to preserve row-stripe patterns
- Raise column limit from 12 to 25 for statistical lookup tables (MWU,
chi-square)
- Skip propagate_merged_cells for wide tables (>10 cols) where spanning
rects are background fills, not true merged cells
- Add numeric cell check to continuation-row heuristic so short text
labels (e.g. "Liquid", "Vapour") are still merged while numeric data
rows are kept separate
- Raise row-stripe content density threshold from 25% to 40% to reject
false-positive tables from alternating-shade prose sections
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Many IRS forms and government PDFs draw table gridlines using path
operators (m/l/S) instead of rectangle (re) operators. This adds
line-based table detection to capture these tables.
- Add PdfLine type for line segments from path operators
- Capture m/l/h/S/s/B/b/f/n path operators in content_stream.rs
- Thread Vec<PdfLine> through extraction pipeline
- New detect_lines.rs: classify lines, snap to grid, validate and
assign items with extensive false-positive filters
- Integrate in markdown pipeline: rects first, then lines as fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip leading/trailing digit sequences (page numbers) before frequency
comparison so headers like "Page 5" and "Page 6" are treated as identical.
Group TextLines at the same Y position into Y-bands and propagate removal
to all siblings when any member is stripped. Increase EDGE_LINE_COUNT from
4 to 5 to cover 5-row form column headers (e.g., IRS p1244).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bumps EDGE_LINE_COUNT from 3 to 4 to strip repeated form column headers
that sit just inside the page margin (4th-from-edge Y position).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add strip_repeated_lines() preprocessing step that removes running
headers and footers before markdown conversion. Uses multiple guard
rails to avoid false positives:
- Edge-line detection: only considers lines among the first/last 3
distinct Y positions on each page (not percentage-based margins)
- Y-position consistency: requires low variance across pages (stddev
< 5% of page span) to distinguish headers from scattered table content
- Position-aware removal: only strips instances at page edges, preserving
body content that happens to match a header/footer text
- Skips structural lines (headings, list items), short lines (<10 chars),
and decorative separators (repeated single characters)
- Frequency threshold: >= max(3, page_count * 30%) distinct pages
New strip_headers_footers option on MarkdownOptions (default: true).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove `process_mode` from `MarkdownOptions` (it controlled the
pipeline, not markdown formatting)
- Remove dead `text` field from `PdfProcessResult`
- Add `PdfOptions` builder consolidating mode, detection, markdown,
and page filter configuration
- Add convenience functions: `detect_pdf()`, `detect_pdf_mem()`,
`process_pdf_with_options()`, `process_pdf_mem_with_options()`
- Eliminate double document parsing: load once, share between
detection and extraction via internal `pub(crate)` helpers
- Deprecate old `process_pdf_with_config*` functions (kept as shims)
- Update binaries to use new API
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
EM SPACE from PDF ActualText entries was lost by text.trim(), breaking
spacing after bullets and numbered list markers. Normalizing to ASCII
space lets should_join_items detect word boundaries naturally. NBSP
(U+00A0) is excluded as it's handled by coordinate-based spacing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Soft hyphens, zero-width spaces, BOM, ZWNJ/ZWJ, and word joiners now get
stripped in expand_ligatures(). Also call expand_ligatures() on the ActualText
code path which was the only TextItem creation site missing it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Map lopdf's Unimplemented("encrypted...") error to PdfError::Encrypted
instead of falling through to PdfError::Parse. Retry all Document::load
calls with an empty password when encryption is detected, so
owner-password-only PDFs can be opened. If the retry also fails, the
user now sees a clear "PDF is encrypted" message.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unit test verifying short sub-header rows (e.g. month names) are not
merged as continuation rows in table formatting. Snapshot test for
2013_app2.pdf pinning the full row-stripe table output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When PDF tables use full-width alternating row shading (row-stripe rects),
the normal rect grid detection fails because all rects share the same X/width,
collapsing to ~1 column. Add a fallback that uses rect Y-edges for rows and
text X-position clustering for columns, with a lower 15pt threshold to
separate narrow columns like row numbers and dates.
Also fix continuation-row merging in table formatting to not merge short
single-cell rows (≤5 chars) that are section sub-headers (e.g. month names).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The chest-wall PDF uses $ as separator but also has trailing $ after
spaces, giving only 13.7% letter-dollar-letter ratio (594 of 4332).
Add absolute count threshold (>20) alongside the ratio check so both
concentrated and dispersed substitution patterns are caught.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add has_encoding_issues field to PdfProcessResult that detects garbled
text from broken ToUnicode CMaps (U+FFFD replacement characters or
systematic dollar-as-space substitution). Surfaced in JSON output so
clients can fall back to OCR for affected PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Symbol/Wingdings fonts often map codes to PUA (U+F000-F0FF) via ToUnicode
CMaps, producing invisible/tofu characters in markdown output. Add
clean_symbol_pua() post-processing to extract_text_from_operand() that
converts these to standard Unicode: bullets to U+2022, checkmark to U+2713,
and ASCII/Latin-1 range by stripping the F000 offset.
Eliminates 1,405 PUA characters across 25+ eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add pdf.js binary CMap (bcmap) parser for Japan1/GB1/CNS1 CID fonts
- Support inline ToUnicode streams (not just object references)
- Add CMapDecisionCache for heuristic primary vs remapped CMap selection
- Build fallback CMaps from embedded font data and CIDSystemInfo
- Handle simple fonts without ToUnicode via embedded font cmap
- Add Symbol/Wingdings/ZapfDingbats font decoding fallback
- Support usecmap chaining in both text and binary CMap formats
- Scan XObject Forms for text operators in detector
- Change default detection strategy from EarlyExit to Sample(8)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Some PDF generators subset-embed fonts by renumbering GIDs sequentially
but fail to update the ToUnicode CMap, which still references original
GID values. Detect this mismatch (Identity-H, no CIDToGIDMap, W array
starts at CID ≤ 2 but CMap min CID > 2) and remap to sequential positions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hand-rolled JSON escaping only handled \, ", and \n but missed
tabs, carriage returns, and other control characters (U+0000..U+001F),
producing invalid JSON that Python's json.loads would reject. Add a
proper json_escape() function covering all required escapes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces a ProcessMode enum that controls how far the PDF pipeline
runs, enabling fast document triage without paying extraction or
markdown conversion costs. Exposed via --detect-only and --analyze
flags in both pdf2md and detect-pdf CLIs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add LayoutComplexity struct to PdfProcessResult so callers can detect
when a PDF has complex layout (tables or multi-column text) and decide
whether to use the extracted markdown or fall back to OCR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a PDF page has a graph directly above a table (e.g. Figure 2 above
Table II in real-estate-pricing.pdf), the heuristic table detector would
merge graph axis labels and legend items into the table, producing a
corrupted result.
Add rect hint regions: when a small cluster of cell-border rects (4-6)
fails full grid validation (e.g. only row borders, no column dividers),
extract their Y bounding box as a "hint region". The heuristic detector
then runs separately on items inside vs outside hint regions, preventing
unrelated content from being merged into tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rects spanning multiple grid rows (e.g. classification labels) now have
their text consolidated into the first sub-row via propagate_merged_cells().
Also prevent continuation-row merge from folding body data into the header.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 5 stripped public-domain PDF fixtures and golden markdown snapshots
for CI regression testing. Fix non-deterministic output caused by
HashMap iteration order in font stats, table heuristics, and rect
clustering by adding deterministic tie-breaking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The snap_edges tolerance of 2.0 was too tight for PDFs where header and
body cell boundaries differ by ~3pt due to cell padding. This created
phantom empty columns that caused valid tables to be rejected. Increase
snap and cell coverage tolerances from 2.0/3.0 to 6.0. Also add debug
logging to rect-based table detection pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
