OCR text layers and some PDF producers emit trailing spaces on each
text item, which combine with gap-based joining to produce double
spaces ("Vice President"). Now collapses runs of 2+ spaces to single
space within lines, preserving leading indentation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scanned PDFs with OCR text layers use rendering mode 3 (invisible text)
positioned behind page images. Previously we skipped all Tr=3 text.
Now for Mixed/template PDFs, if normal extraction produces garbage or
empty output, we retry with invisible text included. This unlocks text
from OCR-generated PDFs without requiring external OCR.
Also adds Windows Unicode BMP (3,1) subtable support to the TrueType
cmap fallback, and allows TrueType fonts with explicit encoding to
extract their embedded cmap (OCR fonts often lie about encoding).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scanned PDFs with JBIG2/tiled image strips were misclassified as
TextBased because no individual image tile exceeded the template
threshold. Now checks aggregate image area per page (≥2M pixels).
Also adds is_garbage_text() check: if a Mixed/template PDF's extracted
text is predominantly non-alphanumeric (<50%), upgrade to Scanned so
callers use proper OCR instead of the garbage text layer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Multi-item lines (titles, section headers) that span across detected
columns were being split into individual column buckets, corrupting
newspaper detection and reading order. Add identify_spanning_lines()
that groups items by Y-proximity and marks lines wider than 1.3x the
widest column that have no gap at a detected gutter boundary.
Uses column-aware thresholds and gutter-interval gap detection for
precision — only fires on pages with detected columns, preserves
original item order via boolean mask.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a heading wraps across multiple visual lines, the structure tree
creates separate heading nodes per line, producing broken output like
`## Snow` / `## Lake`. Extend merge_heading_lines to consider struct-tree
roles (H1-H6) in addition to font-size heuristics. A 20-word combined
limit prevents merging body text mis-tagged as headings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: tagged PDF structure tree support for semantic markdown generation
Parse /StructTreeRoot from tagged PDFs and use semantic roles (H1-H6, P,
LI, BlockQuote, Code, Caption) to improve markdown output. Structure tree
headings add to font-size heuristics without suppressing them. Coverage
threshold (≥50%) ensures only properly tagged PDFs activate this path.
Phase 1: Parse structure tree with role maps, MCID collection, flattening
Phase 2: Capture MCIDs from BMC/BDC operators, tag TextItems
Phase 3: Structure-aware markdown generation in convert loop
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: accumulate consecutive code lines into single fenced block
Per-line code fencing produced broken markdown for multi-line code
blocks (separate ``` open/close per line). Unify struct-tree Code
role and font-based monospace detection into a single is_code_line
check with in_code_block state for proper accumulation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add tagged PDF fixture with Firecrawl docs content
Synthetic 7-page PDF with rich structure tree exercising H1, H2, H3,
P, Code, LI, Caption, TH, TD roles. Generated via fpdf2 script.
Integration test verifies struct tree parsing and code fence output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: remove python PDF generator script from repo
Keep the generated fixture PDF but don't track the generator script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle malformed bare-name struct types in tagged PDFs
Some PDF generators (e.g. fpdf2) write /S Code instead of /S /Code
in structure elements. lopdf silently drops these objects since bare
tokens are invalid PDF syntax. Add a pre-processor that scans for
known bare struct type names and prepends / before loading.
Unifies path and memory loading through the same fix pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: update lopdf dependency to main branch
The firecrawl/zlib-checksum-encrypted branch was merged and deleted.
Point to main which includes all previously merged fixes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: switch lopdf to upstream repo pinned at 845cd3d
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Narrow annotation columns beside wide body columns were being
Y-interleaved with body text, producing garbled reading order.
Add sidebar detection with width ratio, line balance, and sparse
density guards to correctly trigger sequential column reading.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use character-width ratios instead of font_size for Canva-style PDFs:
- Single-char prev: gap/prev.width < 1.25
- Multi→single: gap/avg_prev_char_width < 1.25
- Multi→multi: page-level threshold (gap/font_size)
Also adds second Canva detection path for per-character rendering
without embedded spaces (>50% single-char items).
Improves ebgt7isj04ophcq word accuracy from ~3% to 64%.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Canva-generated PDFs render text character-by-character with CSS-style
letter-spacing (~0.5-0.9× font_size). The hardcoded 0.10 threshold
caused every character to get a space inserted ("K a r i b i b").
Detect Canva pages via fix_letterspaced_items (≥50% items match "a b c"
pattern), compute an IQR-based threshold (median × 1.55) on the gap
distribution BEFORE space removal, then propagate per-page thresholds
through PageThresholds → group_into_lines_with_thresholds → TextLine
→ should_join_items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Drop column-width padding from table output. Saves ~37% bytes on
table-heavy PDFs. Markdown renders identically — padding was purely
cosmetic. Optimizes token efficiency for AI agent consumers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Arabic PDFs using ToUnicode maps to presentation forms (U+FB50-FDFF,
U+FE70-FEFE) produced unreadable output. Apply NFKC normalization to
convert back to base Arabic (U+0600-06FF), then reverse character order
within items that had presentation forms (visual→logical order).
Improves Arabic PDF word overlap with Mistral from ~4.5% to ~78.6%.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add bimodal gap analysis to find_column_boundaries for PDFs with many
narrow columns (e.g. 24-column train schedules at 26pt spacing). When
a clear gap between within-column jitter and between-column spacing is
detected, switch to edge-based clustering with a lower threshold.
Also raise heuristic max columns from 15 to 25, and relax column
consistency validation for very wide tables (>15 cols).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tables with very wide cells (e.g. 230+ chars from merged schedule columns)
caused massive whitespace padding in every row. Capping alignment at 40
characters reduces output size significantly (franklin: 77KB→24KB,
Closed-Business: 571KB→502KB) without affecting content.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Subset fonts number GIDs by document encounter order, not sorted order.
The sequential remap scrambled character mappings, losing 2,532+ CJK
characters. Now promotes TrueType cmap fallback when it has more entries
than the primary ToUnicode CMap. Also adds CJK scoring to score_text()
as defense-in-depth.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Points to firecrawl/lopdf branch that falls back to raw deflate when
zlib adler32 checksum is corrupted after RC4 decryption. Fixes 5
encrypted BrokerCheck PDFs that were misclassified as scanned.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rect-guided tables:
- Add try_build_rect_guided_table() to build tables from rect cluster
X positions as column boundaries, bypassing heuristic detection
- Split merged multi-number TextItems (e.g. "10 11 12...31") into
individual day-column cells using column-advancing boundary assignment
- Interpolate missing column boundaries for holiday/non-work days
that lack colored rects
- Strip tilde-leader noise from cells (legend text bleeding)
- Filter legend text beyond table area via max-X threshold
- Pass cluster_rects through RectHintRegion for downstream use
Extractor improvements:
- Deduplicate clip-path and fill-path rects before using as table hints
- Add font width fallback for missing glyph metrics via average width
- Improve column detection scoring to prefer balanced gutters
- Handle side-by-side layouts with hint-region derived split points
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 48 new tests (140 → 188 total) covering previously untested private
functions with synthetic data. No PDF fixtures needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Images nested inside Form XObjects (e.g. HuttoISDWorkPerks.pdf) were
invisible to the detector. Rewrote analyze_page_images to recursively
follow Form XObjects. Also added pages_with_vector_text tracking so
PDFs with vector-outlined text (e.g. qsop_21-0055.pdf) are correctly
classified as IMAGE-BASED with OCR recommended.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The retry mechanism (excluding page-background rects) was firing for any
grid rejection, including "column 0 is completely empty". This caused a
regression on HTM_02-01_Part_A where the retry accepted a bad 12x9 grid
instead of falling through to heuristic detection.
Introduce GridResult enum so the caller only retries on FewNonEmptyRows
(the specific symptom of propagate_merged_cells collapse), not on
structural failures like empty columns or low fill ratio.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs with full-page background fills (e.g. 6_KE_Chart_5-5-17.pdf) had
their table grids rejected because propagate_merged_cells collapsed all
text into row 0. Add a retry mechanism: normal detection first, then
if it fails and page-origin background rects are present (>=95% width,
>=90% height, near origin), retry with those rects excluded from X-edge
extraction and propagate_merged_cells.
The retry path uses strict validation (50% non-empty rows, 40% content
density, max 200 chars per cell) to avoid false positives on pages where
paragraph text would be incorrectly captured in the grid.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pages using nested Form XObject chains (page → wrapper Form → text Form)
were returning 0 text items because extract_form_xobject_text() didn't
handle the Do operator. Add Do/q/Q/cm handling with recursive descent
(max depth 5) and proper CTM tracking through the nesting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pages with images AND very few text operators (<10) are image pages
with overlay text (headers/footers), not real text pages. This fixes
image-heavy PDFs like newspaper ads being misclassified as TextBased.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The firecrawl/fix-cmap-optional-slash branch caused a regression on
the Italian PDF (garbled CMap-encoded text). The EPEVER fix was a
wrong diagnosis — the PDF extracts correctly on upstream main.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Count path construction/painting operators (m, l, c, h, f, S, re, etc.)
during the existing content stream scan. Pages where path ops vastly
outnumber text ops (>=1000 path ops AND >200x text ops) contain
vector-outlined glyphs that need OCR instead of text extraction.
This fixes EPEVER-DataSheet-XTRA-N-G3-Series-3.pdf page 2 which has
41,263 path ops but only 33 Tj ops — now correctly classified as Mixed
with page 2 flagged for OCR.
Also removes the check_page_decode_health phase (Phase 1.5) which
required parsing content streams into operations — the path-based
detection is both simpler and more effective.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switches to a new lopdf branch that includes the CMap parser fix for
PDFs with uncompressed ToUnicode streams that omit the leading slash
on metadata keys (CIDSystemInfo, CMapName, CMapType).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like fare1011.pdf draw table row stripes and borders using
m/l/h/f* filled paths instead of `re` operators. Extract axis-aligned
rectangles from these filled subpaths as a third-priority rect source
(after `re` rects and clip-path rects).
Key changes:
- Track completed subpaths in `h` handler for f/f* rect extraction
- Update W/W* handler to read from pending_subpaths when pending_lines
is empty (after h clears them)
- Add row-stripe fallback in detect_rects when clustering produces no
large clusters (non-overlapping row stripes), with ≥15 rects and
≥10 rows required to prevent decorative fill false positives
- Revert body-font upper bound to 1.05x (no longer needed at 1.10x)
fare1011.pdf: 12-col heuristic table → 14-col rect-based table with
all fare values correctly captured.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract axis-aligned rectangles from W/W* clip operators in content
streams — many PDFs define table cells as clipping paths instead of
stroked rects. Add merged-cluster fallback in detect_tables_from_rects()
that merges all cluster rects when per-cluster detection fails or only
produces narrow false-positives (≤3 columns). Uses rect Y-edges for
rows and text X-clustering for columns.
Also updates lopdf to firecrawl fork (fix-leading-whitespace branch).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Thread a shared HashSet<u8> through scan_content_for_text_operators and
scan_xobjects_in_resources instead of returning a count and re-scanning
with a separate collect_unique_chars_from_content function.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like wildfire_waiver.pdf render each line as a separate image XObject
with only bullet characters as actual text operators. Add image-dominance
heuristic (image_count > 10 && image_count > text_ops * 3) and unique text
character check (>= 5 unique non-whitespace chars) so these pages are no
longer misclassified as text-based.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only skip text with rendering mode 3 (truly invisible). Removes
fill color tracking that was incorrectly hiding white-on-dark text
in presentations and brochures.
- Lower gap threshold from 40pt to 30pt to catch narrower inter-table gaps
- Constrain candidates to middle 60% of page X range
- Use center-position balance check instead of left-edge counting
- Add balanced-cluster dedup (50pt) to distinguish side-by-side tables
from single wide tables with multiple column gaps
- Improve filter_rects_to_band with 70% containment check to exclude
page-wide background stripes
Split pages with two independent table regions (e.g. D'Addario tension
chart) into separate bands before running rect/line/heuristic detection.
The split function finds X-position gaps ≥40pt, validates that <2% of
items cross the split point, and requires ≥30% balance on each side.
Also adds heuristic table detection fallback in compute_layout_complexity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs that set fill color via `sc`/`scn` (named color space operators)
instead of `g`/`rg`/`k` were incorrectly keeping fill_is_white=true
from earlier page background fills, causing all text to be skipped.
Infer color model from operand count: 3=RGB, 4=CMYK, other=non-white.
Fixes extraction of tension_chart PDF (D'Addario string specifications)
which previously output only glyph names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add width-based outlier filter to remove page-spanning clipping paths
without losing row-stripe background rects
- Deduplicate sub-rects (cell-internal decorations) to prevent spurious
Y-edge splits, with height constraint to preserve row-stripe patterns
- Raise column limit from 12 to 25 for statistical lookup tables (MWU,
chi-square)
- Skip propagate_merged_cells for wide tables (>10 cols) where spanning
rects are background fills, not true merged cells
- Add numeric cell check to continuation-row heuristic so short text
labels (e.g. "Liquid", "Vapour") are still merged while numeric data
rows are kept separate
- Raise row-stripe content density threshold from 25% to 40% to reject
false-positive tables from alternating-shade prose sections
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Many IRS forms and government PDFs draw table gridlines using path
operators (m/l/S) instead of rectangle (re) operators. This adds
line-based table detection to capture these tables.
- Add PdfLine type for line segments from path operators
- Capture m/l/h/S/s/B/b/f/n path operators in content_stream.rs
- Thread Vec<PdfLine> through extraction pipeline
- New detect_lines.rs: classify lines, snap to grid, validate and
assign items with extensive false-positive filters
- Integrate in markdown pipeline: rects first, then lines as fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip leading/trailing digit sequences (page numbers) before frequency
comparison so headers like "Page 5" and "Page 6" are treated as identical.
Group TextLines at the same Y position into Y-bands and propagate removal
to all siblings when any member is stripped. Increase EDGE_LINE_COUNT from
4 to 5 to cover 5-row form column headers (e.g., IRS p1244).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bumps EDGE_LINE_COUNT from 3 to 4 to strip repeated form column headers
that sit just inside the page margin (4th-from-edge Y position).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add strip_repeated_lines() preprocessing step that removes running
headers and footers before markdown conversion. Uses multiple guard
rails to avoid false positives:
- Edge-line detection: only considers lines among the first/last 3
distinct Y positions on each page (not percentage-based margins)
- Y-position consistency: requires low variance across pages (stddev
< 5% of page span) to distinguish headers from scattered table content
- Position-aware removal: only strips instances at page edges, preserving
body content that happens to match a header/footer text
- Skips structural lines (headings, list items), short lines (<10 chars),
and decorative separators (repeated single characters)
- Frequency threshold: >= max(3, page_count * 30%) distinct pages
New strip_headers_footers option on MarkdownOptions (default: true).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove `process_mode` from `MarkdownOptions` (it controlled the
pipeline, not markdown formatting)
- Remove dead `text` field from `PdfProcessResult`
- Add `PdfOptions` builder consolidating mode, detection, markdown,
and page filter configuration
- Add convenience functions: `detect_pdf()`, `detect_pdf_mem()`,
`process_pdf_with_options()`, `process_pdf_mem_with_options()`
- Eliminate double document parsing: load once, share between
detection and extraction via internal `pub(crate)` helpers
- Deprecate old `process_pdf_with_config*` functions (kept as shims)
- Update binaries to use new API
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The R6 encryption fix has been merged upstream, so switch from the
fork back to the canonical repository.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
EM SPACE from PDF ActualText entries was lost by text.trim(), breaking
spacing after bullets and numbered list markers. Normalizing to ASCII
space lets should_join_items detect word boundaries naturally. NBSP
(U+00A0) is excluded as it's handled by coordinate-based spacing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Soft hyphens, zero-width spaces, BOM, ZWNJ/ZWJ, and word joiners now get
stripped in expand_ligatures(). Also call expand_ligatures() on the ActualText
code path which was the only TextItem creation site missing it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use abimaelmartell/lopdf firecrawl/fix-r6-encryption branch which
accepts /O and /U values longer than 48 bytes in V5/R6 encrypted PDFs.
This unblocks 4 encrypted PDFs in the bench suite that were previously
rejected with InvalidHashLength.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Map lopdf's Unimplemented("encrypted...") error to PdfError::Encrypted
instead of falling through to PdfError::Parse. Retry all Document::load
calls with an empty password when encryption is detected, so
owner-password-only PDFs can be opened. If the retry also fails, the
user now sees a clear "PDF is encrypted" message.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unit test verifying short sub-header rows (e.g. month names) are not
merged as continuation rows in table formatting. Snapshot test for
2013_app2.pdf pinning the full row-stripe table output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When PDF tables use full-width alternating row shading (row-stripe rects),
the normal rect grid detection fails because all rects share the same X/width,
collapsing to ~1 column. Add a fallback that uses rect Y-edges for rows and
text X-position clustering for columns, with a lower 15pt threshold to
separate narrow columns like row numbers and dates.
Also fix continuation-row merging in table formatting to not merge short
single-cell rows (≤5 chars) that are section sub-headers (e.g. month names).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The chest-wall PDF uses $ as separator but also has trailing $ after
spaces, giving only 13.7% letter-dollar-letter ratio (594 of 4332).
Add absolute count threshold (>20) alongside the ratio check so both
concentrated and dispersed substitution patterns are caught.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add has_encoding_issues field to PdfProcessResult that detects garbled
text from broken ToUnicode CMaps (U+FFFD replacement characters or
systematic dollar-as-space substitution). Surfaced in JSON output so
clients can fall back to OCR for affected PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Symbol/Wingdings fonts often map codes to PUA (U+F000-F0FF) via ToUnicode
CMaps, producing invisible/tofu characters in markdown output. Add
clean_symbol_pua() post-processing to extract_text_from_operand() that
converts these to standard Unicode: bullets to U+2022, checkmark to U+2713,
and ASCII/Latin-1 range by stripping the F000 offset.
Eliminates 1,405 PUA characters across 25+ eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add pdf.js binary CMap (bcmap) parser for Japan1/GB1/CNS1 CID fonts
- Support inline ToUnicode streams (not just object references)
- Add CMapDecisionCache for heuristic primary vs remapped CMap selection
- Build fallback CMaps from embedded font data and CIDSystemInfo
- Handle simple fonts without ToUnicode via embedded font cmap
- Add Symbol/Wingdings/ZapfDingbats font decoding fallback
- Support usecmap chaining in both text and binary CMap formats
- Scan XObject Forms for text operators in detector
- Change default detection strategy from EarlyExit to Sample(8)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Some PDF generators subset-embed fonts by renumbering GIDs sequentially
but fail to update the ToUnicode CMap, which still references original
GID values. Detect this mismatch (Identity-H, no CIDToGIDMap, W array
starts at CID ≤ 2 but CMap min CID > 2) and remap to sequential positions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hand-rolled JSON escaping only handled \, ", and \n but missed
tabs, carriage returns, and other control characters (U+0000..U+001F),
producing invalid JSON that Python's json.loads would reject. Add a
proper json_escape() function covering all required escapes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces a ProcessMode enum that controls how far the PDF pipeline
runs, enabling fast document triage without paying extraction or
markdown conversion costs. Exposed via --detect-only and --analyze
flags in both pdf2md and detect-pdf CLIs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add LayoutComplexity struct to PdfProcessResult so callers can detect
when a PDF has complex layout (tables or multi-column text) and decide
whether to use the extracted markdown or fall back to OCR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a PDF page has a graph directly above a table (e.g. Figure 2 above
Table II in real-estate-pricing.pdf), the heuristic table detector would
merge graph axis labels and legend items into the table, producing a
corrupted result.
Add rect hint regions: when a small cluster of cell-border rects (4-6)
fails full grid validation (e.g. only row borders, no column dividers),
extract their Y bounding box as a "hint region". The heuristic detector
then runs separately on items inside vs outside hint regions, preventing
unrelated content from being merged into tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rects spanning multiple grid rows (e.g. classification labels) now have
their text consolidated into the first sub-row via propagate_merged_cells().
Also prevent continuation-row merge from folding body data into the header.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 5 stripped public-domain PDF fixtures and golden markdown snapshots
for CI regression testing. Fix non-deterministic output caused by
HashMap iteration order in font stats, table heuristics, and rect
clustering by adding deterministic tie-breaking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The snap_edges tolerance of 2.0 was too tight for PDFs where header and
body cell boundaries differ by ~3pt due to cell padding. This created
phantom empty columns that caused valid tables to be rejected. Increase
snap and cell coverage tolerances from 2.0/3.0 to 6.0. Also add debug
logging to rect-based table detection pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge CMap and Differences encoding at the byte level for single-byte
fonts, fixing garbled output when partial ToUnicode CMaps caused Latin-1
fallback to block the Differences path. Add Adobe-Korea1 CID-to-Unicode
predefined mapping for Identity-H CID fonts without ToUnicode streams,
and support TrueType cmap extraction via ttf-parser for embedded fonts.
Also handle suffixed glyph names (e.g. zero.tf, a.ss01) per Adobe spec.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates double-parsing of PDFs by using the lopdf document model exclusively
for ToUnicode CMap extraction. The old raw byte scanner parsed PDFs separately
from lopdf and only handled FlateDecode, while lopdf handles FlateDecode + LZW +
ASCII85. The new approach walks page fonts and Form XObject fonts via the document
API, yielding ~7x speedup on text-heavy PDFs and ~1.2x overall.
- Add FontCMaps::from_doc() with recursive Form XObject font walking
- Remove ~270 lines of raw byte scanning code (from_pdf_bytes, extract_stream_from_raw_pdf, etc.)
- Remove flate2 dependency (lopdf handles decompression internally)
- Remove dead CMap lookup fallback branches (by_name, get_with_obj, base_font_name)
- Remove unused font_base_names parameter from extract_text_from_operand
- Skip U+FFFD replacement characters in CMap decode (PDF notdef glyph markers)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detect newspaper-style two-column layouts (dense independent text
flows) and emit columns sequentially instead of Y-interleaving them.
This fixes academic papers where column text was being merged across
columns at the same Y position.
Key changes:
- Add is_newspaper_layout() heuristic (≥15 lines/col, >50% Y-collision)
- Add split_column_stragglers() to separate core column clusters from
header remnants and per-word items that landed in column buckets
- Filter wide items (>60% page width) from column detection histogram
so full-width text doesn't fill the gutter and prevent detection
- Emit: above spanning → core columns sequentially → column stragglers
sequentially → below spanning items
- Fix markdown paragraph break to detect backward Y jumps (y_gap.abs())
so sequential columns on the same page get proper paragraph breaks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detect RTL scripts (Hebrew, Arabic, Syriac, etc.) by Unicode ranges and
sort line items by X descending when a line is predominantly RTL. Fix
gap calculation in should_join_items() to be direction-neutral. Extract
WMode from Type0 font dictionaries for vertical text foundation. Add
missing Korean Hangul ranges to is_cjk_char().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like privacy notices have 6-7 distinct visual table sections per
page, each with its own set of cell rects. Previously all rects were
merged into a single giant sparse grid that failed validation. Now
spatially connected rects are clustered via union-find before running
grid detection independently on each cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add U+FB05/FB06 (ſt/st→st) to ligature table matching PyMuPDF's full set
- Refactor expand_ligatures to single-pass match loop (clippy fix)
- Detect dot-leader lines (TOC entries) and prevent list continuation
merging, which was joining sub-entries onto parent numbered items
- Force newline breaks between consecutive dot-leader lines in paragraphs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace section-based column merge (which emitted all left-column lines
then all right-column lines) with Y-interleaved merge that combines
lines at the same Y position from different columns into a single line.
This fixes tabular layouts (like court dockets) where rows span across
columns — previously hearing descriptions were detached from their
entries and line breaks between rows were lost.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fillable PDF forms store typed values in the AcroForm dictionary, not in
the page content stream. This adds extraction of form field names and
values, emitting them as TextItems positioned at the field's Rect so they
flow naturally into the markdown pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
effective_font_size() was computed from the text matrix alone, ignoring
CTM scaling. In PDFs with a small CTM scale (e.g. 0.24×) and large Tm
values (e.g. 58), font_size was inflated (58pt instead of ~14pt), causing
the merge threshold to bridge inter-column gaps and merge two-column
text into single lines. Now compute the combined matrix (text_matrix × CTM)
before calling effective_font_size() at all 6 call sites.
Also adds PdfRect extraction from `re` operators for future table-grid
detection, and related plumbing changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Emit ActualText items at EMC (end of marked content) instead of at the
first Tj/TJ inside the BDC region. This computes the width from the
text matrix delta across the entire BDC..EMC span, so ligature
replacements like "fi" get proper widths and merge correctly with
adjacent text items (e.g., "Defi nitions" → "Definitions").
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Handle text rendering mode (Tr=3) to skip invisible OCR overlay text
- Support BDC/EMC marked content with ActualText for tagged PDFs
- Merge adjacent single-char TextItems into words at extraction layer
- Skip image XObjects instead of emitting [Image: ...] placeholders
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDFs like SEC filings render each glyph as a separate text operator with
intra-word gaps ≈ 0 and word gaps ≈ 0.15× font_size. The previous default
threshold (0.15) was exactly at the boundary, causing words to merge
(e.g. "ofOperations", "endedJune", "furnishedas").
Reduce single-char-to-single-char alphabetic threshold from 0.15 to 0.10,
cleanly separating word boundaries from intra-word kerning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --pages flag to insert <!-- Page N --> markers between pages
- Add --select-pages flag to process only specific pages (e.g. 1,3,5-10)
- Wire MarkdownOptions and page filter through process_pdf_with_config
- Fix fuzzy CMap matching that caused Cyrillic substitution on Latin text
- Fix whitespace Tj items not advancing text matrix (broke gap detection)
- Tighten single-char fragment join threshold from 0.25 to 0.20
- Gitignore debug/diagnostic binaries and remove tracked ones
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the blanket multi-column page guard (which blocked ALL table
detection on two-column pages) with content-based validation that
rejects paragraph text falsely detected as tables:
- Detect word-break hyphens at cell boundaries (paragraph signal)
- Reject high empty-cell ratio with many rows (sparse grid = text)
- Detect letter-spaced text (wide character spacing, not data)
- Reject long sentence fragments (avg cell >40 chars + many >60)
- Recover body-font header rows for small-font tables
- Add X-range filtering to body-font table region detection
olmOCR-bench: table_tests 16.4%→18.7% (+24), overall 31.6%→32.2%
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TJ arrays that span multiple table columns are now split into separate
TextItems at gaps >4x the space threshold, enabling the table detector
to assign text to correct columns. Also relaxes find_first_table_row()
to only skip rows where the majority of cells look form-like, preserving
real table headers that happen to contain colons.
olmOCR-bench table_tests: 10.7% → 16.4% (+53% relative, +58 tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Type3 fonts with FontMatrix [1,0,0,-1,0,0] (units_scale=1.0) had
broken word spacing: intra-word kerns of -4000 exceeded the clamped
threshold of 200, producing "Op en-Domain", "comp etition", etc.
Two fixes:
- Compute space_width fallback after units_scale is known; for
non-standard scales, estimate from average glyph width (~45%)
instead of the hardcoded 250 (calibrated for 1/1000-unit fonts)
- Remove upper clamp on space threshold (200 was too low for fonts
where TJ values naturally range into thousands)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace hardcoded max_pages_to_sample with a ScanStrategy enum that
supports EarlyExit (default), Full, Sample(n), and Pages(vec) modes.
Add process_pdf_with_config and process_pdf_mem_with_config to the
public API. Update README with usage examples and strategy docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change max_pages_to_sample from 5 to u32::MAX so every page is analyzed.
This prevents misclassifying Mixed PDFs as TextBased when scanned pages
fall outside the old 5-page sample window.
Add early-exit: stop scanning as soon as a non-text page is found, since
the PDF can't be purely TextBased. A 492-page mixed PDF exits after 2
pages instead of scanning all 492.
Also add title and confidence fields to PdfProcessResult for downstream
consumers (NAPI wrapper, feature-flag gating).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
