Move detailed Python, Rust, and debugging docs into docs/ to keep
the main README focused on overview and quick start examples.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Drop platform-specific optional deps — ship all .node binaries in one
package (~4 MB total). Simplifies publishing and consumer install.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skip napi prepublish/artifacts commands that require GitHub API auth.
Instead, create platform package.json files and copy binaries directly.
Add optionalDependencies to main package so npm/bun auto-selects the
right binary.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Both bindings now expose the same 6 function families: process, detect,
classify, extractText, extractTextWithPositions, and extractTextInRegions.
Bumps PyO3 from 0.22 to 0.25 for Python 3.14 support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the napi bridge from fire-pdf into pdf-inspector as `napi/`.
Package name: @firecrawl/pdf-inspector-js, published to GitHub Packages
(npm.pkg.github.com) as a public package on v* tags.
Exposes two functions:
- classifyPdf(buffer) → type, page count, pages needing OCR
- extractTextInRegions(buffer, pageRegions) → per-region text with
needsOcr quality flag (GID fonts, garbage, encoding issues)
Includes publish workflow that builds linux-x64-gnu + darwin-arm64
binaries and publishes main + platform-specific packages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Return RegionText with a needs_ocr flag per region, set when:
- extracted text is empty (image region with no PDF text)
- page uses GID-encoded fonts (unreliable CID mapping)
- text fails garbage detection (mostly non-alphanumeric)
- text has encoding issues (U+FFFD, dollar-as-space patterns)
This lets callers skip GPU OCR only when text quality is reliable,
falling back for any region where extraction is suspect.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add `extract_text_in_regions_mem` that takes layout-detected bounding
boxes (top-left origin, PDF points) and returns text within each region
in reading order. Designed for pipelines where a layout model detects
regions and text-based pages can skip GPU OCR by extracting text from
the PDF structure directly.
Also add `classify_pdf_mem` for lightweight PDF type classification
returning 0-indexed pages_needing_ocr.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(extractor): strip PDF comments that break lopdf content stream parsing
Some PDF generators (notably PD4ML used by school districts) embed
comments (% to end of line) in content streams. lopdf's Content::decode
parser fails to parse operators that follow comments, silently dropping
ET (end text) and Q (restore graphics state) operators. This caused
entire pages to produce 0 text items despite having valid text.
Fix: pre-process content streams to strip comments before parsing.
Comments inside string literals (parentheses) and hex strings are
preserved. The comment is replaced with a space to maintain token
separation.
Impact: fixes 13+ school district PDFs and similar PD4ML-generated
documents that were producing near-empty output (454 → 31,955 chars
for a 22-page school improvement plan).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(detect): flag sparse-extraction pages as needing OCR
When a TEXT-BASED PDF produces <50 chars/page average with <500 total
chars, flag all pages as needing OCR. This catches PDFs where the
extractable text is minimal (form templates, image-heavy layouts)
and the bulk of content requires OCR to access.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(detect): improve CID mojibake detection for Japanese/CJK PDFs
Extend is_cid_garbage to detect CID-as-Latin-1 mojibake: when ≥40%
of characters are high Latin-1 (U+00A0-00FF) and <33% are ASCII
letters, the text is likely CID values misinterpreted as Latin-1
characters (common in Japanese/CJK PDFs with broken ToUnicode CMaps).
Also add sparse-extraction OCR flagging: TEXT-BASED PDFs with
<50 chars/page and <500 total chars get all pages flagged for OCR.
Impact: Softbank Japanese PDFs now produce empty output with
pages_needing_ocr=all instead of mojibake garbage. Korean PDFs
with valid extraction (nexo-price-en) remain unaffected.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(detect): sparse extraction check only when markdown is generated
The sparse-extraction OCR check was triggering in Analyze mode where
markdown is not generated (md_len=0), causing false OCR flags on
every PDF processed via detect-pdf --analyze.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): improve heuristic detection for borderless wrapped-cell tables
Three changes to the body-font heuristic detector:
1. Adaptive Y-gap in find_table_regions_strict: use median qualifying-row
spacing × 3 instead of fixed 25pt. Tables with wrapped cells have
larger gaps between qualifying rows (those with 3+ X-clusters).
2. Y-only region filtering: use full X range when collecting region items.
The strict X bounds from qualifying rows excluded continuation lines
in wrapped cells, starving find_column_boundaries of items.
3. Merged-band retry: when split_side_by_side splits a page into bands
but no band produces a table, retry heuristic detection on all items
merged as a single band.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): gap-histogram column detection for small tables + lower avg_cells
Two changes to fix PDF 045 (borderless table with narrow "No." column):
1. Extend gap-histogram column threshold to small tables: when the gap
between within-column jitter and between-column spacing is >10pt
(unambiguous bimodal signal), use the detected threshold even with
fewer than 500 items. Previously only triggered for dense tables.
2. Lower BodyFont avg_cells_per_row minimum from 2.5 to 2.0 to handle
tables with wrapped multi-line cells where continuation lines have
only 1 filled cell.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): trim empty outer columns + relax partial H-line validation
- Rect detection: trim empty first/last columns instead of rejecting
the whole table. Rect edges often extend beyond text boundaries.
- Line detection: accept tables with 6+ partial horizontal lines
(>15% width) when <3 full-spanning lines exist. Handles tables
with column-level separators instead of full-width rules.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): cell-rect fallback for tables with variable-width backgrounds
When rect clustering produces a grid that fails validation (empty
interior columns from variable-width cell backgrounds), fall through
to a new strategy: use rect Y-edges for row boundaries and text
X-position clustering for columns. This handles tables like the
opendataloader-bench 088-090 comparison tables where each cell has
its own background rect at different widths.
Also widen failed-cluster hint width cap for large clusters (≥30 rects)
to allow page-spanning table regions.
TEDS score on opendataloader-bench: 0.300 → 0.353.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): relax vertical line spanning validation for partial borders
Accept tables with 4+ partial vertical lines (>10% table height) when
fewer than 2 span >30%. Handles tables like opendataloader-bench 053
with column-level vertical separators that don't extend the full height.
TEDS: 0.353 → 0.377 on opendataloader-bench.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): lower cell-rect density threshold + add validation logging
- Lower cell-rect density minimum from 25% to 15% to accept sparser
tables with decorative backgrounds (fixes 147).
- Add debug logging to all heuristic validation paths for diagnosability.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): relax body-font validations for text-only and 2-column tables
Four fixes closing 71% of the TEDS gap vs opendataloader:
1. Validation 7 (table-like content): bypass numeric content requirement
for tables with 3+ columns that passed all structural checks. Text-only
tables (category lists, program descriptions) are legitimate.
2. Qualifying row threshold: lower from 3+ to 2+ X-clusters per row.
Enables 2-column body-font table detection (fixes 166).
3. Row-stripe max cell length: raise from 500 to 2000 for 3+ column
tables. Tables with paragraph descriptions in one column are valid
(fixes 121).
4. Row-stripe empty-column trimming: apply the same outer-column trim
as grid detection (fixes 121 column-0 rejection).
TEDS: 0.377 → 0.438 on opendataloader-bench (gap: -0.056 vs odl).
TEDS=0 docs: 14 → 9.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): enable 2-column body-font tables + lower all minimums
- Lower BodyFont minimum columns from 3 to 2 in detect_table_in_region
- Lower BodyFont minimum rows from 3 to 2
- Lower avg_cells_per_row minimum from 2.0 to 1.5 (handles wrapped cells
in 2-column tables)
- Apply empty-outer-column trimming to row-stripe detection (not just grid)
TEDS: 0.438 → 0.468 on opendataloader-bench (gap: -0.027 vs odl).
TEDS=0 docs: 9 → 8. 86% of original gap closed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): text-based row fallback + fix 120 flow-chart and 188 leaderboard
Three changes that push TEDS past opendataloader:
1. Cell-rect Y-edge fallback: when rects have too few Y-edges for row
structure, derive rows from text Y-position clustering within the
rect bounding box. Fixes flow-chart tables (120) and column-header-
only rects (188).
2. Lower cell-rect minimum from 20 to 6 rects to catch smaller tables.
3. Relax validation 1 (first-column presence) from 50% to 25% of rows.
Tables with wrapped model names have continuation lines without first
column content.
TEDS: 0.468 → 0.508 on opendataloader-bench.
Now BEATS opendataloader (0.508 vs 0.494, gap=+0.014).
TEDS=0 docs: 8 → 5.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): wrapped-cell continuation row merging
Merge rows that have fewer filled cells than the header row into the
previous row. Handles wrapped multi-line cells where text overflow
creates extra rows (e.g., "Direct" + "communications" → "Direct
communications").
Conditions: fewer filled cells than header, more than previous row had,
not a data row (numeric), not a short subheader label.
TEDS: 0.508 → 0.522 on opendataloader-bench (now +0.028 vs odl).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): tighten cell-rect validation + fix continuation-row merging
Cell-rect false positives:
- Raise density threshold back to 25% (from 15%)
- Add max cell length check (500 chars) to reject paragraph content
- Reject disproportionate grids (>20 rows, <4 cols)
Continuation-row merging:
- Wide tables (5+ cols): only merge rows with ≤50% header cells
- Narrow tables (2-4 cols): merge rows with fewer cells than header
- Prevents merging normal data rows in large tables (6_KE_Chart)
while keeping wrapped-cell merging for narrow tables (178)
TEDS: 0.498 on opendataloader-bench (still +0.004 vs odl).
pdf-evals: 191/192 passed, 0 regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Struct-tree tables with incomplete page tagging (e.g., only 22 of 50+
rows tagged on a page) would claim items and block rect detection,
leaving unclaimed items as loose text. Now require struct-tree tables
to capture ≥50% of band items before using them; incomplete trees
fall through to geometry-based detection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skip rect-detected tables that overlap with items already claimed by
struct-tree detection. Previously both strategies emitted separate
tables for the same content, doubling the output on tagged PDFs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a PDF has a well-formed structure tree with /Table > /TR > /TD|TH
elements linked to MCIDs, build tables directly from the semantic
hierarchy. Runs as highest-priority detection (step 0) before rect-based,
line-based, and heuristic strategies.
- Add StructTree::extract_tables() to walk the tree and collect table
descriptors with row/cell/MCID info
- Add detect_tables_from_struct_tree() to match MCIDs to TextItems
- Reject tables with <30% MCID cell coverage (stale structure trees)
- Update 2013-app2 snapshot (struct-tree gives valid but different
column ordering)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Origin-anchored full-page rects (x<5, y<5, h>20× median) are clipping
paths or page fills that bridge separate table regions into one cluster,
corrupting row-stripe detection. Exclude them from union-find adjacency
while keeping them available for hint generation and fallback paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Newsletter-style PDFs have decorative background rects (sidebar,
header, section bands) that pass row-stripe detection as false tables.
Reject when any cell exceeds 500 chars — real alternating-row data
tables have short cell content; layout backgrounds produce paragraph-
length "cells".
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(layout): relative valley column detection for justified text
Add fallback column detection using relative valley analysis for PDFs with
justified text where item widths extend past gutter boundaries. The absolute
valley detector fails on these layouts because gutter bins are at ~40% of
peak (well above the 15% noise threshold).
The relative valley detector smooths the histogram with a 5-bin moving
average, finds local minima where contrast < 0.60 of surrounding peaks,
and validates with peak balance >= 0.40. Limited to single best valley
(max 2 columns) and requires >= 100 items per page.
Tested on IRS Publication 17 (2002), a 289-page 2-column justified text
document: column detection went from ~40 pages to 165 pages.
190 passed, 0 regressions across 191 eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(layout): tighten relative valley thresholds to reduce false positives
Reduce PEAK_WINDOW from 40 to 25 bins (50pt) so valleys are only validated
against nearby peaks, not distant ones. Add MIN_PEAK_HEIGHT of 20 (smoothed)
to reject sparse pages where histogram peaks are too low to indicate dense
two-column text.
Previous thresholds caused 13 regressions across the eval suite by splitting
tables, TOCs, checklists, and forms. Now: 188 passed, 0 regressions (2 minor
metadata-only diffs on IRS P17 and 9978293).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(layout): skip relative valley detection on pages with tables
Table column gaps in the histogram look identical to text column gutters
but the table pipeline already handles reading order for those pages.
Pass page_has_table flag through detect_columns to suppress the relative
valley fallback on pages where tables were detected.
This eliminates all remaining regressions from relative valley detection:
190 passed, 0 regressions across 191 eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(layout): prose density validation for relative valley detection
Add columns_have_prose() to validate relative valley column splits.
Checks that both sides of a proposed split contain paragraph-like
content (fill ratio >= 40%, avg items/line <= 3.5) before committing
to a column split. Combined with the table-page guard, this prevents
false column splits on financial statements, forms, and tabular
layouts where long labels or dot leaders fill the column width.
Also tightens find_relative_valleys() thresholds (PEAK_WINDOW 40->25,
MIN_PEAK_HEIGHT 5->20) to reduce false positive valley candidates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Dense multi-column newspapers (WSJ, NYT) have extractable text but
produce poor output due to complex interleaved article layouts that
defeat column-based reading order. Detect these by counting Tf (font
change) operators alongside existing Tj/TJ counts, then flagging
TextBased PDFs where most sampled pages show high text density
(>=1500 ops), moderate font switches (>=50 Tf), and a low Tf/Tj
ratio (<0.15) — the ratio distinguishes newspapers from richly-styled
legal/business docs that have high Tf counts due to per-character
styling.
Calibrated against 108 TextBased PDFs in pdf-evals with zero false
positives. WSJ 50-page newspaper correctly flagged (5/8 sampled pages
match).
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The is_table_of_contents heuristic falsely rejected wide statistical
tables (e.g. ERP appendix tables) where the first column has year
labels with dot leaders ("1973..........") and other columns have
small numeric values. Add column-aware analysis: if dots are confined
to ≤1 column and ≥3 columns contain numbers, it's a data table.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: skip page extraction when operation count exceeds 1M
Vector-heavy architectural PDFs can have 10-26M path operators per page,
causing ~60 GB allocation during per-op processing. The decode itself is
tolerable but the subsequent loop amplifies memory 2-5x with text items,
rects, paths, and state tracking.
Check operation count after Content::decode() and return empty extraction
for pages exceeding the limit, with a warning log.
* test: add unit test for excessive operations guard
Constructs a synthetic PDF with 1.1M path operators to verify
that pages exceeding the operation limit return empty extraction.
* fix(text): handle Tc/Tw character and word spacing in text width computation
PDFs using Tc (character spacing) and Tw (word spacing) operators for text
justification had words incorrectly split across TextItems. The computed
advance width didn't account for these spacing parameters, causing spurious
spaces mid-word (e.g. "deve lopers" instead of "developers").
- Add Tc/Tw operator handling and graphics state save/restore
- Incorporate char_spacing and word_spacing into compute_string_width_ts
- Add adaptive merge threshold: tighter for lowercase→lowercase junctions,
wider before joining punctuation
- Add unit tests for Tc/Tw width computation and merge behavior
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(fonts): add large Tc width computation test
Verifies that large character spacing values are applied in full
without any artificial cap, matching PDF spec behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(text): guard against Tc/Tw-inflated widths in merge and join paths
Two targeted fixes to prevent character-spacing (Tc) and word-spacing
(Tw) inflation from causing data quality regressions:
1. should_join_items: reject large negative gaps (< -font_size) that
arise when Tc/Tw inflate item widths past adjacent items. Fixes
FY_2015 merged numbers (e.g. "239.696.0" → "239.69 6.0").
2. merge_text_items: cap effective width for gap computation when Tw
inflates space-containing items beyond 0.85× font_size per char.
Prevents column-level gaps from collapsing into merge range,
recovering table detection for Baldwin-Edwards and similar PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
PDFs that embed landscape content in portrait pages via a rotated text
matrix (e.g. [0, b, -b, 0, tx, ty] for 90° CCW) produced garbled output
because the layout engine assumed x=horizontal, y=vertical.
Track the dominant text direction from combined matrices during extraction.
When ≥67% of text operators are rotated, swap x↔y coordinates (with
y-negation for correct reading order) for all text items, rects, and lines.
Also estimate text widths from char count × font size since scale_x ≈ 0
for rotated text.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When Identity-H or Type3 fonts lack a ToUnicode CMap and the
CID-as-Unicode passthrough doesn't produce valid text, the raw CID
byte values appear as mojibake (random Latin Extended characters mixed
with C1 control codes).
The detector already flags these pages in pages_needing_ocr, but the
markdown pipeline still emitted the garbage. Now, for TextBased PDFs,
we check each OCR-flagged page's extracted text for CID garbage
(C1 control characters U+0080–U+009F at ≥5% density) and strip items
from pages that fail the check.
This is scoped to TextBased PDFs only — Mixed PDFs flag pages for OCR
due to template images, not font encoding issues.
Adds test fixture (shinagawa_identity_h.pdf) and integration test.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(fonts): CID-as-Unicode passthrough and subscript/superscript merging
Add smart CID-as-Unicode passthrough for Identity-H fonts without
ToUnicode maps. Uses /W array median CID heuristic to distinguish
Unicode-CID PDFs (Chromium-generated) from GID-based subsets.
Add merge_subscript_items() pass that merges small-font items (<75%
of dominant font size, ≤4 chars, tightly adjacent) into parent items.
Fixes chemical formulas (NH3, H2O, KClO3), footnote references, and
subscript notation (vf, Hfg, m3/kg) that were previously orphaned
as separate text items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(subscripts): restrict merge to purely numeric text only
Tighten subscript merging to only merge items containing ASCII digits
(0-9). This avoids false positives with ordinal indicators (º), letter
subscripts (sol, vf), and small bullet characters (▶) that caused
table restructuring regressions.
Numeric-only keeps the primary wins: chemical formulas (NH3, H2O),
footnote references, and unit notation (m2, m3).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(subscripts): restrict merge to parent text ending with a letter
Only merge numeric subscripts when the parent item's text ends with
an alphabetic character. Prevents false merges like "33" + "1" in
fractions (33 1/3%), table credit numbers after spaces, and footnote
refs after punctuation (land.1 → land. 1). Chemical formulas (NH3,
H2O, KClO3) still merge correctly since parent ends with a letter.
Reduces pdf-eval regressions from 13 to 2 (both are correct reversions
of over-aggressive footnote merging from the prior commit).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Points to firecrawl/lopdf@edf6279 which fixes decompressed_content()
for streams without a /Filter entry. Our xobjects.rs workaround remains
as a defensive fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lopdf's decompressed_content() fails on Form XObjects without a /Filter
entry (uncompressed streams). This caused all body text to be lost in PDFs
generated by pdfrw and similar tools that wrap page content in uncompressed
Form XObjects (e.g. Cambridge University Press excerpts).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lopdf's decompressed_content() fails with DictKey("Filter") when a
ToUnicode stream has no /Filter entry (uncompressed raw text). Now
falls back to the raw stream.content when decompression fails.
Fixes Identity-H fonts with valid uncompressed ToUnicode CMaps
producing empty text (e.g. neoenergia tabela PDF).
Type3 fonts render each glyph as a custom drawing/bitmap. Without a
ToUnicode CMap the character codes can't be mapped to Unicode, so
extracted text is garbage. Pages using only Type3 fonts are now
excluded from pages_with_text and added to pages_needing_ocr.
Fixes Korean/CJK PDFs using Type3 fonts (e.g. D2Coding) being
classified as TextBased when no usable text can be extracted.
Document titles, column headers, and newsletter mastheads that repeat
on every page were being fully stripped by strip_repeated_lines. Now
the first page's occurrence is preserved so the content appears once.
Fixes missing titles like "VOICE OF SOUTH MARION" in tax certificate
PDFs and column headers in IRS forms.
When every page uses fonts with unresolvable gid-encoded glyphs,
the extracted text is unreliable garbage. Suppress the markdown
output so consumers know to use OCR instead.
Only triggers for gid-encoded fonts specifically, not for other
OCR signals (Identity-H without ToUnicode, template images) where
the text layer may still be partially useful.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(detect): flag Identity-H fonts without ToUnicode for OCR
Cyrillic (and other non-Latin) PDFs with Type0/Identity-H encoded fonts
and no ToUnicode CMap produce garbage text from direct extraction. Two
fixes:
1. Detector: new `page_has_identity_h_no_tounicode` check adds affected
pages to `pages_needing_ocr` regardless of PDF classification.
2. Extraction: extend garbage-text safety net to TextBased PDFs — when
extracted text is <50% alphanumeric, drop the markdown, set
`has_encoding_issues`, and flag all pages for OCR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: replace integration tests with synthetic unit tests
Remove PDF fixture dependencies from detector and lib tests. Use
in-memory lopdf documents to test Identity-H/ToUnicode detection logic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When rect-based, line-based, and heuristic detection all fail to find
tables on a page, try building a table directly from the layout
engine's column boundaries. Handles borderless tabular layouts like
exam/reference grids where columns are defined purely by text
alignment.
Includes header-row column refinement: when a detected column
contains multiple header items, it gets split at the gap between
them to recover the correct number of columns.
Guards against false positives: requires ≥4 columns, ≤40 rows,
>50% multi-column rows, short cells (avg ≤40 chars), no prose
content, no dominant single column, and no structural elements
(≥6 rects or ≥4 lines) on the page.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fonts using raw glyph ID names (gidNNNNN) in their Differences
encoding cannot be decoded to Unicode without the original font's
cmap table. Detect this pattern during font parsing and add
affected pages to pages_needing_ocr so downstream consumers
know to use OCR instead.
Fixes text extraction on PDFs like Tezukuri_Food-Menu.pdf where
the main body font (AcuminVariableConcept) uses gid-encoded
glyphs — even PyMuPDF and ODL fail on these.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pages with tens of thousands of vector-drawing rects (e.g. architectural
plans with 36k+ rects) caused cluster_rects and sub-rect deduplication to
spend 10-50s in O(n²) loops. No real table has thousands of cell rects,
so once a union-find component exceeds 2000 elements we skip further
comparisons, and skip the O(n²) sub-rect dedup entirely for such pages.
Benchmarks on 8 slow production PDFs show 10-40x speedups on vector-heavy
files (12s→0.3s) with no regression on other file types. All 342 tests
pass.
Two changes to improve balance sheet / financial table detection:
1. split_side_by_side: Don't split when one side is text labels
and the other is numeric data at matching Y positions. This
prevents splitting a single label+number table into two
independent regions.
2. try_add_label_column: After detecting a numeric-only table,
look for unclaimed text items to the left at matching Y
positions and prepend them as column 0 (row labels).
Tested on IN_Annual_Report_2017 balance sheet which now produces
proper 3-column tables (Label|2016|2017) instead of separated
number tables and paragraph text.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
