* add extract_pages_markdown_mem for per-page markdown extraction
Enables hybrid OCR pipelines to skip GPU render+layout for simple text
pages by providing per-page markdown with needs_ocr flags. Font stats
are computed document-wide for consistent header detection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* bump napi package version to 0.6.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace stringly-typed pdf_type and item_type fields with
#[napi(string_enum)] enums for proper TypeScript type checking.
Add link_url field to TextItem instead of encoding URL in the
item_type string.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two changes that reduce false needsOcr rejections without hurting quality:
1. Per-region GID check instead of per-page blanket rejection.
Previously, if ANY font on the page used GID-encoded glyphs (common
in logos, decorative fonts), ALL table and text regions on that page
were forced to GPU OCR via needsOcr=true. Now the page-level bail is
removed; per-region text quality checks (is_garbage_text, is_cid_garbage,
detect_encoding_issues) catch actual GID corruption in the extracted
content. Tables whose text is clean pass through even if an unrelated
font elsewhere on the page is GID-encoded.
2. Relaxed looks_like_partial_table for layout-assisted extraction.
When the layout model already identified a region as a table (i.e.,
extract_tables_in_regions_mem), boundary-detection heuristics are
less necessary — we're not guessing "is this a table?" anymore, only
"can we extract it correctly?". Relaxations:
- Numeric first header cell accepted (e.g., year "2024")
- 1 empty header cell allowed in 3+ column tables (merged headers)
- Sparse first data row threshold relaxed from 33% to 50%
Paragraph detection and duplicate-header checks remain strict.
Eval: 196/196 pass (full regression suite), 91/91 Rust tests pass
including 7 new layout-assisted validation tests. Zero regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a 5th failure-mode check to looks_like_partial_table: when the
heuristic mis-detects text-wrapped paragraph prose as a multi-column
table, cells in the same column tend to start with lowercase letters
or continuation punctuation (commas, closing quotes) — because they're
actually sentence fragments. Real tables almost never have most data
cells starting lowercase.
Trigger: ≥2 cols, ≥4 data rows, ≥60% of non-empty data cells start
with lowercase or continuation punctuation → return needs_ocr=true.
Caught in the eval as the next-largest failure mode after the 0.4.1 fix:
PDFs 088, 182, 090 — heuristic produced "tables" like:
|Approval is needed from the|Acquisitions of|
|Treasurer if the acquisition|residential and|
|constitutes a "significant|agricultural|
|action," including acquiring an|land by foreign|
Reading column 1 top-to-bottom: "Approval is needed from the Treasurer
if the acquisition constitutes a 'significant action,' including
acquiring an interest..." — a paragraph, not tabular data.
Tests: 2 new tests (the 088-style failure case + a real multi-word
table that must NOT be flagged). All 11 looks_like_partial_table tests
pass; 323 unit + 91 integration tests still green.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the heuristic returns markdown that looks like a partial / mis-detected
table, set needs_ocr=true so the caller falls back to GPU OCR. Previously the
same cases returned the broken table with needs_ocr=false, which produced
real-world TEDS=0 scores in fire-pdf evals (heuristic-built table didn't
match ground truth structure at all, but caller had no signal to fall back).
Four failure modes detected, all observed in opendataloader-bench eval losses:
1. **Header looks like a data row** — first cell of header is a bare number
(e.g. `|2|...`), suggesting the actual header row was skipped. Real
headers almost never start with just a number.
2. **Empty header cells in a multi-column table** — ≥3 cols, ≥1 empty cell
in the header row. Indicates poor column boundary detection.
3. **Duplicate header cells** — same non-empty value appearing twice in the
header (e.g. "Administration|Administration"). Means a multi-line header
was collapsed wrong.
4. **Sparse first data row** — ≥3 cols and ≥1/3 of first-data-row cells are
empty. Multi-row headers in the source PDF get smashed into header +
sparse data row by the heuristic; this catches that.
Tests: 9 new unit tests in `looks_like_partial_table_tests` cover each
failure mode plus realistic non-failures (well-formed table, single-column
list, two-col with a single empty cell). All 91 existing tests still pass.
Bumps `napi/package.json` to 0.4.1 since this changes the function's return
behaviour for callers (some inputs that returned needs_ocr=false now return
true). The output text field is also cleared on the new fallback path so
callers don't accidentally use the broken markdown.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a new function that takes a PDF buffer and page+bbox regions (same interface
as extractTextInRegions), runs heuristic table detection on items within each region,
and returns markdown pipe-tables. Falls back to needs_ocr=true when no table
structure is found or text quality is suspect.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Some PDFs (e.g. British Academy grant guidance, Carter BloodCare privacy
policy) have structure trees that incorrectly tag body text as H2 headings.
This caused every line within numbered paragraphs to render as a separate
## heading instead of being joined into flowing paragraph text.
Added detect_overused_struct_heading_levels() which pre-scans heading tag
frequency and suppresses levels appearing on >15% of tagged lines, allowing
those lines to fall through to normal paragraph joining.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The detector flagged pages for OCR whenever any font was Identity-H
without ToUnicode, even when the extraction pipeline could decode the
font via fallback paths (CID-as-Unicode passthrough or embedded TrueType
cmap). This caused false positives on PDFs from Chromium, wkhtmltopdf,
and other generators that use Identity-H with Unicode CID values.
Now checks DescendantFonts W array and embedded font cmap before
flagging. Fonts that are genuinely undecodable (stripped cmap, low GID
CIDs) are still correctly flagged.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Do invokes any XObject (Form or Image), but scan_content_for_text_operators
was counting every Do as an image. PDFs with Form XObjects (e.g. ACS
publisher watermark pages) were misclassified as ImageBased because the
inflated image_count raised the min text ops threshold above the actual
text operator count.
Image detection is already correctly handled by scan_xobjects_in_resources
(checks Subtype) and analyze_page_images (measures pixel area).
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
In multi-column PDFs, column switches break paragraph continuity,
making body text lines appear "standalone". Combined with moderate
font-size rarity from minor size variation between columns, this
caused hundreds of false heading classifications (e.g. 281 false ##
headings on a single academic paper).
Non-bold, non-isolated lines now require very high rarity (≥0.97)
and short word count (≤8) to qualify as headings via the rarity path.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When fire-pdf sends pre-segmented bboxes from the layout model,
pdf-inspector no longer runs column detection, stream-order heuristics,
or newspaper/tabular mode detection within the region. These heuristics
conflict with the layout model's decisions and cause wrong reading order.
Region extraction now simply: Y-sorts items, groups into lines, and
sorts within each line by X position. The heavy heuristics remain
available for standalone full-page extraction.
Eval showed pure OCR (0.2875 NED) beating native+heuristics (0.2916)
across all categories, especially multi-column (-0.08) and newspaper
(-0.16). This change should close that gap.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pre-scan lines to identify "isolated" ones — short lines (1-6 words)
with paragraph breaks both before AND after. These are heading
candidates even at body font size, common in academic papers
("Acknowledgements", "Limitations", "B.3 Prompt Engineering").
Inspired by opendataloader's HeadingProcessor which passes prevNode
and nextNode context to the heading probability scorer.
The isolated signal (+0.3) combines with rarity/bold/standalone
signals. A per-page density guard prevents false positives on
multi-column pages where many lines appear isolated. Continuation
word detection (ending in "the", "and", etc.) filters wrapped
paragraph lines.
MHS=0 docs: 18→13. MHS-S +0.004. No regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the histogram-based column detector finds no valleys (common with
sidebar/asymmetric layouts), fall back to a simplified XY-cut: find the
largest horizontal gap between item edges and split there if both sides
have enough items with vertical overlap.
Inspired by opendataloader's XY-Cut++ algorithm but implemented as a
single-level fallback rather than full recursive segmentation.
Doc 156: NID 0.545→0.966, Doc 157: NID 0.564→0.962.
NID-S +0.007, TEDS-S +0.066 across 200 docs. No regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PDFs that draw table borders as thin filled rectangles (< 2pt, common in
spreadsheet exports) were invisible to both rect-based and line-based
table detectors. Now converts these thin rects to PdfLine objects and
runs line-based detection, but ONLY as a last resort after all other
methods (rect, line, heuristic, column-based) found nothing.
This avoids the regression from the earlier attempt which ran synthesis
at step 2, preempting the heuristic detector on PDFs where it worked
better.
Also relaxes uniform row spacing threshold (CV 0.05→0.02) to accept
spreadsheet-exported tables with even row heights.
Benchmark: TEDS 0.519→0.586 (+0.067), overall +0.006, no regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reverts commits 937311c, 1dcb0c6, 0300e96, 999f9a2. The thin-rect-to-line
synthesis and stacked table splitting improved extraction for specific
government PDFs but caused -0.05 TEDS regression on the benchmark by
preempting the heuristic detector with worse line-based grids.
These features need more targeted guards before re-enabling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rows that sit between horizontal rules but lack vertical border coverage
are not table cells — they're freestanding text (e.g. "Note: The cutoff
mark is out of 120"). These rows now split the grid into separate
sub-tables, with the unbounded text emitted as plain text between them.
Single-cell "tables" (from the split) render as plain text instead of
a degenerate 1x1 markdown table.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rows with 3+ short-valued cells (avg ≤10 chars) and an empty first cell
are column headers (e.g. "UR | SC | ST | OBC | EWS"), not text overflow
from the previous row. Prevents them from being merged into the
preceding section title row.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three fixes for better table extraction from spreadsheet-exported PDFs:
1. Convert thin filled rects (< 2pt) to PdfLine objects before line-based
table detection. Many PDFs draw table borders as narrow filled rectangles
instead of stroked paths — these were invisible to the line detector.
2. Relax uniform row spacing rejection (CV 0.05 → 0.02). Spreadsheet
exports have very even row heights that were being rejected as "chart
grids".
3. Fix continuation row merging: don't merge rows where the only non-first
cell content is a long label (section headers like "Category No. 03").
Don't merge first-cell-only rows with long text ("Note: ...").
Also adds multi-Y row splitting in line-based detection and column-aware
table detection skipping for multi-column pages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Many PDFs (especially spreadsheet exports) draw table borders as thin
filled rectangles (height/width < 2pt) instead of stroked paths. These
were invisible to our line-based table detector since only stroke
operations produced PdfLine objects.
Now synthesizes PdfLine from thin rects before line-based detection,
enabling table detection on border-drawn PDFs like government forms.
Also relaxes the uniform row spacing rejection threshold (CV 0.05→0.02)
to accept spreadsheet-exported tables with even row heights.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On pages where column detection finds 2+ columns, skip body-font
heuristic table detection in the merged-band retry path. This prevents
sidebar/two-column prose from being formatted as markdown tables.
The fix is targeted: per-band heuristic detection still runs (bands
are scoped to single columns), so real tables within columns are
still detected. Only the merged-band retry (which sees all items
across columns) is gated.
Also relaxes column validation to accept asymmetric layouts (sidebars)
where one side has fewer items, and tries center-based item assignment
before edge-based to improve column splitting for asymmetric layouts.
Benchmark: NID 0.865→0.869, NID-S 0.798→0.805, overall +0.002.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace ad-hoc bold/ratio heading checks with a unified scoring system
based on font size rarity. For each line, compute:
score = font_rarity * 0.5 + bold * 0.3 + standalone * 0.2
Font rarity measures how infrequently a font size appears across the
document — heading fonts are rare while body text is common. This
approach (from opendataloader's ModeWeightStatistics) naturally adapts
to each document's font distribution instead of relying on fixed
thresholds.
Guards: require font_size >= 0.95 * base_size (no small-font headings),
word_count >= 3, and standalone (paragraph break before).
Benchmark improvement: MHS 0.56→0.58, MHS-S 0.66→0.70, overall +0.003.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compare pdf-inspector against other direct text extraction engines
(no OCR/ML) on the opendataloader-bench corpus.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lines with font size 1.10-1.20x body text that are standalone and short
(1-8 words) are promoted to headings. This catches academic paper
headings where the font is only ~10% larger than body text, below the
previous 1.2x threshold.
Also syncs the simpler to_markdown_from_lines path to match the
table-aware path (removes stale colon exclusion).
Benchmark improvement: MHS 0.54→0.56, overall 0.761→0.766.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Caption detection was incorrectly classifying "Table of Contents" as a
caption because it starts with "Table ". Now "Table" and "Figure"
prefixes require a digit, parenthesis, or hash after them — matching
actual captions like "Table 1", "Figure 3.2" but not titles.
Also removes debug logging left from previous iteration.
Benchmark improvement: MHS 0.52→0.54, overall 0.757→0.761.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The colon exclusion was preventing legitimate headings like "Steps for
Using the Microscope:" and "Changing objectives:" from being detected.
The single edge case it was protecting (chart sub-headers) is less
impactful than the many headings it was blocking.
Benchmark improvement: MHS 0.51→0.52, MHS-S 0.61→0.62.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Lower minimum item count for body-font table candidates from 9 to 6,
allowing small 2-3 row tables to be detected.
- Allow 2-column body-font tables with short cells (avg ≤25 chars) to
bypass the "table-like content" validation. This catches text-only
definition/category tables (e.g., species lists) without false-positiving
on 2-column paragraph text (which has longer cells).
Benchmark improvement: TEDS 0.498→0.519, overall 0.750→0.754.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bold lines at body font size that are standalone (preceded by a paragraph
break) and have ≥3 words are promoted to headings. This catches the
common pattern in academic/technical PDFs where section headings use
bold text at the same size as body text.
Guards against false positives: minimum word count, colon-ending
exclusion (labels like "Table I:").
Benchmark improvement: MHS 0.37→0.50, overall 0.71→0.75.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align region filtering with rotated-page coordinate rewrites, switch region text assembly to the shared line-grouping pipeline, and retain edge-overlap text to avoid false empty regions that incorrectly trigger OCR fallback. Also make Python region inputs fail fast with clear ValueError messages for malformed boxes.
Made-with: Cursor
Two bugs in collect_text_in_region / extract_text_in_regions_mem:
1. The threshold-based sort comparator in collect_text_in_region was not
transitive, causing Rust's sort to panic on certain PDFs. Replaced with
strict total_cmp ordering — the line-grouping phase already handles
fuzzy Y matching via threshold.
2. The needs_ocr check was missing is_cid_garbage, so Identity-H fonts
with CID garbage (C1 control chars, high Latin mojibake) could pass
all quality checks and be served as real text with needs_ocr=false.
Also adds 7 integration tests for extract_text_in_regions_mem (previously
had zero coverage): basic extraction, Identity-H needs_ocr, multiple
regions, nonexistent page, empty region, invalid input, and a fast-vs-normal
comparison test across all text-based fixtures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
FontCMaps::from_doc can spend 4.5+ seconds decompressing and parsing
large embedded TrueType fonts for CID fonts with sparse ToUnicode
CMaps. For extract_text_in_regions (hybrid OCR pipeline), this is
unnecessary — fonts that can't be decoded cheaply will produce
empty/garbage text, triggering needs_ocr=true and GPU OCR fallback.
Changes:
- Add FontCMaps::from_doc_pages_fast() that skips TrueType font
fallback parsing (build_fallback_cmap_for_type0) and Identity-H/V
second pass entirely
- Add FontCMaps::from_doc_pages() for filtered page sets
- extract_text_in_regions_mem uses fast mode
- Restructure fallback chain: try cheap fallbacks first, only attempt
expensive TrueType parsing when needed and not in fast mode
Benchmark on nihms-1771367.pdf (19-page chemistry paper):
- FontCMaps fast: 201µs
- FontCMaps slow: 4.47s
- 22,000x speedup on font parsing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rust panics in NAPI modules abort the Node.js process with no chance
to report errors. This wraps every exported function in catch_unwind,
converting panics into JS Error exceptions that can be caught and
reported to Sentry.
Buffer data is extracted to Vec<u8> before the catch_unwind boundary
to satisfy UnwindSafe requirements.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
upload-artifact strips the common napi/ prefix, so files are at
artifacts/js-bindings/index.js not artifacts/js-bindings/napi/index.js.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
napi pre-publish expects a version-bump commit message convention.
Instead, upload index.js and index.d.ts generated by napi build
as artifacts and copy them into the publish step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The command is `napi pre-publish` (hyphenated), not `napi prepublish`,
and `--skip-gh-release` is not a valid flag.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
index.js and index.d.ts are generated by napi-rs. Generate them
at publish time via `napi prepublish` instead of checking them in.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix sort panics on NaN values from bogus PDF font metrics
Replace all `partial_cmp(...).unwrap_or(Ordering::Equal)` and bare
`partial_cmp(...).unwrap()` with `total_cmp()` across the codebase.
`partial_cmp` returns `None` for NaN, and mapping that to `Equal`
violates total ordering: `a == NaN` and `NaN == b` but `a != b`.
Rust 1.81+ detects this and panics in sort_by. `total_cmp` handles
NaN deterministically (sorts to end) and guarantees total ordering.
The critical crash was in `extract_text_in_regions` (lib.rs:478)
where PDFs with bogus font ascent/descent values produced NaN in
text item coordinates, causing process abort via NAPI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix missed partial_cmp in layout.rs and restore napi exports
- Convert two remaining b.y.partial_cmp(&a.y) calls to total_cmp
in group_single_column and column layout sorting
- Restore missing napi exports: detectPdf, extractText,
extractTextWithPositions, processPdf
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move detailed Python, Rust, and debugging docs into docs/ to keep
the main README focused on overview and quick start examples.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Drop platform-specific optional deps — ship all .node binaries in one
package (~4 MB total). Simplifies publishing and consumer install.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skip napi prepublish/artifacts commands that require GitHub API auth.
Instead, create platform package.json files and copy binaries directly.
Add optionalDependencies to main package so npm/bun auto-selects the
right binary.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Both bindings now expose the same 6 function families: process, detect,
classify, extractText, extractTextWithPositions, and extractTextInRegions.
Bumps PyO3 from 0.22 to 0.25 for Python 3.14 support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the napi bridge from fire-pdf into pdf-inspector as `napi/`.
Package name: @firecrawl/pdf-inspector-js, published to GitHub Packages
(npm.pkg.github.com) as a public package on v* tags.
Exposes two functions:
- classifyPdf(buffer) → type, page count, pages needing OCR
- extractTextInRegions(buffer, pageRegions) → per-region text with
needsOcr quality flag (GID fonts, garbage, encoding issues)
Includes publish workflow that builds linux-x64-gnu + darwin-arm64
binaries and publishes main + platform-specific packages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Return RegionText with a needs_ocr flag per region, set when:
- extracted text is empty (image region with no PDF text)
- page uses GID-encoded fonts (unreliable CID mapping)
- text fails garbage detection (mostly non-alphanumeric)
- text has encoding issues (U+FFFD, dollar-as-space patterns)
This lets callers skip GPU OCR only when text quality is reliable,
falling back for any region where extraction is suspect.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add `extract_text_in_regions_mem` that takes layout-detected bounding
boxes (top-left origin, PDF points) and returns text within each region
in reading order. Designed for pipelines where a layout model detects
regions and text-based pages can skip GPU OCR by extracting text from
the PDF structure directly.
Also add `classify_pdf_mem` for lightweight PDF type classification
returning 0-indexed pages_needing_ocr.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(extractor): strip PDF comments that break lopdf content stream parsing
Some PDF generators (notably PD4ML used by school districts) embed
comments (% to end of line) in content streams. lopdf's Content::decode
parser fails to parse operators that follow comments, silently dropping
ET (end text) and Q (restore graphics state) operators. This caused
entire pages to produce 0 text items despite having valid text.
Fix: pre-process content streams to strip comments before parsing.
Comments inside string literals (parentheses) and hex strings are
preserved. The comment is replaced with a space to maintain token
separation.
Impact: fixes 13+ school district PDFs and similar PD4ML-generated
documents that were producing near-empty output (454 → 31,955 chars
for a 22-page school improvement plan).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(detect): flag sparse-extraction pages as needing OCR
When a TEXT-BASED PDF produces <50 chars/page average with <500 total
chars, flag all pages as needing OCR. This catches PDFs where the
extractable text is minimal (form templates, image-heavy layouts)
and the bulk of content requires OCR to access.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(detect): improve CID mojibake detection for Japanese/CJK PDFs
Extend is_cid_garbage to detect CID-as-Latin-1 mojibake: when ≥40%
of characters are high Latin-1 (U+00A0-00FF) and <33% are ASCII
letters, the text is likely CID values misinterpreted as Latin-1
characters (common in Japanese/CJK PDFs with broken ToUnicode CMaps).
Also add sparse-extraction OCR flagging: TEXT-BASED PDFs with
<50 chars/page and <500 total chars get all pages flagged for OCR.
Impact: Softbank Japanese PDFs now produce empty output with
pages_needing_ocr=all instead of mojibake garbage. Korean PDFs
with valid extraction (nexo-price-en) remain unaffected.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(detect): sparse extraction check only when markdown is generated
The sparse-extraction OCR check was triggering in Analyze mode where
markdown is not generated (md_len=0), causing false OCR flags on
every PDF processed via detect-pdf --analyze.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): improve heuristic detection for borderless wrapped-cell tables
Three changes to the body-font heuristic detector:
1. Adaptive Y-gap in find_table_regions_strict: use median qualifying-row
spacing × 3 instead of fixed 25pt. Tables with wrapped cells have
larger gaps between qualifying rows (those with 3+ X-clusters).
2. Y-only region filtering: use full X range when collecting region items.
The strict X bounds from qualifying rows excluded continuation lines
in wrapped cells, starving find_column_boundaries of items.
3. Merged-band retry: when split_side_by_side splits a page into bands
but no band produces a table, retry heuristic detection on all items
merged as a single band.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): gap-histogram column detection for small tables + lower avg_cells
Two changes to fix PDF 045 (borderless table with narrow "No." column):
1. Extend gap-histogram column threshold to small tables: when the gap
between within-column jitter and between-column spacing is >10pt
(unambiguous bimodal signal), use the detected threshold even with
fewer than 500 items. Previously only triggered for dense tables.
2. Lower BodyFont avg_cells_per_row minimum from 2.5 to 2.0 to handle
tables with wrapped multi-line cells where continuation lines have
only 1 filled cell.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): trim empty outer columns + relax partial H-line validation
- Rect detection: trim empty first/last columns instead of rejecting
the whole table. Rect edges often extend beyond text boundaries.
- Line detection: accept tables with 6+ partial horizontal lines
(>15% width) when <3 full-spanning lines exist. Handles tables
with column-level separators instead of full-width rules.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): cell-rect fallback for tables with variable-width backgrounds
When rect clustering produces a grid that fails validation (empty
interior columns from variable-width cell backgrounds), fall through
to a new strategy: use rect Y-edges for row boundaries and text
X-position clustering for columns. This handles tables like the
opendataloader-bench 088-090 comparison tables where each cell has
its own background rect at different widths.
Also widen failed-cluster hint width cap for large clusters (≥30 rects)
to allow page-spanning table regions.
TEDS score on opendataloader-bench: 0.300 → 0.353.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): relax vertical line spanning validation for partial borders
Accept tables with 4+ partial vertical lines (>10% table height) when
fewer than 2 span >30%. Handles tables like opendataloader-bench 053
with column-level vertical separators that don't extend the full height.
TEDS: 0.353 → 0.377 on opendataloader-bench.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): lower cell-rect density threshold + add validation logging
- Lower cell-rect density minimum from 25% to 15% to accept sparser
tables with decorative backgrounds (fixes 147).
- Add debug logging to all heuristic validation paths for diagnosability.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): relax body-font validations for text-only and 2-column tables
Four fixes closing 71% of the TEDS gap vs opendataloader:
1. Validation 7 (table-like content): bypass numeric content requirement
for tables with 3+ columns that passed all structural checks. Text-only
tables (category lists, program descriptions) are legitimate.
2. Qualifying row threshold: lower from 3+ to 2+ X-clusters per row.
Enables 2-column body-font table detection (fixes 166).
3. Row-stripe max cell length: raise from 500 to 2000 for 3+ column
tables. Tables with paragraph descriptions in one column are valid
(fixes 121).
4. Row-stripe empty-column trimming: apply the same outer-column trim
as grid detection (fixes 121 column-0 rejection).
TEDS: 0.377 → 0.438 on opendataloader-bench (gap: -0.056 vs odl).
TEDS=0 docs: 14 → 9.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): enable 2-column body-font tables + lower all minimums
- Lower BodyFont minimum columns from 3 to 2 in detect_table_in_region
- Lower BodyFont minimum rows from 3 to 2
- Lower avg_cells_per_row minimum from 2.0 to 1.5 (handles wrapped cells
in 2-column tables)
- Apply empty-outer-column trimming to row-stripe detection (not just grid)
TEDS: 0.438 → 0.468 on opendataloader-bench (gap: -0.027 vs odl).
TEDS=0 docs: 9 → 8. 86% of original gap closed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): text-based row fallback + fix 120 flow-chart and 188 leaderboard
Three changes that push TEDS past opendataloader:
1. Cell-rect Y-edge fallback: when rects have too few Y-edges for row
structure, derive rows from text Y-position clustering within the
rect bounding box. Fixes flow-chart tables (120) and column-header-
only rects (188).
2. Lower cell-rect minimum from 20 to 6 rects to catch smaller tables.
3. Relax validation 1 (first-column presence) from 50% to 25% of rows.
Tables with wrapped model names have continuation lines without first
column content.
TEDS: 0.468 → 0.508 on opendataloader-bench.
Now BEATS opendataloader (0.508 vs 0.494, gap=+0.014).
TEDS=0 docs: 8 → 5.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): wrapped-cell continuation row merging
Merge rows that have fewer filled cells than the header row into the
previous row. Handles wrapped multi-line cells where text overflow
creates extra rows (e.g., "Direct" + "communications" → "Direct
communications").
Conditions: fewer filled cells than header, more than previous row had,
not a data row (numeric), not a short subheader label.
TEDS: 0.508 → 0.522 on opendataloader-bench (now +0.028 vs odl).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): tighten cell-rect validation + fix continuation-row merging
Cell-rect false positives:
- Raise density threshold back to 25% (from 15%)
- Add max cell length check (500 chars) to reject paragraph content
- Reject disproportionate grids (>20 rows, <4 cols)
Continuation-row merging:
- Wide tables (5+ cols): only merge rows with ≤50% header cells
- Narrow tables (2-4 cols): merge rows with fewer cells than header
- Prevents merging normal data rows in large tables (6_KE_Chart)
while keeping wrapped-cell merging for narrow tables (178)
TEDS: 0.498 on opendataloader-bench (still +0.004 vs odl).
pdf-evals: 191/192 passed, 0 regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Struct-tree tables with incomplete page tagging (e.g., only 22 of 50+
rows tagged on a page) would claim items and block rect detection,
leaving unclaimed items as loose text. Now require struct-tree tables
to capture ≥50% of band items before using them; incomplete trees
fall through to geometry-based detection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skip rect-detected tables that overlap with items already claimed by
struct-tree detection. Previously both strategies emitted separate
tables for the same content, doubling the output on tagged PDFs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a PDF has a well-formed structure tree with /Table > /TR > /TD|TH
elements linked to MCIDs, build tables directly from the semantic
hierarchy. Runs as highest-priority detection (step 0) before rect-based,
line-based, and heuristic strategies.
- Add StructTree::extract_tables() to walk the tree and collect table
descriptors with row/cell/MCID info
- Add detect_tables_from_struct_tree() to match MCIDs to TextItems
- Reject tables with <30% MCID cell coverage (stale structure trees)
- Update 2013-app2 snapshot (struct-tree gives valid but different
column ordering)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Origin-anchored full-page rects (x<5, y<5, h>20× median) are clipping
paths or page fills that bridge separate table regions into one cluster,
corrupting row-stripe detection. Exclude them from union-find adjacency
while keeping them available for hint generation and fallback paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Newsletter-style PDFs have decorative background rects (sidebar,
header, section bands) that pass row-stripe detection as false tables.
Reject when any cell exceeds 500 chars — real alternating-row data
tables have short cell content; layout backgrounds produce paragraph-
length "cells".
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(layout): relative valley column detection for justified text
Add fallback column detection using relative valley analysis for PDFs with
justified text where item widths extend past gutter boundaries. The absolute
valley detector fails on these layouts because gutter bins are at ~40% of
peak (well above the 15% noise threshold).
The relative valley detector smooths the histogram with a 5-bin moving
average, finds local minima where contrast < 0.60 of surrounding peaks,
and validates with peak balance >= 0.40. Limited to single best valley
(max 2 columns) and requires >= 100 items per page.
Tested on IRS Publication 17 (2002), a 289-page 2-column justified text
document: column detection went from ~40 pages to 165 pages.
190 passed, 0 regressions across 191 eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(layout): tighten relative valley thresholds to reduce false positives
Reduce PEAK_WINDOW from 40 to 25 bins (50pt) so valleys are only validated
against nearby peaks, not distant ones. Add MIN_PEAK_HEIGHT of 20 (smoothed)
to reject sparse pages where histogram peaks are too low to indicate dense
two-column text.
Previous thresholds caused 13 regressions across the eval suite by splitting
tables, TOCs, checklists, and forms. Now: 188 passed, 0 regressions (2 minor
metadata-only diffs on IRS P17 and 9978293).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(layout): skip relative valley detection on pages with tables
Table column gaps in the histogram look identical to text column gutters
but the table pipeline already handles reading order for those pages.
Pass page_has_table flag through detect_columns to suppress the relative
valley fallback on pages where tables were detected.
This eliminates all remaining regressions from relative valley detection:
190 passed, 0 regressions across 191 eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(layout): prose density validation for relative valley detection
Add columns_have_prose() to validate relative valley column splits.
Checks that both sides of a proposed split contain paragraph-like
content (fill ratio >= 40%, avg items/line <= 3.5) before committing
to a column split. Combined with the table-page guard, this prevents
false column splits on financial statements, forms, and tabular
layouts where long labels or dot leaders fill the column width.
Also tightens find_relative_valleys() thresholds (PEAK_WINDOW 40->25,
MIN_PEAK_HEIGHT 5->20) to reduce false positive valley candidates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Dense multi-column newspapers (WSJ, NYT) have extractable text but
produce poor output due to complex interleaved article layouts that
defeat column-based reading order. Detect these by counting Tf (font
change) operators alongside existing Tj/TJ counts, then flagging
TextBased PDFs where most sampled pages show high text density
(>=1500 ops), moderate font switches (>=50 Tf), and a low Tf/Tj
ratio (<0.15) — the ratio distinguishes newspapers from richly-styled
legal/business docs that have high Tf counts due to per-character
styling.
Calibrated against 108 TextBased PDFs in pdf-evals with zero false
positives. WSJ 50-page newspaper correctly flagged (5/8 sampled pages
match).
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The is_table_of_contents heuristic falsely rejected wide statistical
tables (e.g. ERP appendix tables) where the first column has year
labels with dot leaders ("1973..........") and other columns have
small numeric values. Add column-aware analysis: if dots are confined
to ≤1 column and ≥3 columns contain numbers, it's a data table.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(text): handle Tc/Tw character and word spacing in text width computation
PDFs using Tc (character spacing) and Tw (word spacing) operators for text
justification had words incorrectly split across TextItems. The computed
advance width didn't account for these spacing parameters, causing spurious
spaces mid-word (e.g. "deve lopers" instead of "developers").
- Add Tc/Tw operator handling and graphics state save/restore
- Incorporate char_spacing and word_spacing into compute_string_width_ts
- Add adaptive merge threshold: tighter for lowercase→lowercase junctions,
wider before joining punctuation
- Add unit tests for Tc/Tw width computation and merge behavior
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(fonts): add large Tc width computation test
Verifies that large character spacing values are applied in full
without any artificial cap, matching PDF spec behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(text): guard against Tc/Tw-inflated widths in merge and join paths
Two targeted fixes to prevent character-spacing (Tc) and word-spacing
(Tw) inflation from causing data quality regressions:
1. should_join_items: reject large negative gaps (< -font_size) that
arise when Tc/Tw inflate item widths past adjacent items. Fixes
FY_2015 merged numbers (e.g. "239.696.0" → "239.69 6.0").
2. merge_text_items: cap effective width for gap computation when Tw
inflates space-containing items beyond 0.85× font_size per char.
Prevents column-level gaps from collapsing into merge range,
recovering table detection for Baldwin-Edwards and similar PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
PDFs that embed landscape content in portrait pages via a rotated text
matrix (e.g. [0, b, -b, 0, tx, ty] for 90° CCW) produced garbled output
because the layout engine assumed x=horizontal, y=vertical.
Track the dominant text direction from combined matrices during extraction.
When ≥67% of text operators are rotated, swap x↔y coordinates (with
y-negation for correct reading order) for all text items, rects, and lines.
Also estimate text widths from char count × font size since scale_x ≈ 0
for rotated text.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When Identity-H or Type3 fonts lack a ToUnicode CMap and the
CID-as-Unicode passthrough doesn't produce valid text, the raw CID
byte values appear as mojibake (random Latin Extended characters mixed
with C1 control codes).
The detector already flags these pages in pages_needing_ocr, but the
markdown pipeline still emitted the garbage. Now, for TextBased PDFs,
we check each OCR-flagged page's extracted text for CID garbage
(C1 control characters U+0080–U+009F at ≥5% density) and strip items
from pages that fail the check.
This is scoped to TextBased PDFs only — Mixed PDFs flag pages for OCR
due to template images, not font encoding issues.
Adds test fixture (shinagawa_identity_h.pdf) and integration test.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(fonts): CID-as-Unicode passthrough and subscript/superscript merging
Add smart CID-as-Unicode passthrough for Identity-H fonts without
ToUnicode maps. Uses /W array median CID heuristic to distinguish
Unicode-CID PDFs (Chromium-generated) from GID-based subsets.
Add merge_subscript_items() pass that merges small-font items (<75%
of dominant font size, ≤4 chars, tightly adjacent) into parent items.
Fixes chemical formulas (NH3, H2O, KClO3), footnote references, and
subscript notation (vf, Hfg, m3/kg) that were previously orphaned
as separate text items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(subscripts): restrict merge to purely numeric text only
Tighten subscript merging to only merge items containing ASCII digits
(0-9). This avoids false positives with ordinal indicators (º), letter
subscripts (sol, vf), and small bullet characters (▶) that caused
table restructuring regressions.
Numeric-only keeps the primary wins: chemical formulas (NH3, H2O),
footnote references, and unit notation (m2, m3).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(subscripts): restrict merge to parent text ending with a letter
Only merge numeric subscripts when the parent item's text ends with
an alphabetic character. Prevents false merges like "33" + "1" in
fractions (33 1/3%), table credit numbers after spaces, and footnote
refs after punctuation (land.1 → land. 1). Chemical formulas (NH3,
H2O, KClO3) still merge correctly since parent ends with a letter.
Reduces pdf-eval regressions from 13 to 2 (both are correct reversions
of over-aggressive footnote merging from the prior commit).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Points to firecrawl/lopdf@edf6279 which fixes decompressed_content()
for streams without a /Filter entry. Our xobjects.rs workaround remains
as a defensive fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lopdf's decompressed_content() fails on Form XObjects without a /Filter
entry (uncompressed streams). This caused all body text to be lost in PDFs
generated by pdfrw and similar tools that wrap page content in uncompressed
Form XObjects (e.g. Cambridge University Press excerpts).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lopdf's decompressed_content() fails with DictKey("Filter") when a
ToUnicode stream has no /Filter entry (uncompressed raw text). Now
falls back to the raw stream.content when decompression fails.
Fixes Identity-H fonts with valid uncompressed ToUnicode CMaps
producing empty text (e.g. neoenergia tabela PDF).
Type3 fonts render each glyph as a custom drawing/bitmap. Without a
ToUnicode CMap the character codes can't be mapped to Unicode, so
extracted text is garbage. Pages using only Type3 fonts are now
excluded from pages_with_text and added to pages_needing_ocr.
Fixes Korean/CJK PDFs using Type3 fonts (e.g. D2Coding) being
classified as TextBased when no usable text can be extracted.
Document titles, column headers, and newsletter mastheads that repeat
on every page were being fully stripped by strip_repeated_lines. Now
the first page's occurrence is preserved so the content appears once.
Fixes missing titles like "VOICE OF SOUTH MARION" in tax certificate
PDFs and column headers in IRS forms.
When every page uses fonts with unresolvable gid-encoded glyphs,
the extracted text is unreliable garbage. Suppress the markdown
output so consumers know to use OCR instead.
Only triggers for gid-encoded fonts specifically, not for other
OCR signals (Identity-H without ToUnicode, template images) where
the text layer may still be partially useful.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(detect): flag Identity-H fonts without ToUnicode for OCR
Cyrillic (and other non-Latin) PDFs with Type0/Identity-H encoded fonts
and no ToUnicode CMap produce garbage text from direct extraction. Two
fixes:
1. Detector: new `page_has_identity_h_no_tounicode` check adds affected
pages to `pages_needing_ocr` regardless of PDF classification.
2. Extraction: extend garbage-text safety net to TextBased PDFs — when
extracted text is <50% alphanumeric, drop the markdown, set
`has_encoding_issues`, and flag all pages for OCR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: replace integration tests with synthetic unit tests
Remove PDF fixture dependencies from detector and lib tests. Use
in-memory lopdf documents to test Identity-H/ToUnicode detection logic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When rect-based, line-based, and heuristic detection all fail to find
tables on a page, try building a table directly from the layout
engine's column boundaries. Handles borderless tabular layouts like
exam/reference grids where columns are defined purely by text
alignment.
Includes header-row column refinement: when a detected column
contains multiple header items, it gets split at the gap between
them to recover the correct number of columns.
Guards against false positives: requires ≥4 columns, ≤40 rows,
>50% multi-column rows, short cells (avg ≤40 chars), no prose
content, no dominant single column, and no structural elements
(≥6 rects or ≥4 lines) on the page.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fonts using raw glyph ID names (gidNNNNN) in their Differences
encoding cannot be decoded to Unicode without the original font's
cmap table. Detect this pattern during font parsing and add
affected pages to pages_needing_ocr so downstream consumers
know to use OCR instead.
Fixes text extraction on PDFs like Tezukuri_Food-Menu.pdf where
the main body font (AcuminVariableConcept) uses gid-encoded
glyphs — even PyMuPDF and ODL fail on these.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two changes to improve balance sheet / financial table detection:
1. split_side_by_side: Don't split when one side is text labels
and the other is numeric data at matching Y positions. This
prevents splitting a single label+number table into two
independent regions.
2. try_add_label_column: After detecting a numeric-only table,
look for unclaimed text items to the left at matching Y
positions and prepend them as column 0 (row labels).
Tested on IN_Annual_Report_2017 balance sheet which now produces
proper 3-column tables (Label|2016|2017) instead of separated
number tables and paragraph text.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When rect clusters have valid bounding boxes but insufficient grid
structure (e.g. 2x2 edges from outer borders), emit their bounding
box as a RectHintRegion so the heuristic detector can be scoped to
the table area. Requires reasonable dimensions (100-600pt height,
≤500pt width) and ≥6 text items inside the region.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Screenshot PDFs (e.g., Chrome "Save as PDF") embed images inside
tiling Pattern resources rather than as direct XObject images.
Now traverses Pattern resources during image detection, correctly
classifying these as Mixed with OCR recommended.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
