Combine per-page markdown extraction with layout classification into a
single parse. extractPagesMarkdown now returns PagesExtractionResult with
pages_with_tables, pages_with_columns, pages_needing_ocr, and is_complex
alongside the per-page markdown — eliminating redundant PDF parses for
callers that need both.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* add extract_pages_markdown_mem for per-page markdown extraction
Enables hybrid OCR pipelines to skip GPU render+layout for simple text
pages by providing per-page markdown with needs_ocr flags. Font stats
are computed document-wide for consistent header detection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* bump napi package version to 0.6.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace stringly-typed pdf_type and item_type fields with
#[napi(string_enum)] enums for proper TypeScript type checking.
Add link_url field to TextItem instead of encoding URL in the
item_type string.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two changes that reduce false needsOcr rejections without hurting quality:
1. Per-region GID check instead of per-page blanket rejection.
Previously, if ANY font on the page used GID-encoded glyphs (common
in logos, decorative fonts), ALL table and text regions on that page
were forced to GPU OCR via needsOcr=true. Now the page-level bail is
removed; per-region text quality checks (is_garbage_text, is_cid_garbage,
detect_encoding_issues) catch actual GID corruption in the extracted
content. Tables whose text is clean pass through even if an unrelated
font elsewhere on the page is GID-encoded.
2. Relaxed looks_like_partial_table for layout-assisted extraction.
When the layout model already identified a region as a table (i.e.,
extract_tables_in_regions_mem), boundary-detection heuristics are
less necessary — we're not guessing "is this a table?" anymore, only
"can we extract it correctly?". Relaxations:
- Numeric first header cell accepted (e.g., year "2024")
- 1 empty header cell allowed in 3+ column tables (merged headers)
- Sparse first data row threshold relaxed from 33% to 50%
Paragraph detection and duplicate-header checks remain strict.
Eval: 196/196 pass (full regression suite), 91/91 Rust tests pass
including 7 new layout-assisted validation tests. Zero regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a 5th failure-mode check to looks_like_partial_table: when the
heuristic mis-detects text-wrapped paragraph prose as a multi-column
table, cells in the same column tend to start with lowercase letters
or continuation punctuation (commas, closing quotes) — because they're
actually sentence fragments. Real tables almost never have most data
cells starting lowercase.
Trigger: ≥2 cols, ≥4 data rows, ≥60% of non-empty data cells start
with lowercase or continuation punctuation → return needs_ocr=true.
Caught in the eval as the next-largest failure mode after the 0.4.1 fix:
PDFs 088, 182, 090 — heuristic produced "tables" like:
|Approval is needed from the|Acquisitions of|
|Treasurer if the acquisition|residential and|
|constitutes a "significant|agricultural|
|action," including acquiring an|land by foreign|
Reading column 1 top-to-bottom: "Approval is needed from the Treasurer
if the acquisition constitutes a 'significant action,' including
acquiring an interest..." — a paragraph, not tabular data.
Tests: 2 new tests (the 088-style failure case + a real multi-word
table that must NOT be flagged). All 11 looks_like_partial_table tests
pass; 323 unit + 91 integration tests still green.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the heuristic returns markdown that looks like a partial / mis-detected
table, set needs_ocr=true so the caller falls back to GPU OCR. Previously the
same cases returned the broken table with needs_ocr=false, which produced
real-world TEDS=0 scores in fire-pdf evals (heuristic-built table didn't
match ground truth structure at all, but caller had no signal to fall back).
Four failure modes detected, all observed in opendataloader-bench eval losses:
1. **Header looks like a data row** — first cell of header is a bare number
(e.g. `|2|...`), suggesting the actual header row was skipped. Real
headers almost never start with just a number.
2. **Empty header cells in a multi-column table** — ≥3 cols, ≥1 empty cell
in the header row. Indicates poor column boundary detection.
3. **Duplicate header cells** — same non-empty value appearing twice in the
header (e.g. "Administration|Administration"). Means a multi-line header
was collapsed wrong.
4. **Sparse first data row** — ≥3 cols and ≥1/3 of first-data-row cells are
empty. Multi-row headers in the source PDF get smashed into header +
sparse data row by the heuristic; this catches that.
Tests: 9 new unit tests in `looks_like_partial_table_tests` cover each
failure mode plus realistic non-failures (well-formed table, single-column
list, two-col with a single empty cell). All 91 existing tests still pass.
Bumps `napi/package.json` to 0.4.1 since this changes the function's return
behaviour for callers (some inputs that returned needs_ocr=false now return
true). The output text field is also cleared on the new fallback path so
callers don't accidentally use the broken markdown.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a new function that takes a PDF buffer and page+bbox regions (same interface
as extractTextInRegions), runs heuristic table detection on items within each region,
and returns markdown pipe-tables. Falls back to needs_ocr=true when no table
structure is found or text quality is suspect.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When fire-pdf sends pre-segmented bboxes from the layout model,
pdf-inspector no longer runs column detection, stream-order heuristics,
or newspaper/tabular mode detection within the region. These heuristics
conflict with the layout model's decisions and cause wrong reading order.
Region extraction now simply: Y-sorts items, groups into lines, and
sorts within each line by X position. The heavy heuristics remain
available for standalone full-page extraction.
Eval showed pure OCR (0.2875 NED) beating native+heuristics (0.2916)
across all categories, especially multi-column (-0.08) and newspaper
(-0.16). This change should close that gap.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rust panics in NAPI modules abort the Node.js process with no chance
to report errors. This wraps every exported function in catch_unwind,
converting panics into JS Error exceptions that can be caught and
reported to Sentry.
Buffer data is extracted to Vec<u8> before the catch_unwind boundary
to satisfy UnwindSafe requirements.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
index.js and index.d.ts are generated by napi-rs. Generate them
at publish time via `napi prepublish` instead of checking them in.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix sort panics on NaN values from bogus PDF font metrics
Replace all `partial_cmp(...).unwrap_or(Ordering::Equal)` and bare
`partial_cmp(...).unwrap()` with `total_cmp()` across the codebase.
`partial_cmp` returns `None` for NaN, and mapping that to `Equal`
violates total ordering: `a == NaN` and `NaN == b` but `a != b`.
Rust 1.81+ detects this and panics in sort_by. `total_cmp` handles
NaN deterministically (sorts to end) and guarantees total ordering.
The critical crash was in `extract_text_in_regions` (lib.rs:478)
where PDFs with bogus font ascent/descent values produced NaN in
text item coordinates, causing process abort via NAPI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix missed partial_cmp in layout.rs and restore napi exports
- Convert two remaining b.y.partial_cmp(&a.y) calls to total_cmp
in group_single_column and column layout sorting
- Restore missing napi exports: detectPdf, extractText,
extractTextWithPositions, processPdf
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Drop platform-specific optional deps — ship all .node binaries in one
package (~4 MB total). Simplifies publishing and consumer install.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skip napi prepublish/artifacts commands that require GitHub API auth.
Instead, create platform package.json files and copy binaries directly.
Add optionalDependencies to main package so npm/bun auto-selects the
right binary.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Both bindings now expose the same 6 function families: process, detect,
classify, extractText, extractTextWithPositions, and extractTextInRegions.
Bumps PyO3 from 0.22 to 0.25 for Python 3.14 support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the napi bridge from fire-pdf into pdf-inspector as `napi/`.
Package name: @firecrawl/pdf-inspector-js, published to GitHub Packages
(npm.pkg.github.com) as a public package on v* tags.
Exposes two functions:
- classifyPdf(buffer) → type, page count, pages needing OCR
- extractTextInRegions(buffer, pageRegions) → per-region text with
needsOcr quality flag (GID fonts, garbage, encoding issues)
Includes publish workflow that builds linux-x64-gnu + darwin-arm64
binaries and publishes main + platform-specific packages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
