extract_tables: reject partial extractions in needs_ocr gate (#87)

After enabling the vector-grid detectors on the extract path (#85)
and broadening detection to full-page grids (#83), long-cell tables
(#84), and segment-only layouts (#86), one residual failure shape
remained: detectors finding a valid grid but only capturing a small
fraction of the region's actual text. Two recurring sub-shapes:

  - "header-only": detector captured the column-header band (often a
    multi-line year/units block) but missed every data row below.
    Common in financial statements, securities tables, budget
    appendices.
  - "sparse": detector returned a handful of fragmentary cells from a
    content-rich region, missing the bulk of the page.

Both pass the existing needs_ocr quality gates — the captured cells
are well-formed markdown — but the customer would receive a 5-row
fragment of a 50-row table. Today these regions fell back to GLM-OCR
by default; flipping `__nativeTableExtraction=true` would start
serving the partials.

Add `captured_only_a_fragment(md, region_text_chars)`: rejects when
the captured non-delimiter character count is less than 25% of the
text the page extractor saw inside the region. The 200-char region
floor keeps short legitimate tables (units, axis labels) from being
mis-flagged. Wired into the existing `evaluate` quality gate
alongside is_garbage_text / is_cid_garbage / detect_encoding_issues
/ looks_like_partial_table_ex.

Verified against three representative residual cases from shadow
logs (financial-statement header band, securities-table fragment,
ESIA sparse region): all flip from `needs_ocr=false` with partial
output to `needs_ocr=true` so GLM takes over. Existing full-table
fixtures (governmental ledger, PPRA-style key/value, archival
catalog) still pass through unchanged.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Abimael Martell
2026-05-14 14:20:02 -04:00
committed by GitHub
co-authored by Claude Opus 4.7
parent 79d75dbdca
commit 7539868bf8
2 changed files with 95 additions and 4 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@firecrawl/pdf-inspector",
"version": "1.8.11",
"version": "1.8.12",
"description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
"main": "index.js",
"types": "index.d.ts",