extract_tables: reject partial extractions in needs_ocr gate (#87)
After enabling the vector-grid detectors on the extract path (#85) and broadening detection to full-page grids (#83), long-cell tables (#84), and segment-only layouts (#86), one residual failure shape remained: detectors finding a valid grid but only capturing a small fraction of the region's actual text. Two recurring sub-shapes: - "header-only": detector captured the column-header band (often a multi-line year/units block) but missed every data row below. Common in financial statements, securities tables, budget appendices. - "sparse": detector returned a handful of fragmentary cells from a content-rich region, missing the bulk of the page. Both pass the existing needs_ocr quality gates — the captured cells are well-formed markdown — but the customer would receive a 5-row fragment of a 50-row table. Today these regions fell back to GLM-OCR by default; flipping `__nativeTableExtraction=true` would start serving the partials. Add `captured_only_a_fragment(md, region_text_chars)`: rejects when the captured non-delimiter character count is less than 25% of the text the page extractor saw inside the region. The 200-char region floor keeps short legitimate tables (units, axis labels) from being mis-flagged. Wired into the existing `evaluate` quality gate alongside is_garbage_text / is_cid_garbage / detect_encoding_issues / looks_like_partial_table_ex. Verified against three representative residual cases from shadow logs (financial-statement header band, securities-table fragment, ESIA sparse region): all flip from `needs_ocr=false` with partial output to `needs_ocr=true` so GLM takes over. Existing full-table fixtures (governmental ledger, PPRA-style key/value, archival catalog) still pass through unchanged. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
79d75dbdca
commit
7539868bf8
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@firecrawl/pdf-inspector",
|
||||
"version": "1.8.11",
|
||||
"version": "1.8.12",
|
||||
"description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
|
||||
"main": "index.js",
|
||||
"types": "index.d.ts",
|
||||
|
||||
Reference in New Issue
Block a user