extract_tables: try vector-grid detectors before text heuristic (#85)

* extract_tables: try vector-grid detectors before text heuristic

extract_tables_in_regions_mem previously ran only the text-only
heuristic detector (tables::detect_tables) on the items inside each
region, discarding the rects and lines that extract_page_text_items
returned. That left the rect-backed and line-backed detectors
(detect_tables_from_rects, detect_tables_from_lines) unused by the
public region-scoped extraction path — they only ran through
detect_vector_grid_in_region_mem, which most callers don't use.

Keep the rects and lines, filter them to each region, and try in
order: rect detector → line detector → heuristic. Each candidate's
markdown is quality-gated by the existing needs_ocr checks
(is_garbage_text, is_cid_garbage, detect_encoding_issues,
looks_like_partial_table_ex); only the first clean output wins.
If all three produce empty or noisy output we still return
needs_ocr=true, matching prior behavior.

Effect on real prod-shape inputs from shadow logs:

  Full-page ruled ledger, 6 cols x ~15 rows:
    before: heuristic emits a 355-char two-row fragment
    after:  line detector emits the full 6520-char table

  Multi-row key/value layout with paragraph values:
    before: heuristic emits a 188-char header-only fragment
    after:  rect detector emits the full 1733-char table including
            the multi-bullet description cell

Existing fixtures that already passed via the heuristic continue to
pass: the quality gate rejects partial vector-grid output and falls
through, so the heuristic still wins where it produced the cleaner
result.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Bump version from 1.8.9 to 1.8.10

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Abimael Martell
2026-05-12 19:25:08 -04:00
committed by GitHub
co-authored by Claude Opus 4.7
parent f26efe6673
commit b5b91470db
3 changed files with 111 additions and 42 deletions
+27
View File
@@ -1648,6 +1648,33 @@ fn test_bits_pilani_page8_table_detection() {
assert!(!region.needs_ocr, "Page 8 table should still be detected");
}
#[test]
fn test_extract_tables_in_regions_uses_line_grid() {
// Stroked-grid table (m/l/S path operators forming a 2x2 grid).
// The heuristic text-only detector handles the same cells already,
// so this guards that the line-backed path doesn't regress: the
// markdown still contains all four data cells.
let buf = synthetic_vector_grid_pdf(false);
let results =
extract_tables_in_regions_mem(&buf, &[(0, vec![[40.0, 50.0, 220.0, 760.0]])]).unwrap();
let region = &results[0].regions[0];
assert!(
!region.needs_ocr,
"stroked-grid table should be extracted, got needs_ocr=true"
);
for tok in ["A1", "B1", "A2", "B2"] {
assert!(
region.text.contains(tok),
"expected '{tok}' in output, got: {}",
region.text
);
}
assert!(
region.text.contains('|'),
"expected pipe-delimited markdown"
);
}
// =========================================================================
// extract_tables_with_structure_mem tests (TSR-aware path)
// =========================================================================