Stage 1 exclusive item-to-cell assignment, v1.6.4 (#66)
Closes the row-merge pattern observed against FNBO branch list at the
College/Fairway boundary: when SLANet emits cells whose y-extents
overlap between consecutive rows, an item whose center fell in the
overlap region got pulled into BOTH cells, producing run-on cells
(e.g. "Kansas Kansas" + concatenated addresses).
Cause: stage 1's "for each cell, gather items inside" loop allowed an
item to match multiple cells. `normalize_cell_bands` reduces overlap
but is biased when cell-mean-center is offset from actual text baseline
(SLANet bboxes are typically taller than their text content), so the
midpoint-clamp can land on the wrong side of the row boundary, and
items at the boundary still match two cells.
Fix: invert the matching. For each PDF text item, find candidate cells
(those whose bbox satisfies tsr_region_contains_item — center inside
OR >=60% overlap on both axes), and assign to the cell whose CENTER
is geometrically closest. Build per-cell text from the assigned items.
Stage 2 (orphan recovery) is unchanged.
Exclusivity prevents item duplication across cells. The closest-center
rule disambiguates the cell-overlap case naturally without aggressive
band clamping. normalize_cell_bands stays — it tightens cells before
matching (smaller overlap → fewer ambiguous candidates) but is no
longer load-bearing for correctness of the overlap case.
Local replay against FNBO via api/scripts/local-tsr-replay.ts (which
exercises the full layout-pod → table-pod → pdf-inspector chain
in-process):
Pre-1.6.4 (deployed 1.6.3):
|LITH West|Illinois Illinois|11700 S. IL Route 47, Huntley IL...
|College|||0534.03|
|Fairway|Kansas Kansas|4650 College Blvd... 2828 Shawnee Mission...
Post-1.6.4 (this change):
|Huntley|Illinois|11700 S. IL Route 47, Huntley IL...|8711.15|
|LITH West|Illinois|4520 W Algonquin Rd, Lake in the Hills...
|College|Kansas|4650 College Blvd, Overland Park KS...|0532.01|
|Fairway|Kansas|2828 Shawnee Mission Pkwy, Fairway KS...|0500.00|
One row (Shawnee in the Kansas section) can still get dropped under
SLANet detection variance — the model occasionally under-detects rows
and emits N structure rows for N+1 PDF rows. The squeezed row's text
gets routed to the structurally-nearest existing cell. This is a
SLANet limitation, not addressable in pdf-inspector without
synthesizing rows from PDF text geometry; deferred.
Tests: 416 lib + 120 integration + 2 doctests pass. clippy + fmt clean.
Bump @firecrawl/pdf-inspector to 1.6.4 (patch — refines stage 1's
matching strategy; no API changes).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
9cce4dd161
commit
d8894326e8
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@firecrawl/pdf-inspector",
|
"name": "@firecrawl/pdf-inspector",
|
||||||
"version": "1.6.3",
|
"version": "1.6.4",
|
||||||
"description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
|
"description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
|
||||||
"main": "index.js",
|
"main": "index.js",
|
||||||
"types": "index.d.ts",
|
"types": "index.d.ts",
|
||||||
|
|||||||
+64
-15
@@ -931,24 +931,73 @@ pub fn extract_tables_with_structure_cells_mem(
|
|||||||
|
|
||||||
normalize_cell_bands(&mut cells);
|
normalize_cell_bands(&mut cells);
|
||||||
|
|
||||||
// Stage 1: strict text fill — each cell gets the items whose centers
|
// Stage 1: exclusive per-item assignment. For each PDF text item,
|
||||||
// fall inside its (normalized) bbox or whose >=60% overlap rule fires.
|
// find the cell(s) whose (band-clamped) bbox satisfies the strict
|
||||||
// Track which item indices any cell claimed so the orphan pass below
|
// membership rule (`tsr_region_contains_item`: center inside OR
|
||||||
// doesn't double-assign.
|
// >=60% overlap on both axes). If multiple cells qualify, assign
|
||||||
|
// the item to the cell whose center is geometrically closest. If
|
||||||
|
// exactly one qualifies, assign to that. If none, the item is an
|
||||||
|
// orphan and stage 2 below tries to recover it.
|
||||||
|
//
|
||||||
|
// The exclusivity (one item → one cell) prevents the cell-overlap
|
||||||
|
// bug where SLANet emits cells whose y-extents overlap between
|
||||||
|
// rows: under the previous "for each cell, gather items" approach,
|
||||||
|
// an item whose center fell in two cells' overlap got duplicated
|
||||||
|
// into both. Closest-center disambiguation routes it to the
|
||||||
|
// correct row.
|
||||||
let mut claimed: std::collections::HashSet<usize> = std::collections::HashSet::new();
|
let mut claimed: std::collections::HashSet<usize> = std::collections::HashSet::new();
|
||||||
for cell in &mut cells {
|
let mut item_to_cell: std::collections::HashMap<usize, usize> =
|
||||||
let [x1, y1, x2, y2] = cell.page_pt_bbox;
|
std::collections::HashMap::new();
|
||||||
let bounds = region_bounds(x1, y1, x2, y2, page_h, coords);
|
|
||||||
let mut matched: Vec<TextItem> = Vec::new();
|
// Pre-compute each cell's bounds + center (in PDF-pt-flipped space)
|
||||||
for (i, item) in items.iter().enumerate() {
|
// so we don't redo the work per item.
|
||||||
if tsr_region_contains_item(item, bounds) {
|
let cell_meta: Vec<Option<(RegionBounds, f32, f32)>> = cells
|
||||||
claimed.insert(i);
|
.iter()
|
||||||
matched.push(item.clone());
|
.map(|cell| {
|
||||||
|
let [x1, y1, x2, y2] = cell.page_pt_bbox;
|
||||||
|
if x1 >= x2 || y1 >= y2 {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let bounds = region_bounds(x1, y1, x2, y2, page_h, coords);
|
||||||
|
let cx = (bounds.x_min + bounds.x_max) * 0.5;
|
||||||
|
let cy = (bounds.y_min + bounds.y_max) * 0.5;
|
||||||
|
Some((bounds, cx, cy))
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
for (item_idx, item) in items.iter().enumerate() {
|
||||||
|
let item_w = text_utils::effective_width(item);
|
||||||
|
let item_cx = item.x + item_w * 0.5;
|
||||||
|
let item_cy = item.y + item.height * 0.5;
|
||||||
|
let mut best: Option<(usize, f32)> = None;
|
||||||
|
for (cell_idx, meta) in cell_meta.iter().enumerate() {
|
||||||
|
let Some((bounds, ccx, ccy)) = meta else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
if !tsr_region_contains_item(item, *bounds) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let dx = item_cx - ccx;
|
||||||
|
let dy = item_cy - ccy;
|
||||||
|
let dist_sq = dx * dx + dy * dy;
|
||||||
|
if best.is_none_or(|(_, d)| dist_sq < d) {
|
||||||
|
best = Some((cell_idx, dist_sq));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Markdown cells must be one line — collapse line breaks produced
|
if let Some((ci, _)) = best {
|
||||||
// by the line-grouping pass.
|
claimed.insert(item_idx);
|
||||||
cell.text = collect_text_from_matched_items(matched, adaptive_threshold)
|
item_to_cell.insert(item_idx, ci);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build per-cell text from the assigned items. Markdown cells must
|
||||||
|
// be one line — collapse line breaks from the line-grouping pass.
|
||||||
|
let mut per_cell_items: Vec<Vec<TextItem>> = vec![Vec::new(); cells.len()];
|
||||||
|
for (&item_idx, &cell_idx) in &item_to_cell {
|
||||||
|
per_cell_items[cell_idx].push(items[item_idx].clone());
|
||||||
|
}
|
||||||
|
for (cell_idx, matched) in per_cell_items.into_iter().enumerate() {
|
||||||
|
cells[cell_idx].text = collect_text_from_matched_items(matched, adaptive_threshold)
|
||||||
.replace(['\n', '\r'], " ");
|
.replace(['\n', '\r'], " ");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user