Compare commits

..
Author SHA1 Message Date
Abimael Martell 66103742c3 tables: keep wrapped labels in TSR output 2026-04-30 10:25:38 -07:00
Abimael Martell 97fc32ac70 tables: prefer rect edges for cell-grid fallback (#72)
* tables: prefer rect edges in cell-grid fallback

* Bump version from 1.8.1 to 1.8.2
2026-04-29 21:27:27 -07:00
Abimael Martell a4161c8392 Bump version from 1.8.0 to 1.8.1 2026-04-29 08:23:05 -07:00
Abimael Martell 5b1fe30c66 tables: expand multi-row cells in-place when fallback heuristic is empty (#71)
* tables: expand multi-row TSR cells in place

Recover row-under-counted TSR tables by splitting overstuffed cells with native PDF text bands before falling back to heuristic extraction.

Made-with: Cursor

* docs: note multi-row expansion scope

Clarify that the row-band cap intentionally keeps v1 focused on common small row-loss cases while larger compressions continue to use heuristic fallback.

Made-with: Cursor
2026-04-29 08:18:09 -07:00
Abimael Martell 63b5573133 Bump version from 1.7.2 to 1.8.0 2026-04-28 17:21:44 -07:00
Abimael Martell c186a036fc tables: add detectVectorGridInRegion napi export for region-scoped vector grid detection (#70)
* feat: add vector grid region detector napi export

Expose region-scoped vector PDF grid detection so TSR callers can reuse native geometry before model fallback.

Made-with: Cursor

* fix: address vector grid review feedback

Return null for rotated vector grids until the coordinate transform has coverage and reject out-of-crop cell boxes surfaced by real-PDF smoke testing.

Made-with: Cursor

* test: add crop bbox plausibility coverage

Cover in-crop, out-of-crop, slack-boundary, and non-positive DPI behavior for vector grid cell bbox validation.

Made-with: Cursor
2026-04-28 17:20:15 -07:00
Abimael MartellandClaude Opus 4.7 d196d435d1 fix: TSR auto-fallback bugs found in review, v1.7.2 (#68)
Three fixes to extract_tables_with_structure_auto_mem (added in
1.7.1) caught by external review:

1. multi_row_in_cell over-triggered on legitimate multi-line cells.
   The previous threshold (item span > 1.3× either smallest cell or
   tallest item height) fires on any cell with 2+ y-separated text
   items — including rowspan>1 cells, wrapped descriptions, and
   superscript/subscript runs. Replaced with two gates:
   - skip cells whose declared rowspan > 1 (intentional multi-line)
   - require an actual whitespace gap (>~half a line height)
     between the bottom of one item and the top of the next, in
     PDF-native y-coordinates. Same-line items with tall glyphs or
     superscripts have negative or near-zero gap; truly separate
     visual rows have gap ≈ leading − line-height.
   FNBO regression test still passes; new test covers a rowspan=2
   cell with two visible text lines and verifies no fallback fires.

2. Heuristic returning empty silently replaced TSR markdown with
   "". The auto wrapper now keeps the TSR markdown when the
   heuristic markdown is empty/whitespace and tags fallback_reason
   with `_heuristic_empty` suffix (e.g.
   `multi_row_in_cell_heuristic_empty`). Worst case we ship the
   same wrong-but-non-empty TSR output we'd have shipped before
   1.7.1; we never replace useful output with literally nothing.

3. One bad input blanked the whole batch. Errors from
   detect_tsr_quality_issue or extract_tables_in_regions_mem now
   stay scoped to the single input — that input falls through to
   raw TSR markdown with a `_error` reason label so callers can
   metric on it. Other inputs in the batch are unaffected.

3 new integration tests:
- test_auto_does_not_fire_on_legit_rowspan_cell
- test_auto_keeps_tsr_markdown_when_heuristic_returns_empty
- test_auto_isolates_per_input_failures

All 6 auto tests + full 123-test suite pass. FNBO local replay
still triggers fallback (phantom_empty_row signal in this run) and
emits correct Shawnee/BVP/Sonoma rows with correct census tracts.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 11:23:03 -07:00
Abimael MartellandClaude Opus 4.7 fbab84fc20 feat: TSR auto-fallback to heuristic on quality issues, v1.7.1 (#67)
Adds extract_tables_with_structure_auto_mem (Rust) /
extractTablesWithStructureAuto (napi). Returns
TableExtractionResult { markdown, fallback_reason } per input.

The wrapper runs the existing TSR-hybrid path then checks the
resulting cells for two known SLANet detection pathologies:

* phantom_empty_row: empty row sandwiched between non-empty rows
  (cheap, cell-metadata only).
* multi_row_in_cell: re-reads PDF text items, flags any cell whose
  contained items span >1.3× either the smallest cell height or
  the tallest contained item height. Catches the FNBO failure mode
  where a tall TSR cell absorbs two adjacent PDF rows.

When either fires, extract_tables_in_regions_mem runs over the same
crop bbox and its markdown replaces the TSR markdown.
fallback_reason carries the diagnostic label so callers can emit
metrics and watch each pathology independently.

Validated on FNBO branches PDF page 1 (Kansas region):
- TSR-only: merges Shawnee into BVP, wrong census tract on Sonoma.
- Auto fallback (phantom_empty_row): each row separate, correct tracts.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 10:13:25 -07:00
Abimael Martell bdea4f345a Bump version from 1.6.4 to 1.7.0 2026-04-27 10:01:30 -07:00
Abimael Martell 8a0f98dee7 repair malformed PDF containers (#65) 2026-04-27 09:38:25 -07:00
Abimael MartellandClaude Opus 4.7 d8894326e8 Stage 1 exclusive item-to-cell assignment, v1.6.4 (#66)
Closes the row-merge pattern observed against FNBO branch list at the
College/Fairway boundary: when SLANet emits cells whose y-extents
overlap between consecutive rows, an item whose center fell in the
overlap region got pulled into BOTH cells, producing run-on cells
(e.g. "Kansas Kansas" + concatenated addresses).

Cause: stage 1's "for each cell, gather items inside" loop allowed an
item to match multiple cells. `normalize_cell_bands` reduces overlap
but is biased when cell-mean-center is offset from actual text baseline
(SLANet bboxes are typically taller than their text content), so the
midpoint-clamp can land on the wrong side of the row boundary, and
items at the boundary still match two cells.

Fix: invert the matching. For each PDF text item, find candidate cells
(those whose bbox satisfies tsr_region_contains_item — center inside
OR >=60% overlap on both axes), and assign to the cell whose CENTER
is geometrically closest. Build per-cell text from the assigned items.
Stage 2 (orphan recovery) is unchanged.

Exclusivity prevents item duplication across cells. The closest-center
rule disambiguates the cell-overlap case naturally without aggressive
band clamping. normalize_cell_bands stays — it tightens cells before
matching (smaller overlap → fewer ambiguous candidates) but is no
longer load-bearing for correctness of the overlap case.

Local replay against FNBO via api/scripts/local-tsr-replay.ts (which
exercises the full layout-pod → table-pod → pdf-inspector chain
in-process):

  Pre-1.6.4 (deployed 1.6.3):
    |LITH West|Illinois Illinois|11700 S. IL Route 47, Huntley IL...
    |College|||0534.03|
    |Fairway|Kansas Kansas|4650 College Blvd... 2828 Shawnee Mission...

  Post-1.6.4 (this change):
    |Huntley|Illinois|11700 S. IL Route 47, Huntley IL...|8711.15|
    |LITH West|Illinois|4520 W Algonquin Rd, Lake in the Hills...
    |College|Kansas|4650 College Blvd, Overland Park KS...|0532.01|
    |Fairway|Kansas|2828 Shawnee Mission Pkwy, Fairway KS...|0500.00|

One row (Shawnee in the Kansas section) can still get dropped under
SLANet detection variance — the model occasionally under-detects rows
and emits N structure rows for N+1 PDF rows. The squeezed row's text
gets routed to the structurally-nearest existing cell. This is a
SLANet limitation, not addressable in pdf-inspector without
synthesizing rows from PDF text geometry; deferred.

Tests: 416 lib + 120 integration + 2 doctests pass. clippy + fmt clean.

Bump @firecrawl/pdf-inspector to 1.6.4 (patch — refines stage 1's
matching strategy; no API changes).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 09:22:58 -07:00
Abimael MartellandClaude Opus 4.7 9cce4dd161 TSR stage 2: reject cross-line orphan stacking, v1.6.3 (#64)
Closes the residual run-on-cell pattern observed on FNBO branch list
after 1.6.2 deployed:

  |Shawnee Blue Valley Parkway|Kansas|6301 Pflumm, ...|0523.04|
  |Sonoma Plaza|Kansas Kansas|<addr1> <addr2>|0531.05|
  |Mitchell Woonsocket|South Dakota|<addr1>|9628.01|

Cause: when col-N cells across multiple consecutive rows are y-shifted
the same way (a local SLANet drift), the stage 2 orphan pass sees
multiple orphans qualifying for the same nearest empty cell. The cell
gets all of them appended in order, producing "RowA-text RowB-text".

Fix: track the y-coordinate of the first orphan that lands in each
cell. Subsequent orphans only join that cell if their y is within
half-a-row-height of the first orphan's y (same line). Cross-line
orphans skip that cell and look for the next-nearest empty cell on
their own line.

Same-line slack preserves multi-token branch names like
"Blue Valley Parkway" (3 PDF text items at the same y) — all three
stack into the same cell. Cross-row stacking is what gets rejected.

Two new tests:

  - stage2_rejects_cross_line_stacking_into_same_cell
    Two orphans on different rows, both equidistant to the same empty
    cell. First wins; second routes to its own row's cell.

  - stage2_allows_same_line_orphans_to_stack_into_one_cell
    Three same-line orphans (multi-token branch name) all land in the
    same empty cell, joined by spaces.

The existing four stage-2 tests + the cell-bleed regression test from
PR #62 + #63 all pass: 416 lib + 115 integration + 2 doctests.
clippy + fmt clean.

Bump @firecrawl/pdf-inspector to 1.6.3 (patch — refines 1.6.2; no API
changes).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 08:11:17 -07:00
Abimael MartellandClaude Opus 4.7 f61d139710 Recover orphan text after band normalization (TSR stage 2), v1.6.2 (#63)
PR #62 (1.6.1) closed the cell-bleed regression by clamping SLANet's
loose cell bboxes into non-overlapping row/column bands and tightening
text membership to center-containment OR >=60% overlap. That worked,
but exposed the opposite failure: legitimate native PDF text whose
center fell just outside the *clamped* cell bbox now had nowhere to go.

Two distinct failure modes observed against the FNBO branch-list PDF
after 1.6.1 deployed:

  Symptom A — header text positioned at the LEFT of a column whose band
  was derived from data-cell centers farther right. Header "Address"
  PDF text at x=331..375 fell outside the clamped col band starting
  at x=410. Strict membership rejected it (0% x-overlap, center
  outside).

  Symptom B — local SLANet row drift in col 0 over a 5-row stretch.
  Cell bboxes sat just above the actual branch-name text items
  (x-overlap 100% but y-overlap ~30-43%, below the 60% threshold).

Both share one root: post-normalization bboxes are too tight, and the
strict rule has no escape valve for legitimate edge text.

Fix: add a stage-2 orphan-recovery pass after the strict fill. Items
that NO cell claimed in stage 1 get re-assigned to their nearest
*empty* cell, distance-capped by `(median_col_width, median_row_height)`
so a far-orphan figure title can't get pulled into a faraway empty
cell. Stage 2 only fills empties — never overwrites stage 1 — so the
cell-bleed case PR #62 closed cannot regress.

Three new lib tests cover the bug shapes:
  - stage2_recovers_left_aligned_header_text_outside_data_band
    (Symptom A: header text left-of-band, data cells already filled,
     stage 2 fills only the header)
  - stage2_recovers_y_shifted_col0_in_consecutive_rows
    (Symptom B: 3 col-0 cells shifted vs text, all 3 recovered)
  - stage2_does_not_overwrite_filled_cells_or_admit_far_orphans
    (cap rejects far figure titles; pre-filled cells untouched)

Plus tests for the cap-derivation helper:
  - tsr_assignment_caps_uses_median_geometry
  - tsr_assignment_caps_floor_protects_degenerate_input

The existing dense-overlapping-rows regression test (added in #62) still
passes, confirming no regression on the cell-bleed case.

Test results: 414 lib + 115 integration + 2 doctests pass. clippy + fmt
clean.

Bump @firecrawl/pdf-inspector to 1.6.2 (patch — fixes a regression
introduced by 1.6.1; no API changes).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 23:36:43 -07:00
Abimael MartellandClaude Opus 4.7 3f8fb645c9 Fix TSR cell assignment for overlapping table boxes (#62)
* feat: TSR-aware table extraction (extract_tables_with_structure_mem)

New public function that consumes raw structure-recovery output (HTML
structure tokens + per-cell bboxes from a model like SLANet) and assembles
markdown tables by pulling cell text from the native PDF — no OCR, no
geometry inference.

Why: the existing extract_tables_in_regions_mem infers grid geometry from
text positions only and can't distinguish merged cells from multiple narrow
columns. Pairing structure recovery from a layout/TSR model with native
PDF text gets perfect text quality with proper row/col/span structure.

- New module src/tables/structured.rs: token state machine, polygon→AABB,
  crop-px→page-pt, rowspan/colspan-aware cell layout, markdown emitter.
  Accepts both 4-element rects and 8-element 4-corner polygons.
- New public extract_tables_with_structure_mem in src/lib.rs that reuses
  extract_page_text_items, region_overlaps_item, and the shared region
  text-collection helper. No existing public function modified.
- napi binding extractTablesWithStructure mirroring the existing
  extractTablesInRegions shape (f64 in JS → f32 internally).
- 14 unit tests + 5 integration tests, including a real-PDF gold-standard
  match against bits_pilani_feedback.pdf.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* TSR follow-ups: header-aware separator, cells API, v1.6.0

- cells_to_markdown emits the separator after the LAST row that contains
  is_header=true cells, falling back to "after row 0" when no header is
  flagged. Multi-row theads now render correctly. Three new unit tests
  cover: multi-row header, header not on row 0, no headers (fallback).
- New public extract_tables_with_structure_cells_mem returning
  Vec<Vec<StructuredCell>> so callers can drive their own rendering or
  debug overlays without re-doing the parse + extraction. The markdown
  variant now wraps it. The previously-unused page_pt_bbox field is
  surfaced through this API.
- New napi binding extractTablesWithStructureCells + StructuredCellJs.
- Bump @firecrawl/pdf-inspector to 1.6.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix TSR cell text assignment for overlapping bboxes

Made-with: Cursor

* bump npm package version to 1.6.1

Made-with: Cursor

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 16:35:19 -07:00
Abimael MartellandClaude Opus 4.7 f6d5e214f1 feat: TSR-aware table extraction (extract_tables_with_structure_mem) (#61)
* feat: TSR-aware table extraction (extract_tables_with_structure_mem)

New public function that consumes raw structure-recovery output (HTML
structure tokens + per-cell bboxes from a model like SLANet) and assembles
markdown tables by pulling cell text from the native PDF — no OCR, no
geometry inference.

Why: the existing extract_tables_in_regions_mem infers grid geometry from
text positions only and can't distinguish merged cells from multiple narrow
columns. Pairing structure recovery from a layout/TSR model with native
PDF text gets perfect text quality with proper row/col/span structure.

- New module src/tables/structured.rs: token state machine, polygon→AABB,
  crop-px→page-pt, rowspan/colspan-aware cell layout, markdown emitter.
  Accepts both 4-element rects and 8-element 4-corner polygons.
- New public extract_tables_with_structure_mem in src/lib.rs that reuses
  extract_page_text_items, region_overlaps_item, and the shared region
  text-collection helper. No existing public function modified.
- napi binding extractTablesWithStructure mirroring the existing
  extractTablesInRegions shape (f64 in JS → f32 internally).
- 14 unit tests + 5 integration tests, including a real-PDF gold-standard
  match against bits_pilani_feedback.pdf.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* TSR follow-ups: header-aware separator, cells API, v1.6.0

- cells_to_markdown emits the separator after the LAST row that contains
  is_header=true cells, falling back to "after row 0" when no header is
  flagged. Multi-row theads now render correctly. Three new unit tests
  cover: multi-row header, header not on row 0, no headers (fallback).
- New public extract_tables_with_structure_cells_mem returning
  Vec<Vec<StructuredCell>> so callers can drive their own rendering or
  debug overlays without re-doing the parse + extraction. The markdown
  variant now wraps it. The previously-unused page_pt_bbox field is
  surfaced through this API.
- New napi binding extractTablesWithStructureCells + StructuredCellJs.
- Bump @firecrawl/pdf-inspector to 1.6.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 00:55:39 -07:00
Abimael Martell 5c4c6e8d33 Table improvements (v1.5.0) 2026-04-22 09:52:15 -07:00
Abimael Martell d7fb493a61 fix tagged table header recovery (#60) 2026-04-22 09:51:15 -07:00
16 changed files with 6004 additions and 501 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@firecrawl/pdf-inspector",
"version": "1.4.0",
"version": "1.8.3",
"description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
"main": "index.js",
"types": "index.d.ts",
+239 -3
View File
@@ -99,6 +99,13 @@ pub struct PageRegionTexts {
pub regions: Vec<RegionText>,
}
/// Vector-grid detection result compatible with `extractTablesWithStructure*`.
#[napi(object)]
pub struct VectorGridDetectionJs {
pub structure_tokens: Vec<String>,
pub cell_bboxes: Vec<Vec<f64>>,
}
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
@@ -317,6 +324,236 @@ pub fn extract_tables_in_regions(
})
}
/// Detect a vector ruled-line / rectangle grid inside one page region.
///
/// Returns TSR-compatible structure tokens plus crop-pixel cell bboxes, or
/// `null` when the region does not contain a valid vector grid.
///
/// `pageIdx` is 0-indexed. `regionPdfPtBbox` is `[x1,y1,x2,y2]` in PDF
/// points with top-left origin. `renderDpi` is the DPI of the crop image that
/// will consume the returned cell bboxes.
#[napi]
pub fn detect_vector_grid_in_region(
buffer: Buffer,
page_idx: u32,
region_pdf_pt_bbox: Vec<f64>,
render_dpi: f64,
) -> Result<Option<VectorGridDetectionJs>> {
let bytes: Vec<u8> = buffer.to_vec();
let region = if region_pdf_pt_bbox.len() == 4 {
[
region_pdf_pt_bbox[0] as f32,
region_pdf_pt_bbox[1] as f32,
region_pdf_pt_bbox[2] as f32,
region_pdf_pt_bbox[3] as f32,
]
} else {
[0.0, 0.0, 0.0, 0.0]
};
catch_panic("detect_vector_grid_in_region", move || {
let result = pdf_inspector::detect_vector_grid_in_region_mem(
&bytes,
page_idx,
region,
render_dpi as f32,
)
.map_err(|e| to_napi_err(e, "detect_vector_grid_in_region"))?;
Ok(result.map(|r| VectorGridDetectionJs {
structure_tokens: r.structure_tokens,
cell_bboxes: r
.cell_bboxes
.into_iter()
.map(|bbox| bbox.into_iter().map(|v| v as f64).collect())
.collect(),
}))
})
}
/// One cropped table region plus its raw structure-recovery output, for
/// `extractTablesWithStructure`.
///
/// `structureTokens` and `cellBboxes` are typically produced by an external
/// table-structure recognition model (e.g. SLANet on PaddleOCR) running on
/// a rendered crop of the page. pdf-inspector uses the structure to lay out
/// the cells and pulls the cell text from the native PDF — no OCR involved.
#[napi(object)]
pub struct TsrTableInputJs {
/// 0-indexed page number where the crop was taken from.
pub page: u32,
/// Crop bbox on the page, `[x1, y1, x2, y2]` in PDF points with
/// top-left origin.
pub crop_pdf_pt_bbox: Vec<f64>,
/// DPI the crop image was rendered at (e.g. `200.0`).
pub render_dpi: f64,
/// Raw structure tokens emitted by the TSR model, in document order.
pub structure_tokens: Vec<String>,
/// One bbox per cell (in document order). May be 4-element
/// `[x1,y1,x2,y2]` or 8-element 4-corner polygon, in crop image-pixel
/// space.
pub cell_bboxes: Vec<Vec<f64>>,
}
/// Extract markdown tables using externally-supplied structure recovery.
///
/// For each input, pairs structure tokens with cell bboxes (rowspan/colspan
/// aware), converts each cell bbox from crop image-pixels into page PDF
/// points, pulls the cell's text from the native PDF, and emits a markdown
/// pipe-table.
///
/// Returns one markdown string per input, in input order.
#[napi]
pub fn extract_tables_with_structure(
buffer: Buffer,
inputs: Vec<TsrTableInputJs>,
) -> Result<Vec<String>> {
let bytes: Vec<u8> = buffer.to_vec();
let parsed = parse_tsr_inputs(&inputs);
catch_panic("extract_tables_with_structure", move || {
pdf_inspector::extract_tables_with_structure_mem(&bytes, &parsed)
.map_err(|e| to_napi_err(e, "extract_tables_with_structure"))
})
}
/// One resolved cell from `extractTablesWithStructureCells`.
#[napi(object)]
pub struct StructuredCellJs {
/// 0-indexed grid row.
pub row: u32,
/// 0-indexed grid column.
pub col: u32,
/// 1 for a normal cell.
pub rowspan: u32,
/// 1 for a normal cell.
pub colspan: u32,
/// `true` when the cell is a `<th>` or sits inside `<thead>`.
pub is_header: bool,
/// Text extracted from the native PDF for this cell (may be empty).
pub text: String,
/// Axis-aligned bbox `[x1, y1, x2, y2]` in page PDF-points, top-left
/// origin. Useful for debug overlays or per-cell post-processing.
pub page_pt_bbox: Vec<f64>,
}
/// Extract structured cells using externally-supplied structure recovery.
///
/// Lower-level sibling of [`extractTablesWithStructure`]: instead of
/// rendering markdown, returns the resolved cells (row, col, rowspan,
/// colspan, isHeader, text, pagePtBbox) so callers can drive their own
/// rendering, debug overlays, or per-cell post-processing.
///
/// Returns one `Array<StructuredCellJs>` per input, in input order.
#[napi]
pub fn extract_tables_with_structure_cells(
buffer: Buffer,
inputs: Vec<TsrTableInputJs>,
) -> Result<Vec<Vec<StructuredCellJs>>> {
let bytes: Vec<u8> = buffer.to_vec();
let parsed = parse_tsr_inputs(&inputs);
catch_panic("extract_tables_with_structure_cells", move || {
let result = pdf_inspector::extract_tables_with_structure_cells_mem(&bytes, &parsed)
.map_err(|e| to_napi_err(e, "extract_tables_with_structure_cells"))?;
Ok(result
.into_iter()
.map(|cells| {
cells
.into_iter()
.map(|c| StructuredCellJs {
row: c.row as u32,
col: c.col as u32,
rowspan: c.rowspan as u32,
colspan: c.colspan as u32,
is_header: c.is_header,
text: c.text,
page_pt_bbox: c.page_pt_bbox.iter().map(|v| *v as f64).collect(),
})
.collect()
})
.collect())
})
}
/// One result from `extractTablesWithStructureAuto` — markdown plus a
/// diagnostic flag identifying which path produced it.
///
/// `fallbackReason` is `null` when the TSR-hybrid path produced the
/// markdown directly. When stage 1's quality check fires (the cells
/// look like a SLANet detection pathology — phantom rows or multi-row
/// content in a single cell), the auto path may expand the TSR cells
/// in-place or run the heuristic table extractor on the same region.
/// `fallbackReason` carries the diagnostic label (for example
/// `"multi_row_in_cell_expanded"` or `"phantom_empty_row"`).
#[napi(object)]
pub struct TableExtractionResultJs {
pub markdown: String,
pub fallback_reason: Option<String>,
}
/// Auto-fallback variant of [`extractTablesWithStructure`].
///
/// Runs the TSR-hybrid path, checks the resulting cells for known
/// SLANet detection pathologies, expands multi-row cells in-place when
/// possible, and otherwise falls back to the heuristic
/// `extractTablesInRegions` for inputs where the TSR path looks
/// compromised.
///
/// On clean inputs this returns identical markdown to
/// `extractTablesWithStructure`; on flagged inputs `fallbackReason` is
/// set to the recovery path that produced the result.
#[napi]
pub fn extract_tables_with_structure_auto(
buffer: Buffer,
inputs: Vec<TsrTableInputJs>,
) -> Result<Vec<TableExtractionResultJs>> {
let bytes: Vec<u8> = buffer.to_vec();
let parsed = parse_tsr_inputs(&inputs);
catch_panic("extract_tables_with_structure_auto", move || {
let result = pdf_inspector::extract_tables_with_structure_auto_mem(&bytes, &parsed)
.map_err(|e| to_napi_err(e, "extract_tables_with_structure_auto"))?;
Ok(result
.into_iter()
.map(|r| TableExtractionResultJs {
markdown: r.markdown,
fallback_reason: r.fallback_reason,
})
.collect())
})
}
fn parse_tsr_inputs(inputs: &[TsrTableInputJs]) -> Vec<pdf_inspector::TsrTableInput> {
inputs
.iter()
.map(|i| {
let crop = if i.crop_pdf_pt_bbox.len() == 4 {
[
i.crop_pdf_pt_bbox[0] as f32,
i.crop_pdf_pt_bbox[1] as f32,
i.crop_pdf_pt_bbox[2] as f32,
i.crop_pdf_pt_bbox[3] as f32,
]
} else {
[0.0, 0.0, 0.0, 0.0]
};
let cell_bboxes: Vec<Vec<f32>> = i
.cell_bboxes
.iter()
.map(|bb| bb.iter().map(|v| *v as f32).collect())
.collect();
pdf_inspector::TsrTableInput {
page: i.page,
crop_pdf_pt_bbox: crop,
render_dpi: i.render_dpi as f32,
structure_tokens: i.structure_tokens.clone(),
cell_bboxes,
}
})
.collect()
}
/// Per-page markdown extraction result.
#[napi(object)]
pub struct PageMarkdownResult {
@@ -360,9 +597,8 @@ pub fn extract_pages_markdown(
) -> Result<PagesExtractionResult> {
let bytes: Vec<u8> = buffer.to_vec();
catch_panic("extract_pages_markdown", move || {
let result =
pdf_inspector::extract_pages_markdown_mem(&bytes, pages.as_deref())
.map_err(|e| to_napi_err(e, "extract_pages_markdown"))?;
let result = pdf_inspector::extract_pages_markdown_mem(&bytes, pages.as_deref())
.map_err(|e| to_napi_err(e, "extract_pages_markdown"))?;
Ok(PagesExtractionResult {
pages: result
.pages
+12
View File
@@ -7,6 +7,7 @@ import {
extractText,
extractTextWithPositions,
extractTextInRegions,
detectVectorGridInRegion,
extractPagesMarkdown,
} from './index.js';
@@ -90,6 +91,17 @@ assert.equal(typeof regionResults[0].regions[0].text, 'string');
assert.equal(typeof regionResults[0].regions[0].needsOcr, 'boolean');
console.log(' extractTextInRegions: OK');
// --- detectVectorGridInRegion ---
console.log('Testing detectVectorGridInRegion...');
const vectorGrid = detectVectorGridInRegion(fixture, 0, [0, 0, 600, 800], 72);
assert.ok(vectorGrid === null || typeof vectorGrid === 'object');
if (vectorGrid) {
assert.ok(Array.isArray(vectorGrid.structureTokens));
assert.ok(Array.isArray(vectorGrid.cellBboxes));
assert.ok(vectorGrid.cellBboxes.every(bbox => Array.isArray(bbox) && bbox.length === 4));
}
console.log(' detectVectorGridInRegion: OK');
// --- extractPagesMarkdown ---
console.log('Testing extractPagesMarkdown...');
+33 -11
View File
@@ -1,8 +1,12 @@
//! CLI tool for detecting PDF type (text-based vs scanned)
use pdf_inspector::{detect_pdf_type, process_pdf_with_options, PdfOptions, PdfType, ProcessMode};
use pdf_inspector::{
detect_pdf_type, detector::estimate_page_count_from_bytes, process_pdf_with_options,
PdfOptions, PdfType, ProcessMode,
};
use std::env;
use std::fmt::Write;
use std::fs;
use std::process;
use std::time::Instant;
@@ -64,6 +68,32 @@ fn pdf_type_str(pdf_type: &PdfType) -> &'static str {
}
}
fn page_count_hint(pdf_path: &str) -> Option<u32> {
fs::read(pdf_path)
.ok()
.map(|bytes| estimate_page_count_from_bytes(&bytes))
.filter(|&count| count > 0)
}
fn print_error(e: &pdf_inspector::PdfError, pdf_path: &str, json_output: bool) {
if json_output {
if let Some(count) = page_count_hint(pdf_path) {
println!(
r#"{{"error":"{}","page_count_hint":{}}}"#,
json_escape(&e.to_string()),
count
);
} else {
println!(r#"{{"error":"{}"}}"#, json_escape(&e.to_string()));
}
} else {
eprintln!("Error: {}", e);
if let Some(count) = page_count_hint(pdf_path) {
eprintln!("Page count hint: {}", count);
}
}
}
fn run_analyze(pdf_path: &str, json_output: bool, start: Instant) {
match process_pdf_with_options(pdf_path, PdfOptions::new().mode(ProcessMode::Analyze)) {
Ok(result) => {
@@ -135,11 +165,7 @@ fn run_analyze(pdf_path: &str, json_output: bool, start: Instant) {
}
}
Err(e) => {
if json_output {
println!(r#"{{"error":"{}"}}"#, e);
} else {
eprintln!("Error: {}", e);
}
print_error(&e, pdf_path, json_output);
process::exit(1);
}
}
@@ -236,11 +262,7 @@ fn run_detect_only(pdf_path: &str, json_output: bool, start: Instant) {
}
}
Err(e) => {
if json_output {
println!(r#"{{"error":"{}"}}"#, e);
} else {
eprintln!("Error: {}", e);
}
print_error(&e, pdf_path, json_output);
process::exit(1);
}
}
+58 -36
View File
@@ -97,26 +97,9 @@ pub fn detect_pdf_type_with_config<P: AsRef<Path>>(
) -> Result<PdfTypeResult, PdfError> {
crate::validate_pdf_file(&path)?;
// First, load metadata only (fast operation)
let metadata = match Document::load_metadata(&path) {
Ok(m) => m,
Err(ref e) if crate::is_encrypted_lopdf_error(e) => {
Document::load_metadata_with_password(&path, "")?
}
Err(e) => return Err(e.into()),
};
let (doc, page_count) = crate::load_document_from_path(&path)?;
// Then load the full document for content inspection
// We use filtered loading to skip heavy objects we don't need
let doc = match Document::load(&path) {
Ok(d) => d,
Err(ref e) if crate::is_encrypted_lopdf_error(e) => {
Document::load_with_password(&path, "")?
}
Err(e) => return Err(e.into()),
};
detect_from_document(&doc, metadata.page_count, &config)
detect_from_document(&doc, page_count, &config)
}
/// Detect PDF type from memory buffer
@@ -131,25 +114,64 @@ pub fn detect_pdf_type_mem_with_config(
) -> Result<PdfTypeResult, PdfError> {
crate::validate_pdf_bytes(buffer)?;
// Load metadata first (fast)
let metadata = match Document::load_metadata_mem(buffer) {
Ok(m) => m,
Err(ref e) if crate::is_encrypted_lopdf_error(e) => {
Document::load_metadata_mem_with_password(buffer, "")?
}
Err(e) => return Err(e.into()),
};
let (doc, page_count) = crate::load_document_from_mem(buffer)?;
// Load document for inspection
let doc = match Document::load_mem(buffer) {
Ok(d) => d,
Err(ref e) if crate::is_encrypted_lopdf_error(e) => {
Document::load_mem_with_options(buffer, lopdf::LoadOptions::with_password(""))?
}
Err(e) => return Err(e.into()),
};
detect_from_document(&doc, page_count, &config)
}
detect_from_document(&doc, metadata.page_count, &config)
/// Heuristic page-count fallback for malformed PDFs that cannot be parsed.
///
/// This scans raw bytes for page dictionaries (`/Type /Page`) while excluding
/// the page tree node (`/Type /Pages`). It is intended as a low-confidence hint
/// for diagnostics; parsed page-tree counts remain authoritative.
pub fn estimate_page_count_from_bytes(buffer: &[u8]) -> u32 {
let mut count = 0u32;
let mut pos = 0usize;
while let Some(rel_idx) = find_bytes(&buffer[pos..], b"/Type") {
let mut value_pos = pos + rel_idx + b"/Type".len();
value_pos = skip_pdf_whitespace(buffer, value_pos);
if buffer.get(value_pos) == Some(&b'/') {
let name_start = value_pos + 1;
let name_end = name_start + b"Page".len();
if name_end <= buffer.len()
&& &buffer[name_start..name_end] == b"Page"
&& buffer
.get(name_end)
.is_none_or(|b| is_pdf_name_delimiter(*b))
{
count += 1;
}
}
pos += rel_idx + b"/Type".len();
}
count
}
fn find_bytes(haystack: &[u8], needle: &[u8]) -> Option<usize> {
haystack.windows(needle.len()).position(|w| w == needle)
}
fn skip_pdf_whitespace(buffer: &[u8], mut pos: usize) -> usize {
while pos < buffer.len() && is_pdf_whitespace(buffer[pos]) {
pos += 1;
}
pos
}
fn is_pdf_whitespace(byte: u8) -> bool {
matches!(byte, b'\0' | b'\t' | b'\n' | 0x0C | b'\r' | b' ')
}
fn is_pdf_name_delimiter(byte: u8) -> bool {
is_pdf_whitespace(byte)
|| matches!(
byte,
b'(' | b')' | b'<' | b'>' | b'[' | b']' | b'{' | b'}' | b'/' | b'%'
)
}
/// Detection logic on a pre-loaded document.
+4 -28
View File
@@ -36,26 +36,14 @@ pub(crate) use layout::ColumnRegion;
/// Extract text from PDF file as plain string
pub fn extract_text<P: AsRef<Path>>(path: P) -> Result<String, PdfError> {
crate::validate_pdf_file(&path)?;
let doc = match Document::load(&path) {
Ok(d) => d,
Err(ref e) if crate::is_encrypted_lopdf_error(e) => {
Document::load_with_password(&path, "")?
}
Err(e) => return Err(e.into()),
};
let (doc, _) = crate::load_document_from_path(&path)?;
extract_text_from_doc(&doc)
}
/// Extract text from PDF memory buffer
pub fn extract_text_mem(buffer: &[u8]) -> Result<String, PdfError> {
crate::validate_pdf_bytes(buffer)?;
let doc = match Document::load_mem(buffer) {
Ok(d) => d,
Err(ref e) if crate::is_encrypted_lopdf_error(e) => {
Document::load_mem_with_options(buffer, lopdf::LoadOptions::with_password(""))?
}
Err(e) => return Err(e.into()),
};
let (doc, _) = crate::load_document_from_mem(buffer)?;
extract_text_from_doc(&doc)
}
@@ -91,13 +79,7 @@ pub(crate) fn extract_text_with_positions_and_rects<P: AsRef<Path>>(
page_filter: Option<&HashSet<u32>>,
) -> Result<PageExtraction, PdfError> {
crate::validate_pdf_file(&path)?;
let doc = match Document::load(&path) {
Ok(d) => d,
Err(ref e) if crate::is_encrypted_lopdf_error(e) => {
Document::load_with_password(&path, "")?
}
Err(e) => return Err(e.into()),
};
let (doc, _) = crate::load_document_from_path(&path)?;
let font_cmaps = FontCMaps::from_doc(&doc);
let (extraction, _thresholds, _gid_pages) =
extract_positioned_text_from_doc(&doc, &font_cmaps, page_filter)?;
@@ -124,13 +106,7 @@ pub(crate) fn extract_text_with_positions_mem_and_rects(
page_filter: Option<&HashSet<u32>>,
) -> Result<PageExtraction, PdfError> {
crate::validate_pdf_bytes(buffer)?;
let doc = match Document::load_mem(buffer) {
Ok(d) => d,
Err(ref e) if crate::is_encrypted_lopdf_error(e) => {
Document::load_mem_with_options(buffer, lopdf::LoadOptions::with_password(""))?
}
Err(e) => return Err(e.into()),
};
let (doc, _) = crate::load_document_from_mem(buffer)?;
let font_cmaps = FontCMaps::from_doc(&doc);
let (extraction, _thresholds, _gid_pages) =
extract_positioned_text_from_doc(&doc, &font_cmaps, page_filter)?;
+2362 -8
View File
File diff suppressed because it is too large Load Diff
+3 -331
View File
@@ -14,243 +14,15 @@ mod preprocess;
pub use convert::to_markdown_from_lines;
use std::collections::{BTreeMap, HashMap, HashSet};
use std::collections::{HashMap, HashSet};
use crate::extractor::group_into_lines_with_thresholds;
use crate::tables::Table;
use crate::types::{PdfLine, PdfRect, TextItem};
use analysis::calculate_font_stats_from_items;
use classify::{format_list_item, is_code_like, is_list_item};
use convert::{merge_continuation_tables, to_markdown_from_lines_with_tables_and_images};
/// Promote unclaimed header items sitting directly above a struct-tree table
/// into the table's first row.
///
/// Some tagged PDFs mark only the data rows under `<Table>` / `<TR>` / `<TD>`,
/// leaving the visible column headers above the grid untagged. Those headers
/// then slip through to the heuristic fallback and reassemble into a spurious
/// mini-table next to the real one. Instead, gather unclaimed items within a
/// small vertical gap above the detected table's top and — if they align to
/// the existing column centers — fold them into the table as the header row.
fn promote_implicit_header_rows(mut table: Table, band_items: &[TextItem]) -> Table {
if table.cells.is_empty() || table.columns.is_empty() || table.item_indices.is_empty() {
return table;
}
let claimed: HashSet<usize> = table.item_indices.iter().copied().collect();
let mut x_min = f32::INFINITY;
let mut x_max = f32::NEG_INFINITY;
let mut y_top = f32::NEG_INFINITY;
for &idx in &table.item_indices {
if let Some(item) = band_items.get(idx) {
x_min = x_min.min(item.x);
x_max = x_max.max(item.x + item.width);
y_top = y_top.max(item.y + item.height);
}
}
if !y_top.is_finite() || !x_min.is_finite() {
return table;
}
let num_cols = table.columns.len();
let table_width = (x_max - x_min).max(1.0);
// Require each header item to sit within half a column's nominal width
// of a struct-tree column center. Any farther and we cannot be
// confident which column the text belongs to — e.g. a standalone
// section label above the table would otherwise be force-fit into
// the nearest column.
let col_tolerance = (table_width / num_cols as f32) * 0.5;
let max_gap = 60.0;
// Candidate header items: unclaimed, directly above table, within x span.
let mut candidates: Vec<(usize, &TextItem)> = band_items
.iter()
.enumerate()
.filter(|(i, item)| {
if claimed.contains(i) {
return false;
}
let above = item.y >= y_top - 2.0 && item.y < y_top + max_gap;
let within_x = item.x + item.width >= x_min - 5.0 && item.x <= x_max + 5.0;
above && within_x
})
.collect();
if candidates.len() < 2 {
return table;
}
// Group candidates into visual rows by Y (bucket within ~3pt).
candidates.sort_by(|a, b| b.1.y.total_cmp(&a.1.y)); // top-to-bottom (descending Y)
let mut rows_by_y: Vec<Vec<(usize, &TextItem)>> = Vec::new();
let row_bucket = 3.0;
for (idx, item) in candidates {
if let Some(last) = rows_by_y.last_mut() {
if let Some((_, ref_item)) = last.first() {
if (ref_item.y - item.y).abs() <= row_bucket {
last.push((idx, item));
continue;
}
}
}
rows_by_y.push(vec![(idx, item)]);
}
// Assign each item in each row to its nearest column. Skip items that are
// too far from any column center — they are likely unrelated captions,
// page titles, or body text that happens to sit near the table.
let mut header_rows: Vec<Vec<String>> = Vec::new();
let mut header_indices: Vec<usize> = Vec::new();
for row in &rows_by_y {
let mut cells_by_col: BTreeMap<usize, Vec<&str>> = BTreeMap::new();
let mut row_indices: Vec<usize> = Vec::new();
let mut dropped = 0usize;
for (idx, item) in row {
let text = item.text.trim();
// Skip stray punctuation or otherwise content-free fragments —
// a lone "." or "-" left over from the prior paragraph should
// never be promoted into a header cell.
if text.is_empty() || !text.chars().any(|c| c.is_alphanumeric()) {
continue;
}
// The struct-tree column positions are left edges (min X of the
// column's cell items), so compare against the header item's
// left edge too. Using the item center would penalise long
// wrapped phrases even when they start exactly at a column.
let cx = item.x;
let (col, dist) = table
.columns
.iter()
.enumerate()
.map(|(c, &cv)| (c, (cv - cx).abs()))
.min_by(|a, b| a.1.total_cmp(&b.1))
.expect("columns is non-empty");
if dist > col_tolerance {
dropped += 1;
continue;
}
cells_by_col.entry(col).or_default().push(text);
row_indices.push(*idx);
}
// Drop rows that have no aligned cells or where most items are outliers.
if cells_by_col.is_empty() || dropped > row_indices.len() {
continue;
}
let mut row_cells = vec![String::new(); num_cols];
for (col, parts) in cells_by_col {
let joined: String = parts
.into_iter()
.filter(|s| !s.is_empty())
.collect::<Vec<_>>()
.join(" ");
row_cells[col] = joined;
}
header_rows.push(row_cells);
header_indices.extend(row_indices);
}
if header_rows.is_empty() {
return table;
}
// Coalesce the discovered header rows into a single header row: each
// column's text is concatenated top-to-bottom across the wrapped lines.
// A real table header occupies one logical row even when visually wrapped.
let mut merged = vec![String::new(); num_cols];
for row in &header_rows {
for (c, cell) in row.iter().enumerate() {
if cell.is_empty() {
continue;
}
if !merged[c].is_empty() {
merged[c].push(' ');
}
merged[c].push_str(cell);
}
}
// Require ≥ 75% of columns to be filled before accepting the header.
// Real column headers label most or all columns; stray page headers,
// dates, and captions typically land on only one or two columns, so a
// lower threshold lets a single mis-aligned item drag in unrelated
// text as a first row.
let filled_cols = merged.iter().filter(|c| !c.trim().is_empty()).count();
if filled_cols < 2 || filled_cols * 4 < num_cols * 3 {
return table;
}
// Reject when any cell looks like a caption marker ("Table 1",
// "Appendix Table A2", "Figure 2", "Chart 3"). These are labels sitting
// above the grid, not column headers — folding them into the first row
// moves a caption out of the document flow and into the table itself.
let looks_like_caption = merged.iter().any(|cell| {
let t = cell.trim();
if classify::is_caption_line(t) {
return true;
}
// "Appendix Table N" / "Appendix Figure N" share the same semantics
// but have an extra word prefix, so handle them explicitly.
let lower = t.to_ascii_lowercase();
if let Some(rest) = lower.strip_prefix("appendix ") {
let rest = rest.trim_start();
if rest.starts_with("table ")
|| rest.starts_with("figure ")
|| rest.starts_with("chart ")
{
return true;
}
}
false
});
if looks_like_caption {
return table;
}
// Reject when the merged "header" looks like prose or stray page
// elements: a cell that starts mid-sentence (lowercase), runs long
// (>12 words), or is suspiciously short (<3 chars, e.g. a stray "."
// left behind after a sentence). Real headers are short labels.
let looks_like_non_header = merged.iter().any(|cell| {
let t = cell.trim();
if t.is_empty() {
return false;
}
if t.len() < 3 {
return true;
}
if t.chars().next().is_some_and(|c| c.is_ascii_lowercase()) {
return true;
}
t.split_whitespace().count() > 12
});
if looks_like_non_header {
return table;
}
// Reject when two filled cells hold identical text. Real column headers
// label distinct columns, so duplicates are a sign we grabbed a page
// title or event banner whose words happen to align across columns.
let mut filled: Vec<&str> = merged
.iter()
.map(|c| c.trim())
.filter(|c| !c.is_empty())
.collect();
filled.sort();
if filled.windows(2).any(|w| w[0] == w[1]) {
return table;
}
// Prepend the header row. rows/columns are centers — add an approximate
// Y above the current top row so downstream ordering stays correct.
table.cells.insert(0, merged);
let approx_row_height = 14.0;
let first_row_y = table.rows.first().copied().unwrap_or(y_top);
table.rows.insert(0, first_row_y + approx_row_height);
table.item_indices.extend(header_indices);
table
}
/// Detect side-by-side table layout by finding a significant X-position gap.
///
/// Returns X-band boundaries `[(x_min, split_x), (split_x, x_max)]` when a
@@ -927,12 +699,11 @@ pub(crate) fn to_markdown_from_items_with_rects_and_lines(
// through to geometry detection which sees all items.
if !struct_tables.is_empty() {
let st_tables = detect_tables_from_struct_tree(band_items, struct_tables, page);
for table in st_tables {
for table in &st_tables {
let coverage = table.item_indices.len() as f32 / band_items.len().max(1) as f32;
if coverage < 0.5 {
continue;
}
let table = promote_implicit_header_rows(table, band_items);
for &idx in &table.item_indices {
rect_claimed.insert(idx);
if let Some(&page_idx) = band_index_map.get(idx) {
@@ -942,7 +713,7 @@ pub(crate) fn to_markdown_from_items_with_rects_and_lines(
}
}
let table_y = table.rows.first().copied().unwrap_or(0.0);
let table_md = table_to_markdown(&table);
let table_md = table_to_markdown(table);
page_tables
.entry(page)
.or_default()
@@ -1475,105 +1246,6 @@ mod tests {
assert!(split_from_hint_regions(&items, &rects, 1).is_empty());
}
#[test]
fn promote_implicit_header_rows_merges_wrapped_header() {
// Struct-tree tagging often covers only the data rows and leaves the
// column-header row above the grid untagged. Simulate a 3-row 3-col
// data-only table with 3 wrapped header items above each column,
// aligned to the column left-edges. After promotion, the cells grid
// must carry a header row that concatenates each column's wrapped
// lines.
let make_item_at = |x: f32, y: f32, text: &str| {
let mut it = make_item(x, y, 1);
it.text = text.to_string();
it.width = 60.0;
it
};
let mut items = vec![
// Data row 1 at y=200
make_item_at(100.0, 200.0, "r1c1"),
make_item_at(250.0, 200.0, "r1c2"),
make_item_at(400.0, 200.0, "r1c3"),
// Data row 2 at y=180
make_item_at(100.0, 180.0, "r2c1"),
make_item_at(250.0, 180.0, "r2c2"),
make_item_at(400.0, 180.0, "r2c3"),
// Data row 3 at y=160
make_item_at(100.0, 160.0, "r3c1"),
make_item_at(250.0, 160.0, "r3c2"),
make_item_at(400.0, 160.0, "r3c3"),
];
// Unclaimed header items wrapped onto two visual lines just above
// the data grid.
items.push(make_item_at(100.0, 240.0, "First"));
items.push(make_item_at(250.0, 240.0, "Second"));
items.push(make_item_at(400.0, 240.0, "Third"));
items.push(make_item_at(100.0, 225.0, "column"));
items.push(make_item_at(250.0, 225.0, "column"));
items.push(make_item_at(400.0, 225.0, "column"));
let data_indices: Vec<usize> = (0..9).collect();
let cells = vec![
vec!["r1c1".into(), "r1c2".into(), "r1c3".into()],
vec!["r2c1".into(), "r2c2".into(), "r2c3".into()],
vec!["r3c1".into(), "r3c2".into(), "r3c3".into()],
];
let columns = vec![100.0, 250.0, 400.0];
let rows = vec![200.0, 180.0, 160.0];
let table = Table::new(columns, rows, cells, data_indices);
let promoted = promote_implicit_header_rows(table, &items);
assert_eq!(promoted.cells.len(), 4, "header row should be prepended");
assert_eq!(
promoted.cells[0],
vec![
"First column".to_string(),
"Second column".to_string(),
"Third column".to_string(),
],
"wrapped headers should merge per-column"
);
assert_eq!(promoted.cells[1], vec!["r1c1", "r1c2", "r1c3"]);
}
#[test]
fn promote_implicit_header_rows_ignores_faraway_items() {
// A standalone section title sitting above the table with no column
// alignment must not be folded in as a (single-cell) header row.
let make_item_at = |x: f32, y: f32, text: &str| {
let mut it = make_item(x, y, 1);
it.text = text.to_string();
it.width = 60.0;
it
};
let mut items = vec![
make_item_at(100.0, 200.0, "r1c1"),
make_item_at(250.0, 200.0, "r1c2"),
make_item_at(100.0, 180.0, "r2c1"),
make_item_at(250.0, 180.0, "r2c2"),
];
// Title far above the table and only in one x position — not a header.
items.push(make_item_at(50.0, 240.0, "Unrelated section title"));
let data_indices: Vec<usize> = (0..4).collect();
let cells = vec![
vec!["r1c1".into(), "r1c2".into()],
vec!["r2c1".into(), "r2c2".into()],
];
let columns = vec![100.0, 250.0];
let rows = vec![200.0, 180.0];
let table = Table::new(columns, rows, cells, data_indices);
let promoted = promote_implicit_header_rows(table, &items);
assert_eq!(
promoted.cells.len(),
2,
"no header promotion when only one item aligns"
);
}
#[test]
fn no_split_label_plus_number_table() {
// Balance sheet layout: text labels on left, numbers on right.
+59 -15
View File
@@ -1587,25 +1587,69 @@ fn detect_row_stripe_table_from_cell_rects(
return None;
}
// Derive columns from text X-position clustering
// Derive columns from text X-position clustering, but prefer rect
// X-edges when they already provide a tighter scaffold. Some PDFs draw
// only the row-index cells in the body plus a full header row; that is
// not dense enough for `try_build_grid`, but the header rects still define
// the real columns. Text starts inside wide cells can otherwise split the
// table into spurious sub-columns.
let columns = cluster_x_positions(&page_items, 15.0);
if columns.len() < 2 {
let text_col_edges = if columns.len() >= 2 {
let mut edges: Vec<f32> = Vec::with_capacity(columns.len() + 1);
let min_x = page_items.iter().map(|(_, i)| i.x).reduce(f32::min)?;
edges.push(min_x - 5.0);
for pair in columns.windows(2) {
edges.push((pair[0] + pair[1]) / 2.0);
}
let max_x_right = page_items
.iter()
.map(|(_, i)| i.x + i.width)
.reduce(f32::max)?;
edges.push(max_x_right + 5.0);
Some(edges)
} else {
None
};
let rect_col_edges = {
let mut x_vals = Vec::with_capacity(content_rects.len() * 2);
for &&(x, _, w, _) in &content_rects {
x_vals.push(x);
x_vals.push(x + w);
}
let mut edges = snap_edges(&x_vals, 6.0);
edges.sort_by(|a, b| a.total_cmp(b));
if (3..=26).contains(&edges.len()) {
Some(edges)
} else {
None
}
};
let col_edges = match (rect_col_edges, text_col_edges) {
(Some(rect_edges), Some(text_edges)) if rect_edges.len() <= text_edges.len() => {
debug!(
" cell-rect using {} rect-derived columns over {} text clusters",
rect_edges.len() - 1,
text_edges.len() - 1
);
rect_edges
}
(_, Some(text_edges)) => text_edges,
(Some(rect_edges), None) => rect_edges,
(None, None) => {
debug!(
" cell-rect rejected: only {} columns from text clustering",
columns.len()
);
return None;
}
};
if col_edges.len() < 3 {
return None;
}
// Build column edges
let mut col_edges: Vec<f32> = Vec::with_capacity(columns.len() + 1);
let min_x = page_items.iter().map(|(_, i)| i.x).reduce(f32::min)?;
col_edges.push(min_x - 5.0);
for pair in columns.windows(2) {
col_edges.push((pair[0] + pair[1]) / 2.0);
}
let max_x_right = page_items
.iter()
.map(|(_, i)| i.x + i.width)
.reduce(f32::max)?;
col_edges.push(max_x_right + 5.0);
let num_cols = col_edges.len() - 1;
let num_rows = row_edges.len() - 1;
+847 -63
View File
@@ -4,15 +4,385 @@
//! elements linked to MCIDs, this module builds `Table` structs directly from
//! the semantic hierarchy — no geometry heuristics needed.
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
use log::debug;
use crate::structure_tree::StructTable;
use crate::structure_tree::{StructTable, StructTableRow};
use crate::types::TextItem;
use super::Table;
#[derive(Debug, Clone)]
struct MatchedCell {
text: String,
item_indices: Vec<usize>,
x: Option<f32>,
y: Option<f32>,
}
fn legacy_column_positions(
page_rows: &[&StructTableRow],
mcid_to_items: &HashMap<i64, Vec<usize>>,
items: &[TextItem],
page: u32,
num_cols: usize,
) -> Vec<f32> {
let mut col_positions: Vec<f32> = vec![0.0; num_cols];
for (col, col_pos) in col_positions.iter_mut().enumerate() {
for row in page_rows {
if col < row.cells.len() {
if let Some(x) = row.cells[col]
.mcids
.iter()
.filter(|(_, p)| *p == page)
.filter_map(|(mcid, _)| mcid_to_items.get(mcid))
.flatten()
.map(|&idx| items[idx].x)
.reduce(f32::min)
{
*col_pos = x;
break;
}
}
}
}
col_positions
}
fn infer_column_positions(
raw_rows: &[Vec<MatchedCell>],
fallback_positions: &[f32],
num_cols: usize,
) -> Vec<f32> {
const SAME_COLUMN_TOLERANCE: f32 = 18.0;
let mut anchors = raw_rows
.iter()
.max_by_key(|row| row.iter().filter(|cell| cell.x.is_some()).count())
.map(|row| row.iter().filter_map(|cell| cell.x).collect::<Vec<_>>())
.unwrap_or_default();
if anchors.len() > num_cols {
anchors.truncate(num_cols);
}
let mut additional_positions: Vec<f32> = raw_rows
.iter()
.flat_map(|row| row.iter().filter_map(|cell| cell.x))
.collect();
additional_positions.sort_by(|a, b| a.total_cmp(b));
for x in additional_positions {
if anchors.len() >= num_cols {
break;
}
if anchors
.iter()
.all(|existing| (x - *existing).abs() > SAME_COLUMN_TOLERANCE)
{
anchors.push(x);
anchors.sort_by(|a, b| a.total_cmp(b));
}
}
if anchors.len() < num_cols {
for &x in fallback_positions {
if anchors.len() >= num_cols {
break;
}
if anchors
.iter()
.all(|existing| (x - *existing).abs() > SAME_COLUMN_TOLERANCE)
{
anchors.push(x);
anchors.sort_by(|a, b| a.total_cmp(b));
}
}
}
if anchors.is_empty() {
return fallback_positions.to_vec();
}
while anchors.len() < num_cols {
anchors.push(*anchors.last().unwrap());
}
anchors
}
fn align_positions_to_columns(cell_xs: &[f32], columns: &[f32]) -> Vec<usize> {
if cell_xs.is_empty() || columns.is_empty() {
return Vec::new();
}
if cell_xs.len() >= columns.len() {
return (0..cell_xs.len().min(columns.len())).collect();
}
let mut dp = vec![vec![f32::INFINITY; columns.len() + 1]; cell_xs.len() + 1];
let mut take = vec![vec![false; columns.len() + 1]; cell_xs.len() + 1];
for value in &mut dp[0] {
*value = 0.0;
}
for i in 1..=cell_xs.len() {
for j in 1..=columns.len() {
let skip_cost = dp[i][j - 1];
let take_cost = dp[i - 1][j - 1] + (cell_xs[i - 1] - columns[j - 1]).abs();
if take_cost <= skip_cost {
dp[i][j] = take_cost;
take[i][j] = true;
} else {
dp[i][j] = skip_cost;
}
}
}
let mut assignments_rev = Vec::with_capacity(cell_xs.len());
let mut i = cell_xs.len();
let mut j = columns.len();
while i > 0 && j > 0 {
if take[i][j] {
assignments_rev.push(j - 1);
i -= 1;
j -= 1;
} else {
j -= 1;
}
}
assignments_rev.reverse();
assignments_rev
}
fn align_struct_rows(
raw_rows: &[Vec<MatchedCell>],
col_positions: &[f32],
) -> (Vec<Vec<String>>, Vec<f32>, Vec<usize>) {
let mut cells: Vec<Vec<String>> = Vec::with_capacity(raw_rows.len());
let mut row_positions: Vec<f32> = Vec::with_capacity(raw_rows.len());
let mut all_item_indices: Vec<usize> = Vec::new();
for row in raw_rows {
let present_cells: Vec<&MatchedCell> = row
.iter()
.filter(|cell| {
!cell.item_indices.is_empty() || !cell.text.is_empty() || cell.x.is_some()
})
.collect();
let cell_xs: Vec<f32> = present_cells.iter().filter_map(|cell| cell.x).collect();
let assignments = if cell_xs.len() == present_cells.len() {
align_positions_to_columns(&cell_xs, col_positions)
} else {
(0..present_cells.len().min(col_positions.len())).collect()
};
let mut row_cells = vec![String::new(); col_positions.len()];
for (cell, &col_idx) in present_cells.iter().zip(assignments.iter()) {
if !cell.text.is_empty() {
if !row_cells[col_idx].is_empty() {
row_cells[col_idx].push(' ');
}
row_cells[col_idx].push_str(&cell.text);
}
all_item_indices.extend(cell.item_indices.iter().copied());
}
let row_y = row
.iter()
.filter_map(|cell| cell.y)
.reduce(f32::max)
.unwrap_or(0.0);
cells.push(row_cells);
row_positions.push(row_y);
}
(cells, row_positions, all_item_indices)
}
fn left_align_struct_rows(
raw_rows: &[Vec<MatchedCell>],
num_cols: usize,
) -> (Vec<Vec<String>>, Vec<f32>, Vec<usize>) {
let mut cells: Vec<Vec<String>> = Vec::with_capacity(raw_rows.len());
let mut row_positions: Vec<f32> = Vec::with_capacity(raw_rows.len());
let mut all_item_indices: Vec<usize> = Vec::new();
for row in raw_rows {
let mut row_cells: Vec<String> = row.iter().map(|cell| cell.text.clone()).collect();
row_cells.truncate(num_cols);
while row_cells.len() < num_cols {
row_cells.push(String::new());
}
cells.push(row_cells);
all_item_indices.extend(
row.iter()
.flat_map(|cell| cell.item_indices.iter().copied()),
);
row_positions.push(
row.iter()
.filter_map(|cell| cell.y)
.reduce(f32::max)
.unwrap_or(0.0),
);
}
(cells, row_positions, all_item_indices)
}
fn recover_unclaimed_header_row(table: &mut Table, items: &[TextItem], has_ragged_rows: bool) {
if !has_ragged_rows || table.rows.is_empty() || table.columns.len() < 3 {
return;
}
const MAX_HEADER_DISTANCE: f32 = 90.0;
const MAX_GAP_TO_TABLE: f32 = 35.0;
const MAX_INTER_HEADER_GAP: f32 = 25.0;
const MAX_HEADER_ROWS: usize = 3;
const Y_TOLERANCE: f32 = 5.0;
let top_row_y = table.rows[0];
let x_min = table.columns.first().copied().unwrap_or(0.0) - 25.0;
let x_max = table.columns.last().copied().unwrap_or(0.0) + 120.0;
let claimed: HashSet<usize> = table.item_indices.iter().copied().collect();
let mut candidate_rows: Vec<(f32, Vec<(usize, &TextItem)>)> = Vec::new();
for (idx, item) in items.iter().enumerate() {
if claimed.contains(&idx)
|| item.text.trim().is_empty()
|| item.y <= top_row_y
|| item.y - top_row_y > MAX_HEADER_DISTANCE
|| item.x < x_min
|| item.x > x_max
{
continue;
}
if let Some((_, row_items)) = candidate_rows
.iter_mut()
.find(|(row_y, _)| (item.y - *row_y).abs() < Y_TOLERANCE)
{
row_items.push((idx, item));
} else {
candidate_rows.push((item.y, vec![(idx, item)]));
}
}
if candidate_rows.is_empty() {
return;
}
for (_, row_items) in &mut candidate_rows {
row_items.sort_by(|a, b| a.1.x.total_cmp(&b.1.x));
}
candidate_rows.sort_by(|a, b| a.0.total_cmp(&b.0));
if candidate_rows[0].0 - top_row_y > MAX_GAP_TO_TABLE {
return;
}
let mut candidate_iter = candidate_rows.into_iter();
let Some(first_row) = candidate_iter.next() else {
return;
};
let mut selected_rows: Vec<(f32, Vec<(usize, &TextItem)>)> = vec![first_row];
let mut prev_y = selected_rows[0].0;
for (row_y, row_items) in candidate_iter {
if selected_rows.len() >= MAX_HEADER_ROWS {
break;
}
if row_y - prev_y > MAX_INTER_HEADER_GAP {
break;
}
prev_y = row_y;
selected_rows.push((row_y, row_items));
}
if selected_rows.is_empty() {
return;
}
let mut assigned_rows: Vec<(f32, Vec<String>, Vec<usize>)> = Vec::new();
let mut closest_row_populated = 0usize;
let mut combined_cols: HashSet<usize> = HashSet::new();
for (row_idx, (row_y, row_items)) in selected_rows.iter().enumerate() {
if row_items.len() > table.columns.len() {
return;
}
let row_xs: Vec<f32> = row_items.iter().map(|(_, item)| item.x).collect();
let assignments = align_positions_to_columns(&row_xs, &table.columns);
if assignments.len() != row_items.len() {
return;
}
let mut row_cells = vec![String::new(); table.columns.len()];
let mut row_indices = Vec::with_capacity(row_items.len());
let mut populated_cols: HashSet<usize> = HashSet::new();
for ((idx, item), &col_idx) in row_items.iter().zip(assignments.iter()) {
let text = item.text.trim();
if text.is_empty() {
continue;
}
if !row_cells[col_idx].is_empty() {
row_cells[col_idx].push(' ');
}
row_cells[col_idx].push_str(text);
row_indices.push(*idx);
populated_cols.insert(col_idx);
}
if row_idx == 0 {
closest_row_populated = populated_cols.len();
}
combined_cols.extend(populated_cols.iter().copied());
assigned_rows.push((*row_y, row_cells, row_indices));
}
let required_cols = if table.columns.len() <= 4 {
table.columns.len()
} else {
table.columns.len() - 1
};
if closest_row_populated < 2 || combined_cols.len() < required_cols {
return;
}
let mut header_cells = vec![String::new(); table.columns.len()];
let mut header_indices = Vec::new();
for (_, row_cells, row_indices) in assigned_rows.iter().rev() {
for (col_idx, cell_text) in row_cells.iter().enumerate() {
if cell_text.is_empty() {
continue;
}
if !header_cells[col_idx].is_empty() {
header_cells[col_idx].push(' ');
}
header_cells[col_idx].push_str(cell_text);
}
header_indices.extend(row_indices.iter().copied());
}
table.rows.insert(
0,
assigned_rows
.iter()
.map(|(row_y, _, _)| *row_y)
.reduce(f32::max)
.unwrap_or(top_row_y),
);
table.cells.insert(0, header_cells);
table.item_indices.extend(header_indices);
table.item_indices.sort_unstable();
table.item_indices.dedup();
}
/// Build tables from structure-tree table descriptors by matching MCIDs to TextItems.
///
/// Returns tables for the given page. Tables where fewer than 50% of cells
@@ -67,18 +437,14 @@ pub fn detect_tables_from_struct_tree(
continue;
}
// Build cell text and collect item indices
let mut cells: Vec<Vec<String>> = Vec::new();
let mut all_item_indices: Vec<usize> = Vec::new();
// Build cell text and geometry for alignment and header recovery.
let mut raw_rows: Vec<Vec<MatchedCell>> = Vec::new();
let mut total_cells = 0u32;
let mut matched_cells = 0u32;
for row in &page_rows {
let mut row_cells = Vec::with_capacity(num_cols);
for (col_idx, cell) in row.cells.iter().enumerate() {
if col_idx >= num_cols {
break;
}
let mut row_cells = Vec::with_capacity(row.cells.len());
for cell in &row.cells {
total_cells += 1;
// Collect all items for this cell's MCIDs
@@ -115,18 +481,18 @@ pub fn detect_tables_from_struct_tree(
.collect::<Vec<_>>()
.join(" ");
for (idx, _) in &cell_items {
all_item_indices.push(*idx);
}
let item_indices = cell_items.iter().map(|(idx, _)| *idx).collect::<Vec<_>>();
let x = cell_items.iter().map(|(_, item)| item.x).reduce(f32::min);
let y = cell_items.iter().map(|(_, item)| item.y).reduce(f32::max);
row_cells.push(text);
row_cells.push(MatchedCell {
text,
item_indices,
x,
y,
});
}
// Pad to num_cols
while row_cells.len() < num_cols {
row_cells.push(String::new());
}
cells.push(row_cells);
raw_rows.push(row_cells);
}
// Reject if too few cells matched (stale structure tree)
@@ -148,52 +514,55 @@ pub fn detect_tables_from_struct_tree(
continue;
}
// Derive row/column positions from item geometry
let mut row_positions: Vec<f32> = Vec::new();
for row in &page_rows {
let y = row
.cells
.iter()
.flat_map(|c| c.mcids.iter())
.filter(|(_, p)| *p == page)
.filter_map(|(mcid, _)| mcid_to_items.get(mcid))
.flatten()
.map(|&idx| items[idx].y)
.reduce(f32::max)
.unwrap_or(0.0);
row_positions.push(y);
}
let has_ragged_rows = raw_rows
.iter()
.any(|row| row.iter().filter(|cell| cell.x.is_some()).count() < num_cols);
let first_row_has_tagged_header = page_rows.first().is_some_and(|row| {
let header_cells = row.cells.iter().filter(|cell| cell.is_header).count();
header_cells * 2 >= row.cells.len()
});
let fallback_col_positions =
legacy_column_positions(&page_rows, &mcid_to_items, items, page, num_cols);
let (legacy_cells, legacy_row_positions, mut legacy_item_indices) =
left_align_struct_rows(&raw_rows, num_cols);
legacy_item_indices.sort_unstable();
legacy_item_indices.dedup();
let legacy_table = Table::new(
fallback_col_positions.clone(),
legacy_row_positions,
legacy_cells,
legacy_item_indices,
);
// Column positions: use X positions of first non-empty cell in each column
let mut col_positions: Vec<f32> = vec![0.0; num_cols];
for (col, col_pos) in col_positions.iter_mut().enumerate() {
for row in &page_rows {
if col < row.cells.len() {
if let Some(x) = row.cells[col]
.mcids
.iter()
.filter(|(_, p)| *p == page)
.filter_map(|(mcid, _)| mcid_to_items.get(mcid))
.flatten()
.map(|&idx| items[idx].x)
.reduce(f32::min)
{
*col_pos = x;
break;
}
}
}
}
let col_positions = infer_column_positions(&raw_rows, &fallback_col_positions, num_cols);
let (aligned_cells, aligned_row_positions, mut aligned_item_indices) =
align_struct_rows(&raw_rows, &col_positions);
aligned_item_indices.sort_unstable();
aligned_item_indices.dedup();
all_item_indices.sort_unstable();
all_item_indices.dedup();
tables.push(Table::new(
let mut aligned_table = Table::new(
col_positions,
row_positions,
cells,
all_item_indices,
));
aligned_row_positions,
aligned_cells,
aligned_item_indices,
);
let item_count_before_header = aligned_table.item_indices.len();
let row_count_before_header = aligned_table.cells.len();
recover_unclaimed_header_row(
&mut aligned_table,
items,
has_ragged_rows && !first_row_has_tagged_header,
);
let recovered_header = aligned_table.item_indices.len() > item_count_before_header
|| aligned_table.cells.len() > row_count_before_header;
let prefer_aligned = recovered_header;
tables.push(if prefer_aligned {
aligned_table
} else {
legacy_table
});
}
tables
@@ -374,4 +743,419 @@ mod tests {
let tables = detect_tables_from_struct_tree(&items, &struct_tables, 2);
assert_eq!(tables.len(), 1);
}
#[test]
fn realigns_ragged_rows_and_recovers_untagged_header() {
let items = vec![
make_item("Category", 50.0, 120.0, 1, None),
make_item("Potentially", 150.0, 120.0, 1, None),
make_item("Summary", 250.0, 120.0, 1, None),
make_item("Most commonly", 350.0, 120.0, 1, None),
make_item("concerning aspect", 150.0, 110.0, 1, None),
make_item("suggested", 350.0, 110.0, 1, None),
make_item("of circumstances", 150.0, 100.0, 1, None),
make_item("intervention", 350.0, 100.0, 1, None),
make_item("Existence of red-teaming", 150.0, 80.0, 1, Some(10)),
make_item("Important for safety", 250.0, 80.0, 1, Some(11)),
make_item("Ensure welfare interviews", 350.0, 80.0, 1, Some(12)),
make_item("Identity & self-knowledge", 50.0, 60.0, 1, Some(20)),
make_item("Lack of knowledge", 150.0, 60.0, 1, Some(21)),
make_item("Overall negative", 250.0, 60.0, 1, Some(22)),
make_item("Describe training process", 350.0, 60.0, 1, Some(23)),
make_item("Uncertainty around other copies", 150.0, 40.0, 1, Some(30)),
make_item("High uncertainty", 250.0, 40.0, 1, Some(31)),
make_item("No intervention suggested", 350.0, 40.0, 1, Some(32)),
];
let struct_tables = vec![StructTable {
rows: vec![
StructTableRow {
cells: vec![
StructTableCell {
is_header: false,
mcids: vec![(10, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(11, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(12, 1)],
},
],
},
StructTableRow {
cells: vec![
StructTableCell {
is_header: false,
mcids: vec![(20, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(21, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(22, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(23, 1)],
},
],
},
StructTableRow {
cells: vec![
StructTableCell {
is_header: false,
mcids: vec![(30, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(31, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(32, 1)],
},
],
},
],
}];
let tables = detect_tables_from_struct_tree(&items, &struct_tables, 1);
assert_eq!(tables.len(), 1);
let table = &tables[0];
assert_eq!(table.cells.len(), 4);
assert_eq!(
table.cells[0],
vec![
"Category",
"Potentially concerning aspect of circumstances",
"Summary",
"Most commonly suggested intervention",
]
);
assert_eq!(table.cells[1][0], "");
assert_eq!(table.cells[1][1], "Existence of red-teaming");
assert_eq!(table.cells[2][0], "Identity & self-knowledge");
assert_eq!(table.cells[3][0], "");
assert_eq!(table.columns.len(), 4);
assert!(table.columns.windows(2).all(|w| w[0] < w[1]));
assert_eq!(table.item_indices.len(), items.len());
}
#[test]
fn does_not_absorb_caption_above_ragged_struct_table() {
let items = vec![
make_item("Table 5-7: Summary of responses", 50.0, 120.0, 1, None),
make_item("Aspect one", 150.0, 80.0, 1, Some(10)),
make_item("Summary one", 250.0, 80.0, 1, Some(11)),
make_item("Category", 50.0, 60.0, 1, Some(20)),
make_item("Aspect two", 150.0, 60.0, 1, Some(21)),
make_item("Summary two", 250.0, 60.0, 1, Some(22)),
make_item("Aspect three", 150.0, 40.0, 1, Some(30)),
make_item("Summary three", 250.0, 40.0, 1, Some(31)),
];
let struct_tables = vec![StructTable {
rows: vec![
StructTableRow {
cells: vec![
StructTableCell {
is_header: false,
mcids: vec![(10, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(11, 1)],
},
],
},
StructTableRow {
cells: vec![
StructTableCell {
is_header: false,
mcids: vec![(20, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(21, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(22, 1)],
},
],
},
StructTableRow {
cells: vec![
StructTableCell {
is_header: false,
mcids: vec![(30, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(31, 1)],
},
],
},
],
}];
let tables = detect_tables_from_struct_tree(&items, &struct_tables, 1);
assert_eq!(tables.len(), 1);
let table = &tables[0];
assert_eq!(table.cells.len(), 3);
assert!(
table
.cells
.iter()
.flatten()
.all(|cell| !cell.contains("Table 5-7")),
"caption must stay outside the table"
);
assert!(!table.item_indices.contains(&0));
}
#[test]
fn keeps_existing_tagged_header_without_absorbing_intro_or_caption() {
let items = vec![
make_item(
"Eighteen people left other comments regarding the Project.",
50.0,
130.0,
1,
None,
),
make_item("Table 5-1:", 220.0, 130.0, 1, None),
make_item("Other Comments", 350.0, 130.0, 1, None),
make_item("Theme", 50.0, 110.0, 1, Some(10)),
make_item("Specific Concern/Inquiry", 200.0, 110.0, 1, Some(11)),
make_item("Response", 420.0, 110.0, 1, Some(12)),
make_item("Traffic", 50.0, 90.0, 1, Some(20)),
make_item("Road conditions", 200.0, 90.0, 1, Some(21)),
make_item("Maintenance response", 420.0, 90.0, 1, Some(22)),
make_item("Noise", 50.0, 70.0, 1, Some(30)),
make_item("Dust concerns", 200.0, 70.0, 1, Some(31)),
make_item("Mitigation response", 420.0, 70.0, 1, Some(32)),
make_item("Resource Use", 50.0, 50.0, 1, Some(40)),
make_item("Snowmobile trails", 200.0, 50.0, 1, Some(41)),
make_item("Access response", 420.0, 50.0, 1, Some(42)),
];
let struct_tables = vec![StructTable {
rows: vec![
StructTableRow {
cells: vec![
StructTableCell {
is_header: true,
mcids: vec![(10, 1)],
},
StructTableCell {
is_header: true,
mcids: vec![(11, 1)],
},
StructTableCell {
is_header: true,
mcids: vec![(12, 1)],
},
],
},
StructTableRow {
cells: vec![
StructTableCell {
is_header: false,
mcids: vec![(20, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(21, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(22, 1)],
},
],
},
StructTableRow {
cells: vec![
StructTableCell {
is_header: false,
mcids: vec![(30, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(31, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(32, 1)],
},
],
},
StructTableRow {
cells: vec![
StructTableCell {
is_header: false,
mcids: vec![(40, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(41, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(42, 1)],
},
],
},
],
}];
let tables = detect_tables_from_struct_tree(&items, &struct_tables, 1);
assert_eq!(tables.len(), 1);
let table = &tables[0];
assert_eq!(table.cells.len(), 4);
assert_eq!(
table.cells[0],
vec!["Theme", "Specific Concern/Inquiry", "Response"]
);
assert!(!table.item_indices.contains(&0));
assert!(!table.item_indices.contains(&1));
assert!(!table.item_indices.contains(&2));
}
#[test]
fn does_not_recover_header_for_narrow_two_column_table() {
let items = vec![
make_item("Alpha", 50.0, 120.0, 1, None),
make_item("Beta", 200.0, 120.0, 1, None),
make_item("First value", 200.0, 80.0, 1, Some(10)),
make_item("Only labeled row", 50.0, 60.0, 1, Some(20)),
make_item("Second value", 200.0, 60.0, 1, Some(21)),
make_item("Third value", 200.0, 40.0, 1, Some(30)),
];
let struct_tables = vec![StructTable {
rows: vec![
StructTableRow {
cells: vec![StructTableCell {
is_header: false,
mcids: vec![(10, 1)],
}],
},
StructTableRow {
cells: vec![
StructTableCell {
is_header: false,
mcids: vec![(20, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(21, 1)],
},
],
},
StructTableRow {
cells: vec![StructTableCell {
is_header: false,
mcids: vec![(30, 1)],
}],
},
],
}];
let tables = detect_tables_from_struct_tree(&items, &struct_tables, 1);
assert_eq!(tables.len(), 1);
let table = &tables[0];
assert_eq!(table.cells.len(), 3);
assert_eq!(table.cells[0], vec!["First value", ""]);
assert_eq!(table.cells[1], vec!["Only labeled row", "Second value"]);
assert_eq!(table.cells[2], vec!["Third value", ""]);
assert!(!table.item_indices.contains(&0));
assert!(!table.item_indices.contains(&1));
}
#[test]
fn ragged_rows_without_recovered_header_keep_legacy_alignment() {
let items = vec![
make_item("Date", 150.0, 120.0, 1, Some(10)),
make_item("Title", 250.0, 120.0, 1, Some(11)),
make_item("PE", 350.0, 120.0, 1, Some(12)),
make_item("Bidder", 450.0, 120.0, 1, Some(13)),
make_item("Amount", 550.0, 120.0, 1, Some(14)),
make_item("1", 50.0, 100.0, 1, Some(20)),
make_item("8/1", 150.0, 100.0, 1, Some(21)),
make_item("Procurement", 250.0, 100.0, 1, Some(22)),
make_item("PUC", 350.0, 100.0, 1, Some(23)),
make_item("Vendor", 450.0, 100.0, 1, Some(24)),
make_item("SR1", 550.0, 100.0, 1, Some(25)),
];
let struct_tables = vec![StructTable {
rows: vec![
StructTableRow {
cells: vec![
StructTableCell {
is_header: true,
mcids: vec![(10, 1)],
},
StructTableCell {
is_header: true,
mcids: vec![(11, 1)],
},
StructTableCell {
is_header: true,
mcids: vec![(12, 1)],
},
StructTableCell {
is_header: true,
mcids: vec![(13, 1)],
},
StructTableCell {
is_header: true,
mcids: vec![(14, 1)],
},
],
},
StructTableRow {
cells: vec![
StructTableCell {
is_header: false,
mcids: vec![(20, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(21, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(22, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(23, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(24, 1)],
},
StructTableCell {
is_header: false,
mcids: vec![(25, 1)],
},
],
},
],
}];
let tables = detect_tables_from_struct_tree(&items, &struct_tables, 1);
assert_eq!(tables.len(), 1);
let table = &tables[0];
assert_eq!(table.cells[0][0], "Date");
assert_eq!(table.cells[0][4], "Amount");
assert_eq!(table.cells[0][5], "");
}
}
+75
View File
@@ -154,6 +154,12 @@ fn is_dots_only(cell: &str) -> bool {
dots >= 3 && t.chars().all(|c| c == '.' || c.is_whitespace())
}
fn starts_with_uppercase_word(cell: &str) -> bool {
cell.chars()
.find(|c| c.is_alphanumeric())
.is_some_and(|c| c.is_uppercase())
}
/// Clean up table cells: merge continuation rows, extract footnotes, remove empty rows
fn clean_table_cells(cells: &[Vec<String>]) -> (Vec<Vec<String>>, Vec<String>) {
let mut cleaned: Vec<Vec<String>> = Vec::new();
@@ -212,11 +218,20 @@ fn clean_table_cells(cells: &[Vec<String>]) -> (Vec<Vec<String>>, Vec<String>) {
let looks_like_data_row = non_first_cells.len() >= 2
&& avg_cell_len <= 10.0
&& numeric_cells > non_first_cells.len() / 2;
let uppercase_leading_cells = non_first_cells
.iter()
.filter(|cell| starts_with_uppercase_word(cell))
.count();
let looks_like_spanning_first_column_row = first_cell.is_empty()
&& row.len() >= 4
&& non_first_cells.len() == row.len().saturating_sub(1)
&& uppercase_leading_cells >= non_first_cells.len().saturating_sub(1);
// Classic continuation: first cell empty, content in other cells
let is_classic_continuation = first_cell.is_empty()
&& !non_first_cells.is_empty()
&& !is_short_subheader
&& !looks_like_data_row
&& !looks_like_spanning_first_column_row
&& cleaned.len() > 1;
// Wrapped-cell continuation: row has fewer filled cells than the header
@@ -246,6 +261,7 @@ fn clean_table_cells(cells: &[Vec<String>]) -> (Vec<Vec<String>>, Vec<String>) {
&& filled_cells <= max_filled_for_merge
&& prev_filled > filled_cells
&& !looks_like_data_row
&& !looks_like_spanning_first_column_row
&& !is_short_subheader;
let is_continuation = is_classic_continuation || is_wrapped_continuation;
@@ -415,6 +431,65 @@ mod tests {
assert_eq!(cleaned.len(), 3);
}
#[test]
fn test_clean_table_cells_spanning_first_column_row_not_merged() {
let cells = vec![
vec![
"Category".into(),
"Potentially concerning aspect".into(),
"Summary".into(),
"Intervention".into(),
],
vec![
"Identity & self-knowledge".into(),
"Lack of knowledge".into(),
"Overall negative".into(),
"Describe training".into(),
],
vec![
"".into(),
"Uncertainty around other copies".into(),
"High uncertainty".into(),
"No intervention suggested".into(),
],
];
let (cleaned, _) = clean_table_cells(&cells);
assert_eq!(cleaned.len(), 3);
assert_eq!(cleaned[2][0], "");
assert_eq!(cleaned[2][1], "Uncertainty around other copies");
}
#[test]
fn test_clean_table_cells_full_width_continuation_row_still_merges_when_lowercase() {
let cells = vec![
vec![
"Classification".into(),
"Before tax".into(),
"After tax".into(),
"Standard equipment".into(),
"Options".into(),
],
vec![
"Exclusive Special".into(),
"83,500,000".into(),
"79,275,000".into(),
"Standard equipment".into(),
"Option A".into(),
],
vec![
"".into(),
"with 3.5% individual consumption tax applied".into(),
"with 3.5% individual consumption tax applied".into(),
"lighting(crash pad)".into(),
"sound system".into(),
],
];
let (cleaned, _) = clean_table_cells(&cells);
assert_eq!(cleaned.len(), 2);
assert!(cleaned[1][1].contains("83,500,000"));
assert!(cleaned[1][1].contains("with 3.5%"));
}
#[test]
fn test_clean_table_cells_header_row_not_merged() {
// Continuation requires cleaned.len() > 1 (don't merge into header)
+2
View File
@@ -9,6 +9,7 @@ mod detect_struct;
mod financial;
mod format;
mod grid;
pub mod structured;
pub use detect_heuristic::detect_tables;
pub(crate) use detect_heuristic::is_table_of_contents;
@@ -17,6 +18,7 @@ pub(crate) use detect_rects::cluster_rects;
pub use detect_rects::{detect_tables_from_rects, RectHintRegion};
pub use detect_struct::detect_tables_from_struct_tree;
pub use format::table_to_markdown;
pub use structured::{cells_to_markdown, StructuredCell};
use crate::types::TextItem;
+972
View File
@@ -0,0 +1,972 @@
//! Structure-recovery-aware (TSR) table assembly.
//!
//! Consumes the raw output of an external table-structure recognition model
//! (e.g. SLANet on PaddleOCR): a flat list of HTML structure tokens plus a
//! parallel list of per-cell bboxes. Pairs each cell open-tag with its bbox
//! in document order, tracks row/column position with rowspan/colspan
//! awareness, and emits a markdown pipe-table.
//!
//! No real HTML parser is needed — the token grammar is restricted (see
//! [`parse_structure`]), so a small state machine is enough.
//!
//! Cell text is supplied separately by the caller (typically by overlap-
//! testing PDF text items against each cell's page-PDF-pt bbox).
use std::collections::{HashMap, HashSet};
/// A single resolved cell, with both structural metadata and its bbox in
/// page PDF-points (top-left origin).
#[derive(Debug, Clone)]
pub struct StructuredCell {
/// 0-indexed grid row.
pub row: usize,
/// 0-indexed grid column.
pub col: usize,
/// 1 for a normal cell.
pub rowspan: usize,
/// 1 for a normal cell.
pub colspan: usize,
/// `true` when the cell is a `<th>` or sits inside `<thead>`.
pub is_header: bool,
/// Cell text (filled in by the caller after overlap-testing PDF items).
pub text: String,
/// Axis-aligned bbox `[x1, y1, x2, y2]` in page PDF-points, top-left origin.
pub page_pt_bbox: [f32; 4],
}
/// Intermediate parse result before the caller fills in text + page coords.
#[derive(Debug, Clone)]
pub(crate) struct CellSlot {
pub row: usize,
pub col: usize,
pub rowspan: usize,
pub colspan: usize,
pub is_header: bool,
/// Index into the parallel `cell_bboxes` array.
pub bbox_idx: usize,
}
/// Parse a sequence of SLANet structure tokens into ordered cell slots.
///
/// Token grammar (no real HTML parsing required):
/// - Section markers: `<thead>`, `</thead>`, `<tbody>`, `</tbody>` and
/// wrapper tokens (`<html>`, `<body>`, `<table>`, plus closing variants)
/// are tracked or skipped.
/// - Row markers: `<tr>` opens a new row, `</tr>` is informational.
/// - Empty cell, single token: `<td></td>` or `<th></th>`.
/// - Cell with attributes, multi-token sequence: `<td` (or `<th`), then
/// attribute fragments like ` colspan="4"`, then `>`, then later `</td>`
/// (or `</th>`). Cells get paired with the next bbox in document order.
///
/// Cells inside `<thead>` and any `<th>` cells are flagged as headers.
/// rowspan/colspan attributes are honoured and prior-row rowspans push
/// later-row cells to the right.
pub(crate) fn parse_structure(tokens: &[String]) -> Vec<CellSlot> {
let mut slots: Vec<CellSlot> = Vec::new();
let mut occupied: HashSet<(usize, usize)> = HashSet::new();
let mut row: usize = 0;
let mut col: usize = 0;
let mut bbox_idx: usize = 0;
let mut in_thead = false;
let mut started_first_row = false;
let mut i = 0;
while i < tokens.len() {
let tok = tokens[i].trim();
match tok {
"<thead>" => {
in_thead = true;
}
"</thead>" => {
in_thead = false;
}
"<tr>" => {
if started_first_row {
row += 1;
}
col = 0;
started_first_row = true;
}
"<td></td>" | "<th></th>" => {
let is_th = tok == "<th></th>";
while occupied.contains(&(row, col)) {
col += 1;
}
slots.push(CellSlot {
row,
col,
rowspan: 1,
colspan: 1,
is_header: in_thead || is_th,
bbox_idx,
});
bbox_idx += 1;
col += 1;
}
"<td" | "<th" => {
let is_th = tok == "<th";
let mut rowspan: usize = 1;
let mut colspan: usize = 1;
// Consume attribute fragments until we hit ">".
i += 1;
while i < tokens.len() && tokens[i].trim() != ">" {
let attr = tokens[i].as_str();
if let Some(v) = parse_int_attr(attr, "rowspan") {
rowspan = v.max(1);
} else if let Some(v) = parse_int_attr(attr, "colspan") {
colspan = v.max(1);
}
i += 1;
}
// i now points at the `>` token (or off the end if malformed).
while occupied.contains(&(row, col)) {
col += 1;
}
slots.push(CellSlot {
row,
col,
rowspan,
colspan,
is_header: in_thead || is_th,
bbox_idx,
});
for r in row..row + rowspan {
for c in col..col + colspan {
occupied.insert((r, c));
}
}
bbox_idx += 1;
col += colspan;
}
// Wrapper / informational tokens — no-op.
_ => {}
}
i += 1;
}
slots
}
/// Parse an attribute fragment like ` colspan="4"` or `rowspan='2'`.
///
/// Tolerates leading whitespace and either single or double quotes.
fn parse_int_attr(s: &str, name: &str) -> Option<usize> {
let trimmed = s.trim();
if !trimmed.starts_with(name) {
return None;
}
let rest = trimmed[name.len()..].trim_start();
let rest = rest.strip_prefix('=')?.trim_start();
let value = rest
.trim_start_matches(['"', '\''])
.trim_end_matches(['"', '\'']);
value.parse().ok()
}
/// Convert a SLANet polygon (4 or 8 elements) into an axis-aligned
/// `[x1, y1, x2, y2]` rect.
///
/// 8-element form: `[x1,y1, x2,y1, x2,y2, x1,y2]` (4 corners). We ignore the
/// implicit corner order and just take min/max so rotated polygons collapse
/// to a sane bounding box.
///
/// 4-element form: `[x1, y1, x2, y2]` (axis-aligned, older SLANet variants).
pub(crate) fn polygon_to_aabb(coords: &[f32]) -> Option<[f32; 4]> {
match coords.len() {
4 => {
let x1 = coords[0].min(coords[2]);
let y1 = coords[1].min(coords[3]);
let x2 = coords[0].max(coords[2]);
let y2 = coords[1].max(coords[3]);
Some([x1, y1, x2, y2])
}
8 => {
let xs = [coords[0], coords[2], coords[4], coords[6]];
let ys = [coords[1], coords[3], coords[5], coords[7]];
let x1 = xs.iter().copied().fold(f32::INFINITY, f32::min);
let y1 = ys.iter().copied().fold(f32::INFINITY, f32::min);
let x2 = xs.iter().copied().fold(f32::NEG_INFINITY, f32::max);
let y2 = ys.iter().copied().fold(f32::NEG_INFINITY, f32::max);
if x1.is_finite() && y1.is_finite() && x2.is_finite() && y2.is_finite() {
Some([x1, y1, x2, y2])
} else {
None
}
}
_ => None,
}
}
/// Convert a cell rect from crop image-pixel space to page PDF-points
/// (top-left origin), given the crop's PDF-point offset on the page and the
/// DPI the crop image was rendered at.
pub(crate) fn cell_px_to_page_pt(
cell_px: [f32; 4],
render_dpi: f32,
crop_origin_pt: [f32; 2],
) -> [f32; 4] {
let pt_per_px = if render_dpi > 0.0 {
72.0 / render_dpi
} else {
1.0
};
let [x_off, y_off] = crop_origin_pt;
[
cell_px[0] * pt_per_px + x_off,
cell_px[1] * pt_per_px + y_off,
cell_px[2] * pt_per_px + x_off,
cell_px[3] * pt_per_px + y_off,
]
}
/// Refine TSR cell bboxes into non-overlapping row/column bands.
///
/// SLANet-style bboxes are often plausible but too tall on dense borderless
/// tables. Native PDF text assignment is more reliable when each parsed row
/// owns the band between neighboring row centers instead of the full model box.
pub(crate) fn normalize_cell_bands(cells: &mut [StructuredCell]) {
if cells.len() < 2 {
return;
}
let row_bands = derive_axis_bands(cells, Axis::Y);
let col_bands = derive_axis_bands(cells, Axis::X);
for cell in cells {
let row_end = cell.row + cell.rowspan.max(1).saturating_sub(1);
if let (Some(&(y1, _)), Some(&(_, y2))) =
(row_bands.get(&cell.row), row_bands.get(&row_end))
{
let clamped_y1 = cell.page_pt_bbox[1].max(y1);
let clamped_y2 = cell.page_pt_bbox[3].min(y2);
if clamped_y1 < clamped_y2 {
cell.page_pt_bbox[1] = clamped_y1;
cell.page_pt_bbox[3] = clamped_y2;
}
}
let col_end = cell.col + cell.colspan.max(1).saturating_sub(1);
if let (Some(&(x1, _)), Some(&(_, x2))) =
(col_bands.get(&cell.col), col_bands.get(&col_end))
{
let clamped_x1 = cell.page_pt_bbox[0].max(x1);
let clamped_x2 = cell.page_pt_bbox[2].min(x2);
if clamped_x1 < clamped_x2 {
cell.page_pt_bbox[0] = clamped_x1;
cell.page_pt_bbox[2] = clamped_x2;
}
}
}
}
#[derive(Clone, Copy)]
enum Axis {
X,
Y,
}
fn derive_axis_bands(cells: &[StructuredCell], axis: Axis) -> HashMap<usize, (f32, f32)> {
let mut by_index: HashMap<usize, Vec<(f32, f32)>> = HashMap::new();
// Prefer non-spanning cells so colspan/rowspan boxes do not skew a single
// column/row center. If an axis has no non-spanning examples for an index,
// fall back to anchored cells below.
for cell in cells {
let span = match axis {
Axis::X => cell.colspan.max(1),
Axis::Y => cell.rowspan.max(1),
};
if span == 1 {
let idx = match axis {
Axis::X => cell.col,
Axis::Y => cell.row,
};
by_index
.entry(idx)
.or_default()
.push(axis_bounds(cell.page_pt_bbox, axis));
}
}
for cell in cells {
let idx = match axis {
Axis::X => cell.col,
Axis::Y => cell.row,
};
if !by_index.contains_key(&idx) {
by_index
.entry(idx)
.or_default()
.push(axis_bounds(cell.page_pt_bbox, axis));
}
}
let mut rows: Vec<(usize, f32, f32, f32)> = by_index
.into_iter()
.filter_map(|(idx, bounds)| {
let mut min_edge = f32::INFINITY;
let mut max_edge = f32::NEG_INFINITY;
let mut center_sum = 0.0;
let mut count = 0usize;
for (lo, hi) in bounds {
if lo.is_finite() && hi.is_finite() && lo < hi {
min_edge = min_edge.min(lo);
max_edge = max_edge.max(hi);
center_sum += (lo + hi) * 0.5;
count += 1;
}
}
(count > 0).then_some((idx, center_sum / count as f32, min_edge, max_edge))
})
.collect();
if rows.len() < 2 {
return rows
.into_iter()
.map(|(idx, _center, lo, hi)| (idx, (lo, hi)))
.collect();
}
rows.sort_by_key(|(idx, _, _, _)| *idx);
let mut bands = HashMap::new();
for i in 0..rows.len() {
let (idx, _center, min_edge, max_edge) = rows[i];
let lo = if i == 0 {
min_edge
} else {
(rows[i - 1].1 + rows[i].1) * 0.5
};
let hi = if i + 1 == rows.len() {
max_edge
} else {
(rows[i].1 + rows[i + 1].1) * 0.5
};
if lo.is_finite() && hi.is_finite() && lo < hi {
bands.insert(idx, (lo, hi));
}
}
bands
}
fn axis_bounds(bbox: [f32; 4], axis: Axis) -> (f32, f32) {
match axis {
Axis::X => (bbox[0].min(bbox[2]), bbox[0].max(bbox[2])),
Axis::Y => (bbox[1].min(bbox[3]), bbox[1].max(bbox[3])),
}
}
/// Sanitize cell text for inclusion in a markdown pipe-table cell:
/// collapse whitespace runs, drop newlines/tabs (cells must be one line),
/// and escape pipes that would otherwise break the table.
fn sanitize_cell(text: &str) -> String {
let mut s = String::with_capacity(text.len());
let mut prev_space = false;
for c in text.chars() {
match c {
'|' => {
s.push_str("\\|");
prev_space = false;
}
'\n' | '\r' | '\t' | ' ' => {
if !prev_space {
s.push(' ');
}
prev_space = true;
}
other => {
s.push(other);
prev_space = false;
}
}
}
s.trim().to_string()
}
/// Render a list of explicitly-positioned cells as a markdown pipe-table.
///
/// Grid dimensions are inferred from the cells' (row, col, rowspan, colspan)
/// extents. A cell with colspan/rowspan > 1 is rendered in its top-left
/// position; the absorbed grid positions are emitted as empty cells so the
/// markdown stays a valid rectangular grid that downstream readers can
/// column-count correctly.
///
/// The separator row (`|---|...|`) is emitted after the **last** row that
/// contains a header cell (`is_header == true`). When no cells are flagged
/// as headers — e.g. the upstream TSR model didn't emit `<thead>`/`<th>` —
/// the separator falls back to "after row 0" so the output is still a
/// valid pipe-table.
pub fn cells_to_markdown(cells: &[StructuredCell]) -> String {
if cells.is_empty() {
return String::new();
}
let num_rows = cells
.iter()
.map(|c| c.row + c.rowspan.max(1))
.max()
.unwrap_or(0);
let num_cols = cells
.iter()
.map(|c| c.col + c.colspan.max(1))
.max()
.unwrap_or(0);
if num_rows == 0 || num_cols == 0 {
return String::new();
}
// Separator goes after the last header row, falling back to row 0 when
// no header cells exist. Clamped into range so a malformed cell with
// row >= num_rows can't push it past the table.
let separator_after_row = cells
.iter()
.filter(|c| c.is_header)
.map(|c| c.row)
.max()
.unwrap_or(0)
.min(num_rows.saturating_sub(1));
let mut grid: Vec<Vec<String>> = vec![vec![String::new(); num_cols]; num_rows];
for cell in cells {
if cell.row < num_rows && cell.col < num_cols {
grid[cell.row][cell.col] = sanitize_cell(&cell.text);
}
}
let mut output = String::new();
for (row_idx, row) in grid.iter().enumerate() {
output.push('|');
for cell in row {
output.push_str(cell);
output.push('|');
}
output.push('\n');
if row_idx == separator_after_row {
output.push('|');
for _ in 0..num_cols {
output.push_str("---|");
}
output.push('\n');
}
}
output
}
#[cfg(test)]
mod tests {
use super::*;
fn t(s: &str) -> String {
s.to_string()
}
/// Tokens for the synthetic 3×3 grid example (one colspan-4 row + two
/// data rows of 4 cells each = 9 cells total, 3 rows × 4 cols).
fn synthetic_3x3_tokens() -> Vec<String> {
vec![
"<html>",
"<body>",
"<table>",
"<tbody>",
"<tr>",
"<td",
" colspan=\"4\"",
">",
"</td>",
"</tr>",
"<tr>",
"<td></td>",
"<td></td>",
"<td></td>",
"<td></td>",
"</tr>",
"<tr>",
"<td></td>",
"<td></td>",
"<td></td>",
"<td></td>",
"</tr>",
"</tbody>",
"</table>",
"</body>",
"</html>",
]
.into_iter()
.map(t)
.collect()
}
/// Bboxes for the synthetic 3×3 grid (8-element polygon form), all
/// within a 400×120 px crop.
fn synthetic_3x3_bboxes() -> Vec<Vec<f32>> {
vec![
vec![3.0, 2.0, 395.0, 2.0, 396.0, 59.0, 3.0, 59.0],
vec![26.0, 62.0, 140.0, 62.0, 141.0, 120.0, 26.0, 120.0],
vec![149.0, 64.0, 248.0, 64.0, 248.0, 119.0, 149.0, 119.0],
vec![257.0, 64.0, 350.0, 64.0, 350.0, 119.0, 257.0, 119.0],
vec![359.0, 64.0, 395.0, 64.0, 395.0, 119.0, 359.0, 119.0],
vec![26.0, 122.0, 140.0, 122.0, 140.0, 178.0, 26.0, 178.0],
vec![149.0, 124.0, 248.0, 124.0, 248.0, 179.0, 149.0, 179.0],
vec![257.0, 124.0, 350.0, 124.0, 350.0, 179.0, 257.0, 179.0],
vec![359.0, 124.0, 395.0, 124.0, 395.0, 179.0, 359.0, 179.0],
]
}
#[test]
fn parse_structure_synthetic_3x3() {
let tokens = synthetic_3x3_tokens();
let slots = parse_structure(&tokens);
assert_eq!(slots.len(), 9, "should parse 9 cells");
// Cell 0: row 0 col 0, colspan 4
assert_eq!(slots[0].row, 0);
assert_eq!(slots[0].col, 0);
assert_eq!(slots[0].colspan, 4);
assert_eq!(slots[0].rowspan, 1);
// Cells 1..5: row 1, cols 0..3
for (i, slot) in slots.iter().enumerate().skip(1).take(4) {
assert_eq!(slot.row, 1, "cell {i}: row should be 1");
assert_eq!(slot.col, i - 1, "cell {i}: col should be {}", i - 1);
assert_eq!(slot.colspan, 1);
assert_eq!(slot.rowspan, 1);
}
// Cells 5..9: row 2, cols 0..3
for (i, slot) in slots.iter().enumerate().skip(5).take(4) {
assert_eq!(slot.row, 2, "cell {i}: row should be 2");
assert_eq!(slot.col, i - 5);
assert_eq!(slot.colspan, 1);
}
}
#[test]
fn polygon_to_aabb_8elt() {
// Synthetic cell bbox 0
let coords = vec![3.0, 2.0, 395.0, 2.0, 396.0, 59.0, 3.0, 59.0];
let aabb = polygon_to_aabb(&coords).unwrap();
assert_eq!(aabb, [3.0, 2.0, 396.0, 59.0]);
}
#[test]
fn polygon_to_aabb_4elt() {
let coords = vec![5.0, 10.0, 50.0, 60.0];
let aabb = polygon_to_aabb(&coords).unwrap();
assert_eq!(aabb, [5.0, 10.0, 50.0, 60.0]);
}
#[test]
fn polygon_to_aabb_4elt_unordered() {
// Caller may pass corners in any order; min/max should normalise.
let coords = vec![50.0, 60.0, 5.0, 10.0];
let aabb = polygon_to_aabb(&coords).unwrap();
assert_eq!(aabb, [5.0, 10.0, 50.0, 60.0]);
}
#[test]
fn polygon_to_aabb_invalid_len() {
assert!(polygon_to_aabb(&[1.0, 2.0, 3.0]).is_none());
assert!(polygon_to_aabb(&[1.0; 6]).is_none());
assert!(polygon_to_aabb(&[]).is_none());
}
#[test]
fn synthetic_3x3_aabbs_inside_crop() {
// All 9 bboxes should produce valid (x1<x2, y1<y2) rects within the
// crop bounds (400 wide, ~180 tall by inspection of the fixture).
let bboxes = synthetic_3x3_bboxes();
assert_eq!(bboxes.len(), 9);
for (i, bb) in bboxes.iter().enumerate() {
let aabb = polygon_to_aabb(bb).unwrap_or_else(|| panic!("bbox {i} invalid"));
assert!(aabb[0] < aabb[2], "bbox {i}: x1 < x2");
assert!(aabb[1] < aabb[3], "bbox {i}: y1 < y2");
assert!(aabb[0] >= 0.0 && aabb[2] <= 500.0, "bbox {i}: within crop");
assert!(aabb[1] >= 0.0 && aabb[3] <= 200.0, "bbox {i}: within crop");
}
}
#[test]
fn normalize_cell_bands_splits_overlapping_slanet_rows() {
let mut cells = vec![
StructuredCell {
row: 0,
col: 0,
rowspan: 1,
colspan: 1,
is_header: true,
text: String::new(),
page_pt_bbox: [10.0, 100.0, 90.0, 120.0],
},
StructuredCell {
row: 0,
col: 1,
rowspan: 1,
colspan: 1,
is_header: true,
text: String::new(),
page_pt_bbox: [90.0, 100.0, 170.0, 120.0],
},
StructuredCell {
row: 1,
col: 0,
rowspan: 1,
colspan: 1,
is_header: false,
text: String::new(),
page_pt_bbox: [10.0, 116.0, 90.0, 136.0],
},
StructuredCell {
row: 1,
col: 1,
rowspan: 1,
colspan: 1,
is_header: false,
text: String::new(),
page_pt_bbox: [90.0, 116.0, 170.0, 136.0],
},
];
normalize_cell_bands(&mut cells);
assert_eq!(cells[0].page_pt_bbox[3], cells[2].page_pt_bbox[1]);
assert_eq!(cells[1].page_pt_bbox[3], cells[3].page_pt_bbox[1]);
assert!(
(cells[0].page_pt_bbox[3] - 118.0).abs() < 0.01,
"row separator should be midpoint between row centers: {:?}",
cells
);
}
#[test]
fn normalize_cell_bands_preserves_colspan_extent() {
let mut cells = vec![
StructuredCell {
row: 0,
col: 0,
rowspan: 1,
colspan: 2,
is_header: true,
text: String::new(),
page_pt_bbox: [8.0, 80.0, 172.0, 98.0],
},
StructuredCell {
row: 1,
col: 0,
rowspan: 1,
colspan: 1,
is_header: false,
text: String::new(),
page_pt_bbox: [10.0, 96.0, 90.0, 114.0],
},
StructuredCell {
row: 1,
col: 1,
rowspan: 1,
colspan: 1,
is_header: false,
text: String::new(),
page_pt_bbox: [88.0, 96.0, 170.0, 114.0],
},
];
normalize_cell_bands(&mut cells);
assert!(
cells[0].page_pt_bbox[0] <= cells[1].page_pt_bbox[0],
"spanning cell should retain the first column's left edge"
);
assert!(
cells[0].page_pt_bbox[2] >= cells[2].page_pt_bbox[2],
"spanning cell should retain the last column's right edge"
);
}
#[test]
fn parse_int_attr_basic() {
assert_eq!(parse_int_attr(" colspan=\"4\"", "colspan"), Some(4));
assert_eq!(parse_int_attr(" rowspan=\"2\"", "rowspan"), Some(2));
assert_eq!(parse_int_attr("colspan='3'", "colspan"), Some(3));
assert_eq!(parse_int_attr(" colspan=\"4\"", "rowspan"), None);
assert_eq!(parse_int_attr(" class=\"foo\"", "colspan"), None);
}
#[test]
fn parse_structure_rowspan_pushes_next_row_right() {
// <tr><td rowspan="2">A</td><td>B</td></tr><tr><td>C</td></tr>
// Expected: A at (0,0), B at (0,1), C at (1,1) — col 0 of row 1
// is occupied by A's rowspan.
let tokens: Vec<String> = vec![
"<table>",
"<tbody>",
"<tr>",
"<td",
" rowspan=\"2\"",
">",
"</td>",
"<td></td>",
"</tr>",
"<tr>",
"<td></td>",
"</tr>",
"</tbody>",
"</table>",
]
.into_iter()
.map(t)
.collect();
let slots = parse_structure(&tokens);
assert_eq!(slots.len(), 3);
assert_eq!((slots[0].row, slots[0].col), (0, 0));
assert_eq!(slots[0].rowspan, 2);
assert_eq!((slots[1].row, slots[1].col), (0, 1));
// C should be at (1, 1) because (1, 0) is occupied by A's rowspan.
assert_eq!((slots[2].row, slots[2].col), (1, 1));
}
#[test]
fn parse_structure_thead_marks_headers() {
// <thead><tr><th>H1</th><th>H2</th></tr></thead>
// <tbody><tr><td>D1</td><td>D2</td></tr></tbody>
let tokens: Vec<String> = vec![
"<table>",
"<thead>",
"<tr>",
"<th></th>",
"<th></th>",
"</tr>",
"</thead>",
"<tbody>",
"<tr>",
"<td></td>",
"<td></td>",
"</tr>",
"</tbody>",
"</table>",
]
.into_iter()
.map(t)
.collect();
let slots = parse_structure(&tokens);
assert_eq!(slots.len(), 4);
assert!(slots[0].is_header && slots[1].is_header);
assert!(!slots[2].is_header && !slots[3].is_header);
}
#[test]
fn parse_structure_th_outside_thead_still_header() {
// A row-header style: leading <th> in tbody.
let tokens: Vec<String> = vec![
"<table>",
"<tbody>",
"<tr>",
"<th></th>",
"<td></td>",
"</tr>",
"</tbody>",
"</table>",
]
.into_iter()
.map(t)
.collect();
let slots = parse_structure(&tokens);
assert_eq!(slots.len(), 2);
assert!(slots[0].is_header);
assert!(!slots[1].is_header);
}
#[test]
fn parse_structure_th_with_attrs() {
let tokens: Vec<String> = vec![
"<table>",
"<thead>",
"<tr>",
"<th",
" colspan=\"2\"",
">",
"</th>",
"</tr>",
"</thead>",
"</table>",
]
.into_iter()
.map(t)
.collect();
let slots = parse_structure(&tokens);
assert_eq!(slots.len(), 1);
assert_eq!(slots[0].colspan, 2);
assert!(slots[0].is_header);
}
#[test]
fn cells_to_markdown_synthetic_3x3() {
// Build the cells the parser would produce for the synthetic grid,
// and provide some sample text so we can sanity-check output.
let cells = vec![
StructuredCell {
row: 0,
col: 0,
rowspan: 1,
colspan: 4,
is_header: false,
text: "Title".into(),
page_pt_bbox: [0.0, 0.0, 0.0, 0.0],
},
StructuredCell {
row: 1,
col: 0,
rowspan: 1,
colspan: 1,
is_header: false,
text: "a".into(),
page_pt_bbox: [0.0, 0.0, 0.0, 0.0],
},
StructuredCell {
row: 1,
col: 1,
rowspan: 1,
colspan: 1,
is_header: false,
text: "b".into(),
page_pt_bbox: [0.0, 0.0, 0.0, 0.0],
},
StructuredCell {
row: 1,
col: 2,
rowspan: 1,
colspan: 1,
is_header: false,
text: "c".into(),
page_pt_bbox: [0.0, 0.0, 0.0, 0.0],
},
StructuredCell {
row: 1,
col: 3,
rowspan: 1,
colspan: 1,
is_header: false,
text: "d".into(),
page_pt_bbox: [0.0, 0.0, 0.0, 0.0],
},
];
let md = cells_to_markdown(&cells);
// Header row contains the spanning cell text in col 0 and pads to 4 cols.
// Absorbed-by-colspan positions render as empty cells (no padding).
assert!(md.starts_with("|Title||||\n"), "got: {md}");
assert!(md.contains("|---|---|---|---|\n"));
assert!(md.contains("|a|b|c|d|\n"));
}
#[test]
fn cells_to_markdown_escapes_pipes() {
let cells = vec![
StructuredCell {
row: 0,
col: 0,
rowspan: 1,
colspan: 1,
is_header: false,
text: "a|b".into(),
page_pt_bbox: [0.0, 0.0, 0.0, 0.0],
},
StructuredCell {
row: 0,
col: 1,
rowspan: 1,
colspan: 1,
is_header: false,
text: "x".into(),
page_pt_bbox: [0.0, 0.0, 0.0, 0.0],
},
];
let md = cells_to_markdown(&cells);
assert!(md.contains("|a\\|b|x|"));
}
#[test]
fn cells_to_markdown_collapses_whitespace_and_newlines() {
let cells = vec![StructuredCell {
row: 0,
col: 0,
rowspan: 1,
colspan: 1,
is_header: false,
text: "foo \n bar\tbaz".into(),
page_pt_bbox: [0.0, 0.0, 0.0, 0.0],
}];
let md = cells_to_markdown(&cells);
assert!(md.contains("|foo bar baz|"));
}
fn cell(row: usize, col: usize, is_header: bool, text: &str) -> StructuredCell {
StructuredCell {
row,
col,
rowspan: 1,
colspan: 1,
is_header,
text: text.into(),
page_pt_bbox: [0.0, 0.0, 0.0, 0.0],
}
}
#[test]
fn cells_to_markdown_separator_after_last_header_row() {
// Two-row header (a multi-row thead), then two body rows. Separator
// should land after row 1 (the LAST header row), not after row 0.
let cells = vec![
cell(0, 0, true, "H0a"),
cell(0, 1, true, "H0b"),
cell(1, 0, true, "H1a"),
cell(1, 1, true, "H1b"),
cell(2, 0, false, "d0a"),
cell(2, 1, false, "d0b"),
cell(3, 0, false, "d1a"),
cell(3, 1, false, "d1b"),
];
let md = cells_to_markdown(&cells);
let expected = "|H0a|H0b|\n|H1a|H1b|\n|---|---|\n|d0a|d0b|\n|d1a|d1b|\n";
assert_eq!(md, expected, "got: {md}");
}
#[test]
fn cells_to_markdown_separator_when_row_0_not_header() {
// Row 0 is not flagged as a header but row 1 is. Separator should
// follow row 1 (the header), demonstrating that we don't blindly
// emit after row 0.
let cells = vec![
cell(0, 0, false, "x0a"),
cell(0, 1, false, "x0b"),
cell(1, 0, true, "Hdr1"),
cell(1, 1, true, "Hdr2"),
cell(2, 0, false, "data1"),
cell(2, 1, false, "data2"),
];
let md = cells_to_markdown(&cells);
// Confirm the separator is NOT after row 0.
assert!(!md.starts_with("|x0a|x0b|\n|---|"), "got: {md}");
// Confirm it IS after row 1.
assert!(
md.contains("|Hdr1|Hdr2|\n|---|---|\n|data1|data2|"),
"got: {md}"
);
}
#[test]
fn cells_to_markdown_no_headers_falls_back_to_row_0() {
// No header cells at all — fallback: separator after row 0 so the
// output is still a valid markdown pipe-table.
let cells = vec![
cell(0, 0, false, "a"),
cell(0, 1, false, "b"),
cell(1, 0, false, "c"),
cell(1, 1, false, "d"),
];
let md = cells_to_markdown(&cells);
assert_eq!(md, "|a|b|\n|---|---|\n|c|d|\n");
}
}
Binary file not shown.
Binary file not shown.
File diff suppressed because it is too large Load Diff