fix(extract): pass page lines to per-page markdown conversion (#434)

* fix(layout): keep relative-valley column detection on table pages

The page_has_table guard predates item claiming: by grouping time the
detected table's items have already left the flow, so a table cannot
fake a gutter in the histogram this guard protects, and the prose gate
inside relative-valley acceptance rejects any residual table-shaped
split. Without the fallback, the prose remainder of a table-bearing
two-column page fell to single-column Y-sorting and its columns
interleaved line by line.

* fix(layout): lower the relative-valley floor to sparse pages

The 100-item floor guarded against shallow histogram dips on sparse
pages, but OCR'd multi-column pages produce few long line-runs (a
two-column French academic page arrives as ~60 items) and were falling
to single-column Y-sorting, weaving their columns line by line. The
prose gate inside relative-valley acceptance is the real defense
against spurious dips; 30 items is enough for it to judge.

* fix(layout): re-guard the ungated XY-cut fallthrough on table pages

Removing page_has_table from the relative-valley condition also
unlocked the XY-cut fallback inside that block, which has no prose
gate — a table page whose valley candidate was just rejected could
take an unvalidated split. The guard is restored on that call
specifically; the relative-valley path keeps its prose-gated access.
Also rewrite the stale dense-page comment for the 30-item floor.
Bench and corpus unchanged (462/884, corpus byte-identical).

* fix(extract): pass page lines to per-page markdown conversion

The per-page extraction path (extract_pages_markdown_mem_impl, used by the
pages API and every --ocr auto run) partitioned each page's rects but
passed an empty slice for PDF lines to markdown conversion, while the
extracted all_lines sat unused. Every table only the line-based detector
finds (text-anchor rule tables, ruled grids) was silently dropped in that
mode, even though the whole-document path emitted it fine.

Partition all_lines per page like rects and pass them through. Regression
test: a rule-anchored table fixture must survive the pages API.

src/extractor/layout.rs

@@ -225,11 +225,19 @@ pub(crate) fn detect_columns(
     // Justified text can leave gutter bins non-empty because item widths extend
     // to the column edge. Look for local minima that are significantly lower
     // than the peaks on either side.
-    // Only attempt this for dense pages (>=100 items) — sparse pages with shallow
-    // histogram dips are likely not multi-column.
-    // Skip on pages with detected tables — table column gaps look like gutters
-    // in the histogram but the table pipeline already handles reading order.
-    if valleys.is_empty() && page_items.len() >= 100 && !page_has_table {
+    //
+    // The 30-item floor admits sparse pages: OCR'd multi-column pages arrive
+    // as few long line-runs and were falling to single-column Y-sorting.
+    // Below 30 items the histogram is too shallow for even the prose gate
+    // to judge a dip.
+    //
+    // Pages with detected tables take the relative-valley path too: the
+    // table's items have already left the flow by the time grouping runs,
+    // so a table cannot fake a gutter here, and the prose gate below
+    // rejects any residual table-shaped split. Without this, the prose
+    // REMAINDER of a table-bearing two-column page falls to single-column
+    // Y-sorting and the columns interleave line by line.
+    if valleys.is_empty() && page_items.len() >= 30 {
         let rel_valleys = find_relative_valleys(
             &histogram,
             num_bins,
@@ -270,9 +278,14 @@ pub(crate) fn detect_columns(
                 }
             }
         }
-        // Try XY-cut fallback before giving up
-        if let Some(columns) = try_xy_cut_split(&page_items, x_min, x_max, page) {
-            return columns;
+        // Try XY-cut fallback before giving up. Unlike the relative-valley
+        // path above, XY-cut has no prose gate, so the table-page guard
+        // stays here: without it a table page whose valley candidate was
+        // just rejected could take an unvalidated split.
+        if !page_has_table {
+            if let Some(columns) = try_xy_cut_split(&page_items, x_min, x_max, page) {
+                return columns;
+            }
         }
         return vec![ColumnRegion { x_min, x_max }];
     }

src/lib.rs

@@ -592,6 +592,12 @@ fn extract_pages_markdown_mem_impl(
             .cloned()
             .collect();
 
+        let page_lines: Vec<types::PdfLine> = all_lines
+            .iter()
+            .filter(|l| l.page == page_1idx)
+            .cloned()
+            .collect();
+
         let has_gid = gid_pages.contains(&page_1idx);
         let has_text_quality_issue = text_quality.pages_needing_ocr.contains(&page_1idx);
 
@@ -629,7 +635,7 @@ fn extract_pages_markdown_mem_impl(
                 page_items,
                 options,
                 &page_rects,
-                &[],
+                &page_lines,
                 markdown::MarkdownDocumentContext {
                     page_thresholds: &page_thresholds,
                     struct_roles: None,

src/markdown/mod.rs

@@ -635,7 +635,12 @@ fn is_parallel_prose_table(table: &crate::tables::Table) -> bool {
         }
     }
 
-    let is_parallel = !has_compact_header
+    // A compact header row is evidence for a real table — unless cross-row
+    // prose continuations outnumber the rows, which no genuine table
+    // produces: the "header" is then just two short line fragments at the
+    // top of parallel prose columns.
+    let header_blocks = has_compact_header && continuation_fragments <= table.cells.len();
+    let is_parallel = !header_blocks
         && non_empty >= 5
         // Independent prose columns have asynchronous line/paragraph breaks;
         // a fully populated grid is positive evidence for a real descriptive
@@ -1375,7 +1380,6 @@ pub(crate) fn to_markdown_from_items_with_rects_and_lines(
             chart_page_prose_column_split(&page_layout_items)
                 .filter(|&split_x| chart_spans_prose_split(region, split_x))
         });
-        let chart_prose_columns = chart_prose_split.is_some();
 
         // Check for side-by-side table layout using the original items. Sparse
         // numeric cells need table context before they can be distinguished
@@ -1616,10 +1620,16 @@ pub(crate) fn to_markdown_from_items_with_rects_and_lines(
                     if subset_items.len() < min_items {
                         return;
                     }
-                    // Keep body-font detection available on chart pages: a real
-                    // table can share the prose anchors. Reject only candidates
-                    // whose cells prove they are parallel prose fragments.
-                    let reject_parallel_prose = chart_prose_columns && !was_split;
+                    // Reject candidates whose cells prove they are parallel
+                    // prose fragments — the shape produced when the body-font
+                    // pass projects a multi-column text page onto one table
+                    // grid (two-column reference sections are the classic
+                    // case). The check needs internal transition evidence
+                    // (unterminated cells flowing into lowercase starts in
+                    // the same column), so genuine tables with long cells
+                    // pass. Band-split retries stay exempt: they exist for
+                    // tables that only assemble after recombining bands.
+                    let reject_parallel_prose = !was_split;
                     let tables = detect_tables_with_page_width(
                         subset_items,
                         base_size,
@@ -2674,6 +2684,35 @@ mod tests {
         );
         assert!(!is_parallel_prose_table(&data));
 
+        // A compact header row atop parallel prose columns: cross-row prose
+        // continuations outnumber the rows, so the header cannot save the
+        // candidate — this is page prose with two short fragments on top.
+        let headed_parallel_prose = crate::tables::Table::new(
+            vec![90.0, 340.0],
+            vec![340.0, 320.0, 300.0, 280.0, 260.0],
+            vec![
+                vec!["June 2023".into(), "Page 5".into()],
+                vec![
+                    "the committee reviewed the proposal and decided that the".into(),
+                    "funding for the second phase would continue subject to the".into(),
+                ],
+                vec![
+                    "implementation schedule should be extended by another".into(),
+                    "quarterly reviews established during the first phase of the".into(),
+                ],
+                vec![
+                    "six months to accommodate the revised procurement rules".into(),
+                    "".into(),
+                ],
+                vec![
+                    "adopted at the previous meeting of the governing board".into(),
+                    "participating institutions across the partner regions".into(),
+                ],
+            ],
+            (0..10).collect(),
+        );
+        assert!(is_parallel_prose_table(&headed_parallel_prose));
+
         let headed_text_table = crate::tables::Table::new(
             vec![90.0, 340.0],
             vec![320.0, 300.0, 280.0],

tests/integration_tests.rs

@@ -3495,6 +3495,28 @@ fn test_extract_pages_markdown_basic() {
     assert!(!result.pages[0].needs_ocr);
 }
 
+#[test]
+fn test_extract_pages_markdown_keeps_line_based_tables() {
+    // The per-page path (used by every `--ocr auto` run) once passed an
+    // empty line slice to markdown conversion, silently dropping every
+    // table that only the line-based detector finds. This fixture's table
+    // is rule-anchored: it must survive the pages API exactly as it does
+    // the whole-document API.
+    let buf = std::fs::read("tests/fixtures/bits_pilani_feedback.pdf").unwrap();
+    let result = extract_pages_markdown_mem(&buf, None).unwrap();
+
+    let all_markdown: String = result
+        .pages
+        .iter()
+        .map(|p| p.markdown.as_str())
+        .collect::<Vec<_>>()
+        .join("\n");
+    assert!(
+        all_markdown.contains("|BIO|"),
+        "line-based table rows missing from pages API output"
+    );
+}
+
 #[test]
 fn test_extract_pages_markdown_uses_document_wide_folio_context() {
     let pdf = make_recurring_contextual_folio_pdf();