fix(extract): pass page lines to per-page markdown conversion (#434)

* fix(layout): keep relative-valley column detection on table pages

The page_has_table guard predates item claiming: by grouping time the
detected table's items have already left the flow, so a table cannot
fake a gutter in the histogram this guard protects, and the prose gate
inside relative-valley acceptance rejects any residual table-shaped
split. Without the fallback, the prose remainder of a table-bearing
two-column page fell to single-column Y-sorting and its columns
interleaved line by line.

* fix(layout): lower the relative-valley floor to sparse pages

The 100-item floor guarded against shallow histogram dips on sparse
pages, but OCR'd multi-column pages produce few long line-runs (a
two-column French academic page arrives as ~60 items) and were falling
to single-column Y-sorting, weaving their columns line by line. The
prose gate inside relative-valley acceptance is the real defense
against spurious dips; 30 items is enough for it to judge.

* fix(layout): re-guard the ungated XY-cut fallthrough on table pages

Removing page_has_table from the relative-valley condition also
unlocked the XY-cut fallback inside that block, which has no prose
gate — a table page whose valley candidate was just rejected could
take an unvalidated split. The guard is restored on that call
specifically; the relative-valley path keeps its prose-gated access.
Also rewrite the stale dense-page comment for the 30-item floor.
Bench and corpus unchanged (462/884, corpus byte-identical).

* fix(extract): pass page lines to per-page markdown conversion

The per-page extraction path (extract_pages_markdown_mem_impl, used by the
pages API and every --ocr auto run) partitioned each page's rects but
passed an empty slice for PDF lines to markdown conversion, while the
extracted all_lines sat unused. Every table only the line-based detector
finds (text-anchor rule tables, ruled grids) was silently dropped in that
mode, even though the whole-document path emitted it fine.

Partition all_lines per page like rects and pass them through. Regression
test: a rule-anchored table fixture must survive the pages API.

src/extractor/layout.rs

@@ -225,11 +225,19 @@ pub(crate) fn detect_columns(
     // Justified text can leave gutter bins non-empty because item widths extend
     // to the column edge. Look for local minima that are significantly lower
     // than the peaks on either side.
-    // Only attempt this for dense pages (>=100 items) — sparse pages with shallow
+    //
-    // histogram dips are likely not multi-column.
+    // The 30-item floor admits sparse pages: OCR'd multi-column pages arrive
-    // Skip on pages with detected tables — table column gaps look like gutters
+    // as few long line-runs and were falling to single-column Y-sorting.
-    // in the histogram but the table pipeline already handles reading order.
+    // Below 30 items the histogram is too shallow for even the prose gate
-    if valleys.is_empty() && page_items.len() >= 100 && !page_has_table {
+    // to judge a dip.
+    //
+    // Pages with detected tables take the relative-valley path too: the
+    // table's items have already left the flow by the time grouping runs,
+    // so a table cannot fake a gutter here, and the prose gate below
+    // rejects any residual table-shaped split. Without this, the prose
+    // REMAINDER of a table-bearing two-column page falls to single-column
+    // Y-sorting and the columns interleave line by line.
+    if valleys.is_empty() && page_items.len() >= 30 {
         let rel_valleys = find_relative_valleys(
             &histogram,
             num_bins,
@@ -270,10 +278,15 @@ pub(crate) fn detect_columns(
                 }
             }
         }
-        // Try XY-cut fallback before giving up
+        // Try XY-cut fallback before giving up. Unlike the relative-valley
+        // path above, XY-cut has no prose gate, so the table-page guard
+        // stays here: without it a table page whose valley candidate was
+        // just rejected could take an unvalidated split.
+        if !page_has_table {
             if let Some(columns) = try_xy_cut_split(&page_items, x_min, x_max, page) {
                 return columns;
             }
+        }
         return vec![ColumnRegion { x_min, x_max }];
     }
 

src/lib.rs

@@ -592,6 +592,12 @@ fn extract_pages_markdown_mem_impl(
             .cloned()
             .collect();
 
+        let page_lines: Vec<types::PdfLine> = all_lines
+            .iter()
+            .filter(|l| l.page == page_1idx)
+            .cloned()
+            .collect();
+
         let has_gid = gid_pages.contains(&page_1idx);
         let has_text_quality_issue = text_quality.pages_needing_ocr.contains(&page_1idx);
 
@@ -629,7 +635,7 @@ fn extract_pages_markdown_mem_impl(
                 page_items,
                 options,
                 &page_rects,
-                &[],
+                &page_lines,
                 markdown::MarkdownDocumentContext {
                     page_thresholds: &page_thresholds,
                     struct_roles: None,

tests/integration_tests.rs

@@ -3495,6 +3495,28 @@ fn test_extract_pages_markdown_basic() {
     assert!(!result.pages[0].needs_ocr);
 }
 
+#[test]
+fn test_extract_pages_markdown_keeps_line_based_tables() {
+    // The per-page path (used by every `--ocr auto` run) once passed an
+    // empty line slice to markdown conversion, silently dropping every
+    // table that only the line-based detector finds. This fixture's table
+    // is rule-anchored: it must survive the pages API exactly as it does
+    // the whole-document API.
+    let buf = std::fs::read("tests/fixtures/bits_pilani_feedback.pdf").unwrap();
+    let result = extract_pages_markdown_mem(&buf, None).unwrap();
+
+    let all_markdown: String = result
+        .pages
+        .iter()
+        .map(|p| p.markdown.as_str())
+        .collect::<Vec<_>>()
+        .join("\n");
+    assert!(
+        all_markdown.contains("|BIO|"),
+        "line-based table rows missing from pages API output"
+    );
+}
+
 #[test]
 fn test_extract_pages_markdown_uses_document_wide_folio_context() {
     let pdf = make_recurring_contextual_folio_pdf();