fix(extract): pass page lines to per-page markdown conversion (#434)

* fix(layout): keep relative-valley column detection on table pages

The page_has_table guard predates item claiming: by grouping time the
detected table's items have already left the flow, so a table cannot
fake a gutter in the histogram this guard protects, and the prose gate
inside relative-valley acceptance rejects any residual table-shaped
split. Without the fallback, the prose remainder of a table-bearing
two-column page fell to single-column Y-sorting and its columns
interleaved line by line.

* fix(layout): lower the relative-valley floor to sparse pages

The 100-item floor guarded against shallow histogram dips on sparse
pages, but OCR'd multi-column pages produce few long line-runs (a
two-column French academic page arrives as ~60 items) and were falling
to single-column Y-sorting, weaving their columns line by line. The
prose gate inside relative-valley acceptance is the real defense
against spurious dips; 30 items is enough for it to judge.

* fix(layout): re-guard the ungated XY-cut fallthrough on table pages

Removing page_has_table from the relative-valley condition also
unlocked the XY-cut fallback inside that block, which has no prose
gate — a table page whose valley candidate was just rejected could
take an unvalidated split. The guard is restored on that call
specifically; the relative-valley path keeps its prose-gated access.
Also rewrite the stale dense-page comment for the 30-item floor.
Bench and corpus unchanged (462/884, corpus byte-identical).

* fix(extract): pass page lines to per-page markdown conversion

The per-page extraction path (extract_pages_markdown_mem_impl, used by the
pages API and every --ocr auto run) partitioned each page's rects but
passed an empty slice for PDF lines to markdown conversion, while the
extracted all_lines sat unused. Every table only the line-based detector
finds (text-anchor rule tables, ruled grids) was silently dropped in that
mode, even though the whole-document path emitted it fine.

Partition all_lines per page like rects and pass them through. Regression
test: a rule-anchored table fixture must survive the pages API.

src/lib.rs

@@ -592,6 +592,12 @@ fn extract_pages_markdown_mem_impl(
             .cloned()
             .collect();
 
+        let page_lines: Vec<types::PdfLine> = all_lines
+            .iter()
+            .filter(|l| l.page == page_1idx)
+            .cloned()
+            .collect();
+
         let has_gid = gid_pages.contains(&page_1idx);
         let has_text_quality_issue = text_quality.pages_needing_ocr.contains(&page_1idx);
 
@@ -629,7 +635,7 @@ fn extract_pages_markdown_mem_impl(
                 page_items,
                 options,
                 &page_rects,
-                &[],
+                &page_lines,
                 markdown::MarkdownDocumentContext {
                     page_thresholds: &page_thresholds,
                     struct_roles: None,

tests/integration_tests.rs

@@ -3495,6 +3495,28 @@ fn test_extract_pages_markdown_basic() {
     assert!(!result.pages[0].needs_ocr);
 }
 
+#[test]
+fn test_extract_pages_markdown_keeps_line_based_tables() {
+    // The per-page path (used by every `--ocr auto` run) once passed an
+    // empty line slice to markdown conversion, silently dropping every
+    // table that only the line-based detector finds. This fixture's table
+    // is rule-anchored: it must survive the pages API exactly as it does
+    // the whole-document API.
+    let buf = std::fs::read("tests/fixtures/bits_pilani_feedback.pdf").unwrap();
+    let result = extract_pages_markdown_mem(&buf, None).unwrap();
+
+    let all_markdown: String = result
+        .pages
+        .iter()
+        .map(|p| p.markdown.as_str())
+        .collect::<Vec<_>>()
+        .join("\n");
+    assert!(
+        all_markdown.contains("|BIO|"),
+        "line-based table rows missing from pages API output"
+    );
+}
+
 #[test]
 fn test_extract_pages_markdown_uses_document_wide_folio_context() {
     let pdf = make_recurring_contextual_folio_pdf();