split TOC detection into dot-leader and tabular patterns

Previous is_table_of_contents rejected every TOC-shaped table at detect
time, dropping dot-leader TOCs that render well as flat-list output and
misclassifying data tables with trailing "..." cells as TOCs.

Now is_dot_leader_toc (structural + inline-leader) keeps per-row flat
layouts for the formatter, while only wide inline-leader indices are
rejected at detect time.  row_cell_is_page_number accepts dashed
section-page IDs ("A-1", "5-21") and rejects decimals/thousands
separators; cell_has_trailing_leader requires alphabetic content so
numeric data-row labels ("1973 ... ") no longer qualify.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

AGENTS.md

@@ -0,0 +1,79 @@
+# pdf-inspector
+
+Fast PDF text extraction to structured Markdown. CLI binary: `pdf2md`. Detection binary: `detect-pdf`.
+
+## Build & Test
+
+```bash
+cargo fmt                                    # format
+cargo clippy -- -D warnings                  # lint (enforced, zero warnings)
+cargo test                                   # unit + integration tests (267+ unit, 73+ integration)
+cargo build --release                        # release binary for benchmarks
+```
+
+All three must pass before committing.
+
+## Binaries
+
+- `pdf2md` — extract PDF → Markdown. Supports `--json` for structured output.
+- `detect-pdf` — classify PDF type (TextBased/Scanned/Mixed/ImageBased). Supports `--analyze --json`.
+
+## Architecture
+
+```
+src/
+  lib.rs                        – public API, process_pdf_with_options, encoding issue detection
+  detector.rs                   – PDF type classification, tiled-scan detection, page sampling
+  types.rs                      – TextItem, TextLine, PdfRect, PdfLine
+  tounicode.rs                  – CMap/ToUnicode parsing, CID decoding
+  text_utils.rs                 – CJK/RTL handling, Otsu threshold, ligature expansion, NFKC
+  extractor/
+    mod.rs                      – top-level extraction orchestrator
+    content_stream.rs           – PDF operator state machine (Tj/TJ/Td/Tm/q/Q)
+    fonts.rs                    – font width/encoding, CMapDecisionCache, TrueType cmap fallback
+    layout.rs                   – column detection (histogram), newspaper/tabular classification,
+                                  spanning-line pre-masking, sidebar detection
+  tables/
+    detect_rects.rs             – rect-based table detection (union-find clustering)
+    detect_heuristic.rs         – heuristic table detection (gap-histogram, body-font tables)
+    detect_lines.rs             – line-based table detection (H/V line grids)
+    grid.rs                     – column/row boundaries, cell assignment
+    format.rs                   – table→Markdown formatting, continuation row merging
+  markdown/
+    convert.rs                  – core line→Markdown loop, struct-tree role support
+    analysis.rs                 – font stats, heading tiers, paragraph thresholds
+    classify.rs                 – line classification (header, list, code, caption)
+    preprocess.rs               – drop cap merging, heading line merging
+    postprocess.rs              – dot leaders, hyphenation, page numbers, URL formatting
+```
+
+## Key design decisions
+
+- **Primary audience is AI agents.** Output optimized for token efficiency and semantic quality, not visual formatting. No cosmetic padding.
+- **Three table detection strategies** run in priority order: rect-based → line-based → heuristic. First valid result wins.
+- **Column detection** uses horizontal projection histograms with valley detection. Multi-item spanning lines (titles, headers) are pre-masked using column-aware thresholds before column assignment.
+- **Newspaper vs tabular** classification determines reading order: newspaper reads columns sequentially, tabular Y-interleaves them.
+- **Tiled-scan detection** catches scanned PDFs with JBIG2/strip images where no single tile exceeds the template threshold but aggregate area does (≥2M pixels).
+- **Garbage text upgrade** reclassifies Mixed PDFs as Scanned when extracted text is <50% alphanumeric.
+- **Tagged PDF support** uses structure tree roles (H1-H6, P, L, Code, BlockQuote) when available, falling back to font-size heuristics.
+
+## Testing
+
+- **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.
+- **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.
+- **Regression suite**: sibling repo `pdf-evals` with 179+ snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing.
+
+## Debugging
+
+```bash
+RUST_LOG=pdf_inspector::extractor::layout=debug cargo run --bin pdf2md -- file.pdf
+RUST_LOG=pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf
+RUST_LOG=pdf_inspector::detector=debug cargo run --release --bin detect-pdf -- file.pdf
+```
+
+## Conventions
+
+- Clippy: use `is_some_and(...)` not `map_or(false, ...)`
+- lopdf quirk: `ParseError` is private — match by string for `InvalidFileHeader`
+- Column limit for tables: 25 (wide statistical tables)
+- `propagate_merged_cells` skipped for >10 columns (spanning rects = background fills)

README.md

@@ -55,12 +55,12 @@ print(result.markdown)   # Markdown string or None
 ### Node.js
 
 ```bash
-npm install @firecrawl/pdf-inspector-js
+npm install @firecrawl/pdf-inspector
 ```
 
 ```javascript
 import { readFileSync } from 'fs';
-import { processPdf, classifyPdf } from '@firecrawl/pdf-inspector-js';
+import { processPdf, classifyPdf } from '@firecrawl/pdf-inspector';
 
 const result = processPdf(readFileSync('document.pdf'));
 console.log(result.pdfType);   // "TextBased", "Scanned", "ImageBased", "Mixed"

napi/README.md

@@ -1,4 +1,4 @@
-# firecrawl-pdf-inspector
+# PDF Inspector
 
 Fast PDF classification and region-based text extraction for Node.js/Bun. Native Rust performance via [napi-rs](https://napi.rs).
 
@@ -7,9 +7,9 @@ Built by [Firecrawl](https://firecrawl.dev) for hybrid OCR pipelines — extract
 ## Install
 
 ```bash
-npm install firecrawl-pdf-inspector
+npm install @firecrawl/pdf-inspector
 # or
-bun add firecrawl-pdf-inspector
+bun add @firecrawl/pdf-inspector
 ```
 
 Prebuilt binaries included for **linux-x64** and **macOS ARM64**. No Rust toolchain needed.
@@ -21,7 +21,7 @@ Prebuilt binaries included for **linux-x64** and **macOS ARM64**. No Rust toolch
 Classify a PDF as TextBased, Scanned, Mixed, or ImageBased (~10-50ms). Returns which pages need OCR.
 
 ```typescript
-import { classifyPdf } from 'firecrawl-pdf-inspector'
+import { classifyPdf } from '@firecrawl/pdf-inspector'
 import { readFileSync } from 'fs'
 
 const pdf = readFileSync('document.pdf')
@@ -40,7 +40,7 @@ Extract text within bounding-box regions from a PDF. Designed for hybrid OCR pip
 Each region result includes a `needsOcr` flag that signals unreliable extraction (empty text, GID-encoded fonts, garbage text, encoding issues).
 
 ```typescript
-import { extractTextInRegions } from 'firecrawl-pdf-inspector'
+import { extractTextInRegions } from '@firecrawl/pdf-inspector'
 
 const result = extractTextInRegions(pdf, [
   {

napi/bin/pdf-inspector.mjs

@@ -0,0 +1,131 @@
+#!/usr/bin/env node
+
+import { readFileSync, writeFileSync } from "fs";
+import { createRequire } from "module";
+
+const require = createRequire(import.meta.url);
+const { version } = require("../package.json");
+
+const HELP = `pdf-inspector v${version} — Fast PDF text extraction to Markdown
+
+Usage:
+  pdf-inspector <file>                  Extract markdown (default)
+  pdf-inspector detect <file>           Classify PDF type
+
+Options:
+  --json                  Output as JSON
+  --pages <pages>         Comma-separated page numbers (e.g. 1,3,5)
+  -o, --output <file>     Write output to file instead of stdout
+  -h, --help              Show this help
+  -v, --version           Show version
+
+Examples:
+  pdf-inspector document.pdf
+  pdf-inspector document.pdf --json
+  pdf-inspector document.pdf --pages 1,2,3
+  pdf-inspector detect document.pdf --json
+  cat document.pdf | pdf-inspector -`;
+
+function die(msg) {
+  process.stderr.write(`error: ${msg}\n`);
+  process.exit(1);
+}
+
+function parseArgs(argv) {
+  const opts = { json: false, pages: null, output: null, file: null, command: "extract" };
+  let i = 0;
+
+  // Check for subcommand
+  if (argv[0] === "detect") {
+    opts.command = "detect";
+    i = 1;
+  }
+
+  while (i < argv.length) {
+    const arg = argv[i];
+    if (arg === "-h" || arg === "--help") {
+      process.stdout.write(HELP + "\n");
+      process.exit(0);
+    } else if (arg === "-v" || arg === "--version") {
+      process.stdout.write(`${version}\n`);
+      process.exit(0);
+    } else if (arg === "--json") {
+      opts.json = true;
+    } else if (arg === "--pages") {
+      i++;
+      if (!argv[i]) die("--pages requires a value (e.g. 1,3,5)");
+      opts.pages = argv[i].split(",").map((p) => {
+        const n = parseInt(p.trim(), 10);
+        if (Number.isNaN(n) || n < 1) die(`invalid page number: ${p}`);
+        return n;
+      });
+    } else if (arg === "-o" || arg === "--output") {
+      i++;
+      if (!argv[i]) die("-o requires a filename");
+      opts.output = argv[i];
+    } else if (arg === "-" || !arg.startsWith("-")) {
+      if (opts.file) die(`unexpected argument: ${arg}`);
+      opts.file = arg;
+    } else {
+      die(`unknown option: ${arg}`);
+    }
+    i++;
+  }
+
+  return opts;
+}
+
+function readInput(file) {
+  if (file === "-") {
+    return readFileSync(0); // stdin fd
+  }
+  try {
+    return readFileSync(file);
+  } catch (err) {
+    if (err.code === "ENOENT") die(`file not found: ${file}`);
+    die(err.message);
+  }
+}
+
+function output(text, outputPath) {
+  if (outputPath) {
+    writeFileSync(outputPath, text);
+  } else {
+    process.stdout.write(text);
+  }
+}
+
+// ---- main ----
+
+const opts = parseArgs(process.argv.slice(2));
+
+if (!opts.file) {
+  // Check if stdin is piped
+  if (process.stdin.isTTY !== false) {
+    process.stderr.write(HELP + "\n");
+    process.exit(1);
+  }
+  opts.file = "-";
+}
+
+const { processPdf, classifyPdf } = await import("../index.js");
+const buffer = readInput(opts.file);
+
+if (opts.command === "detect") {
+  const result = classifyPdf(buffer);
+  if (opts.json) {
+    output(JSON.stringify(result, null, 2) + "\n", opts.output);
+  } else {
+    const ocr = result.pagesNeedingOcr.length > 0
+      ? `, ${result.pagesNeedingOcr.length} pages need OCR`
+      : "";
+    output(`${result.pdfType} (${result.pageCount} pages, confidence: ${result.confidence.toFixed(2)}${ocr})\n`, opts.output);
+  }
+} else {
+  const result = processPdf(buffer, opts.pages ?? undefined);
+  if (opts.json) {
+    output(JSON.stringify(result, null, 2) + "\n", opts.output);
+  } else {
+    output((result.markdown ?? "") + "\n", opts.output);
+  }
+}

napi/package.json

@@ -1,9 +1,12 @@
 {
-  "name": "firecrawl-pdf-inspector",
-  "version": "0.7.5",
+  "name": "@firecrawl/pdf-inspector",
+  "version": "1.0.0",
   "description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
   "main": "index.js",
   "types": "index.d.ts",
+  "bin": {
+    "pdf-inspector": "bin/pdf-inspector.mjs"
+  },
   "license": "MIT",
   "keywords": [
     "pdf",
@@ -20,6 +23,7 @@
     "index.js",
     "index.d.ts",
     "*.node",
+    "bin/",
     "README.md"
   ],
   "repository": {

napi/src/lib.rs

@@ -99,26 +99,6 @@ pub struct PageRegionTexts {
     pub regions: Vec<RegionText>,
 }
 
-/// LaTeX reconstruction result for a single formula region.
-#[napi(object)]
-pub struct FormulaLatexResult {
-    /// Reconstructed LaTeX string.
-    pub latex: String,
-    /// The linearized raw text (before LaTeX reconstruction).
-    pub raw_text: String,
-    /// Heuristic confidence in the LaTeX output (0.0–1.0).
-    pub confidence: f64,
-    /// `true` when extraction failed entirely and GPU OCR is needed.
-    pub needs_ocr: bool,
-}
-
-/// LaTeX reconstruction results for one page's formula regions.
-#[napi(object)]
-pub struct PageFormulaLatexResults {
-    pub page: u32,
-    pub regions: Vec<FormulaLatexResult>,
-}
-
 // ---------------------------------------------------------------------------
 // Helpers
 // ---------------------------------------------------------------------------
@@ -337,70 +317,6 @@ pub fn extract_tables_in_regions(
     })
 }
 
-/// Extract formula text within bounding-box regions from a PDF.
-///
-/// Like `extractTextInRegions` but uses formula-specific quality checks.
-/// Formula text is legitimately symbol-heavy (Greek letters, math operators)
-/// so the generic garbage-text check is relaxed. When the text decodes
-/// cleanly, `needsOcr` is `false` — the caller can skip GPU OCR.
-///
-/// Coordinates are PDF points with top-left origin.
-#[napi]
-pub fn extract_formulas_in_regions(
-    buffer: Buffer,
-    page_regions: Vec<PageRegions>,
-) -> Result<Vec<PageRegionTexts>> {
-    let bytes: Vec<u8> = buffer.to_vec();
-    let regions = parse_page_regions(&page_regions);
-
-    catch_panic("extract_formulas_in_regions", move || {
-        let results = pdf_inspector::extract_formulas_in_regions_mem(&bytes, &regions)
-            .map_err(|e| to_napi_err(e, "extract_formulas_in_regions"))?;
-        Ok(to_page_region_texts(results))
-    })
-}
-
-/// Extract formula text within bounding-box regions and reconstruct LaTeX.
-///
-/// Like `extractFormulasInRegions` but additionally reconstructs LaTeX from
-/// the positioned text items. Each result includes the raw text, reconstructed
-/// LaTeX, a confidence score, and the `needsOcr` flag.
-///
-/// The confidence score (0.0–1.0) indicates how reliable the heuristic LaTeX
-/// reconstruction is. The caller should use this to decide whether to trust
-/// the LaTeX or fall back to GPU OCR.
-///
-/// Coordinates are PDF points with top-left origin.
-#[napi]
-pub fn extract_formulas_in_regions_as_latex(
-    buffer: Buffer,
-    page_regions: Vec<PageRegions>,
-) -> Result<Vec<PageFormulaLatexResults>> {
-    let bytes: Vec<u8> = buffer.to_vec();
-    let regions = parse_page_regions(&page_regions);
-
-    catch_panic("extract_formulas_in_regions_as_latex", move || {
-        let results = pdf_inspector::extract_formulas_in_regions_as_latex(&bytes, &regions)
-            .map_err(|e| to_napi_err(e, "extract_formulas_in_regions_as_latex"))?;
-        Ok(results
-            .into_iter()
-            .map(|page_result| PageFormulaLatexResults {
-                page: page_result.page,
-                regions: page_result
-                    .regions
-                    .into_iter()
-                    .map(|r| FormulaLatexResult {
-                        latex: r.latex,
-                        raw_text: r.raw_text,
-                        confidence: r.confidence as f64,
-                        needs_ocr: r.needs_ocr,
-                    })
-                    .collect(),
-            })
-            .collect())
-    })
-}
-
 /// Per-page markdown extraction result.
 #[napi(object)]
 pub struct PageMarkdownResult {

src/extractor/content_stream.rs

@@ -216,6 +216,7 @@ pub(crate) fn extract_page_text_items(
     let mut marked_content_stack: Vec<MarkedContentEntry> = Vec::new();
     let mut suppress_glyph_extraction = false;
     let mut actual_text_start_tm: Option<[f32; 6]> = None; // text matrix at BDC entry
+    let mut actual_text_glyph_tm: Option<[f32; 6]> = None; // text matrix at first glyph inside BDC
     /// Get the innermost MCID from the marked content stack.
     fn current_mcid(stack: &[MarkedContentEntry]) -> Option<i64> {
         stack.iter().rev().find_map(|e| e.mcid)
@@ -349,8 +350,15 @@ pub(crate) fn extract_page_text_items(
                             )
                         })
                     });
-                    // ActualText: suppress glyph extraction, just advance text matrix
+                    // ActualText: suppress glyph extraction, just advance text matrix.
+                    // Capture the FIRST glyph's text matrix as the rendering position
+                    // for the ActualText item. Td ops between BDC and the first Tj
+                    // may have moved the position to the correct line — the BDC-entry
+                    // position (actual_text_start_tm) can be on the previous line.
                     if suppress_glyph_extraction {
+                        if actual_text_glyph_tm.is_none() {
+                            actual_text_glyph_tm = Some(text_matrix);
+                        }
                         if let Some(w_ts) = w_ts_opt {
                             text_matrix[4] += w_ts * text_matrix[0];
                             text_matrix[5] += w_ts * text_matrix[1];
@@ -425,6 +433,10 @@ pub(crate) fn extract_page_text_items(
                         let font_info = font_widths.get(&current_font);
                         let is_invisible = (text_rendering_mode == 3 && !include_invisible)
                             || suppress_glyph_extraction;
+                        // Capture first-glyph position for ActualText
+                        if suppress_glyph_extraction && actual_text_glyph_tm.is_none() {
+                            actual_text_glyph_tm = Some(text_matrix);
+                        }
 
                         // Compute space threshold based on font metrics when available
                         let space_threshold = if let Some(font_info) = font_info {
@@ -700,6 +712,7 @@ pub(crate) fn extract_page_text_items(
                 if actual_text.is_some() {
                     suppress_glyph_extraction = true;
                     actual_text_start_tm = Some(text_matrix);
+                    actual_text_glyph_tm = None; // reset — will be captured at first Tj/TJ
                 }
                 marked_content_stack.push(MarkedContentEntry { actual_text, mcid });
             }
@@ -707,8 +720,13 @@ pub(crate) fn extract_page_text_items(
                 // End Marked Content — emit ActualText item with correct width
                 if let Some(entry) = marked_content_stack.pop() {
                     if let Some(at) = entry.actual_text {
-                        // Compute width from text matrix advancement during BDC..EMC
-                        if let Some(start_tm) = actual_text_start_tm.take() {
+                        // Use the first-glyph position (if available) instead of the
+                        // BDC-entry position. Td operators between BDC and the first
+                        // Tj may have moved the text position to the correct line —
+                        // the BDC-entry position can be on the previous line.
+                        let glyph_tm = actual_text_glyph_tm.take();
+                        let entry_tm = actual_text_start_tm.take();
+                        if let Some(start_tm) = glyph_tm.or(entry_tm) {
                             let combined = multiply_matrices(&start_tm, &ctm);
                             if combined[0].abs() >= combined[1].abs() {
                                 rotation_votes.horizontal += 1;

src/formula_latex/unicode_map.rs

@@ -1,418 +0,0 @@
-//! Unicode to LaTeX character mapping.
-//!
-//! Maps Unicode math symbols, Greek letters, operators, and relations to their
-//! LaTeX command equivalents. Only includes characters that commonly appear in
-//! PDF text extraction of mathematical formulas.
-
-use std::collections::HashMap;
-use std::sync::OnceLock;
-
-/// Returns a reference to the global Unicode → LaTeX mapping table.
-pub fn unicode_to_latex_map() -> &'static HashMap<char, &'static str> {
-    static MAP: OnceLock<HashMap<char, &'static str>> = OnceLock::new();
-    MAP.get_or_init(build_map)
-}
-
-fn build_map() -> HashMap<char, &'static str> {
-    let entries: &[(char, &str)] = &[
-        // ── Greek lowercase ─────────────────────────────────────────
-        ('\u{03B1}', r"\alpha"),
-        ('\u{03B2}', r"\beta"),
-        ('\u{03B3}', r"\gamma"),
-        ('\u{03B4}', r"\delta"),
-        ('\u{03B5}', r"\varepsilon"),
-        ('\u{03F5}', r"\epsilon"),
-        ('\u{03B6}', r"\zeta"),
-        ('\u{03B7}', r"\eta"),
-        ('\u{03B8}', r"\theta"),
-        ('\u{03D1}', r"\vartheta"),
-        ('\u{03B9}', r"\iota"),
-        ('\u{03BA}', r"\kappa"),
-        ('\u{03BB}', r"\lambda"),
-        ('\u{03BC}', r"\mu"),
-        ('\u{03BD}', r"\nu"),
-        ('\u{03BE}', r"\xi"),
-        ('\u{03C0}', r"\pi"),
-        ('\u{03D6}', r"\varpi"),
-        ('\u{03C1}', r"\rho"),
-        ('\u{03C2}', r"\varsigma"),
-        ('\u{03C3}', r"\sigma"),
-        ('\u{03C4}', r"\tau"),
-        ('\u{03C5}', r"\upsilon"),
-        ('\u{03C6}', r"\varphi"),
-        ('\u{03D5}', r"\phi"),
-        ('\u{03C7}', r"\chi"),
-        ('\u{03C8}', r"\psi"),
-        ('\u{03C9}', r"\omega"),
-        // ── Greek uppercase ─────────────────────────────────────────
-        ('\u{0393}', r"\Gamma"),
-        ('\u{0394}', r"\Delta"),
-        ('\u{0398}', r"\Theta"),
-        ('\u{039B}', r"\Lambda"),
-        ('\u{039E}', r"\Xi"),
-        ('\u{03A0}', r"\Pi"),
-        ('\u{03A3}', r"\Sigma"),
-        ('\u{03A5}', r"\Upsilon"),
-        ('\u{03A6}', r"\Phi"),
-        ('\u{03A8}', r"\Psi"),
-        ('\u{03A9}', r"\Omega"),
-        // ── Large operators ─────────────────────────────────────────
-        ('\u{222B}', r"\int"),
-        ('\u{222C}', r"\iint"),
-        ('\u{222D}', r"\iiint"),
-        ('\u{222E}', r"\oint"),
-        ('\u{2211}', r"\sum"),
-        ('\u{220F}', r"\prod"),
-        ('\u{2210}', r"\coprod"),
-        // ── Roots / radicals ────────────────────────────────────────
-        ('\u{221A}', r"\sqrt"),
-        // ── Calculus / differential ─────────────────────────────────
-        ('\u{2202}', r"\partial"),
-        ('\u{2207}', r"\nabla"),
-        // ── Binary operators ────────────────────────────────────────
-        ('\u{00B1}', r"\pm"),
-        ('\u{2213}', r"\mp"),
-        ('\u{00D7}', r"\times"),
-        ('\u{00F7}', r"\div"),
-        ('\u{2217}', r"\ast"),
-        ('\u{22C6}', r"\star"),
-        ('\u{00B7}', r"\cdot"),
-        ('\u{2219}', r"\bullet"),
-        ('\u{2218}', r"\circ"),
-        ('\u{2020}', r"\dagger"),
-        ('\u{2021}', r"\ddagger"),
-        ('\u{2295}', r"\oplus"),
-        ('\u{2297}', r"\otimes"),
-        ('\u{2227}', r"\wedge"),
-        ('\u{2228}', r"\vee"),
-        ('\u{2229}', r"\cap"),
-        ('\u{222A}', r"\cup"),
-        // ── Relations ───────────────────────────────────────────────
-        ('\u{2264}', r"\leq"),
-        ('\u{2265}', r"\geq"),
-        ('\u{2260}', r"\neq"),
-        ('\u{2248}', r"\approx"),
-        ('\u{223C}', r"\sim"),
-        ('\u{2243}', r"\simeq"),
-        ('\u{2261}', r"\equiv"),
-        ('\u{226A}', r"\ll"),
-        ('\u{226B}', r"\gg"),
-        ('\u{221D}', r"\propto"),
-        ('\u{2208}', r"\in"),
-        ('\u{2209}', r"\notin"),
-        ('\u{220B}', r"\ni"),
-        ('\u{2282}', r"\subset"),
-        ('\u{2283}', r"\supset"),
-        ('\u{2286}', r"\subseteq"),
-        ('\u{2287}', r"\supseteq"),
-        ('\u{22A2}', r"\vdash"),
-        ('\u{22A3}', r"\dashv"),
-        ('\u{22A4}', r"\top"),
-        ('\u{22A5}', r"\bot"),
-        ('\u{2225}', r"\parallel"),
-        ('\u{22A5}', r"\perp"),
-        // ── Arrows ──────────────────────────────────────────────────
-        ('\u{2190}', r"\leftarrow"),
-        ('\u{2192}', r"\to"),
-        ('\u{2191}', r"\uparrow"),
-        ('\u{2193}', r"\downarrow"),
-        ('\u{2194}', r"\leftrightarrow"),
-        ('\u{21D0}', r"\Leftarrow"),
-        ('\u{21D2}', r"\Rightarrow"),
-        ('\u{21D4}', r"\Leftrightarrow"),
-        ('\u{21A6}', r"\mapsto"),
-        ('\u{2197}', r"\nearrow"),
-        ('\u{2198}', r"\searrow"),
-        // ── Miscellaneous symbols ───────────────────────────────────
-        ('\u{221E}', r"\infty"),
-        ('\u{2200}', r"\forall"),
-        ('\u{2203}', r"\exists"),
-        ('\u{2204}', r"\nexists"),
-        ('\u{2205}', r"\emptyset"),
-        ('\u{00AC}', r"\neg"),
-        ('\u{00B0}', r"^\circ"),
-        ('\u{2032}', r"'"),  // prime (common in physics: x')
-        ('\u{2033}', r"''"), // double prime
-        ('\u{210F}', r"\hbar"),
-        ('\u{2113}', r"\ell"),
-        ('\u{211C}', r"\Re"),
-        ('\u{2111}', r"\Im"),
-        ('\u{2118}', r"\wp"),
-        ('\u{2135}', r"\aleph"),
-        // ── Dots ────────────────────────────────────────────────────
-        ('\u{22EF}', r"\cdots"),
-        ('\u{22EE}', r"\vdots"),
-        ('\u{22F1}', r"\ddots"),
-        ('\u{2026}', r"\ldots"),
-        // ── Delimiters / brackets ───────────────────────────────────
-        ('\u{27E8}', r"\langle"),
-        ('\u{27E9}', r"\rangle"),
-        ('\u{2308}', r"\lceil"),
-        ('\u{2309}', r"\rceil"),
-        ('\u{230A}', r"\lfloor"),
-        ('\u{230B}', r"\rfloor"),
-        ('\u{2016}', r"\|"),
-        // ── Accents / decorations (as standalone chars) ─────────────
-        ('\u{0302}', r"\hat{}"),
-        ('\u{0303}', r"\tilde{}"),
-        ('\u{0304}', r"\bar{}"),
-        ('\u{0307}', r"\dot{}"),
-        ('\u{0308}', r"\ddot{}"),
-        ('\u{20D7}', r"\vec{}"),
-        // Hat/tilde as standalone characters (sometimes extracted separately)
-        ('\u{02C6}', r"\hat{}"),
-        ('\u{02DC}', r"\tilde{}"),
-        // ── Subscript/superscript digits (Unicode) ──────────────────
-        ('\u{2070}', "^{0}"),
-        ('\u{00B9}', "^{1}"),
-        ('\u{00B2}', "^{2}"),
-        ('\u{00B3}', "^{3}"),
-        ('\u{2074}', "^{4}"),
-        ('\u{2075}', "^{5}"),
-        ('\u{2076}', "^{6}"),
-        ('\u{2077}', "^{7}"),
-        ('\u{2078}', "^{8}"),
-        ('\u{2079}', "^{9}"),
-        ('\u{207A}', "^{+}"),
-        ('\u{207B}', "^{-}"),
-        ('\u{2080}', "_{0}"),
-        ('\u{2081}', "_{1}"),
-        ('\u{2082}', "_{2}"),
-        ('\u{2083}', "_{3}"),
-        ('\u{2084}', "_{4}"),
-        ('\u{2085}', "_{5}"),
-        ('\u{2086}', "_{6}"),
-        ('\u{2087}', "_{7}"),
-        ('\u{2088}', "_{8}"),
-        ('\u{2089}', "_{9}"),
-        ('\u{208A}', "_{+}"),
-        ('\u{208B}', "_{-}"),
-        // ── Math italic letters (sometimes used in PDF fonts) ───────
-        // These map back to plain ASCII in LaTeX (math mode handles italics)
-    ];
-
-    let mut map = HashMap::with_capacity(entries.len());
-    for &(ch, latex) in entries {
-        map.insert(ch, latex);
-    }
-    map
-}
-
-/// Convert a single character to its LaTeX representation.
-///
-/// Returns `Some(latex_str)` if the character has a known mapping,
-/// or `None` if it should be kept as-is.
-pub fn char_to_latex(ch: char) -> Option<&'static str> {
-    unicode_to_latex_map().get(&ch).copied()
-}
-
-/// Returns true if a character is a "known math character" — either ASCII
-/// alphanumeric, basic punctuation used in math, or a mapped Unicode symbol.
-pub fn is_known_math_char(ch: char) -> bool {
-    if ch.is_ascii_alphanumeric() {
-        return true;
-    }
-    // Common ASCII math characters
-    matches!(
-        ch,
-        '+' | '-'
-            | '*'
-            | '/'
-            | '='
-            | '<'
-            | '>'
-            | '('
-            | ')'
-            | '['
-            | ']'
-            | '{'
-            | '}'
-            | ','
-            | '.'
-            | ':'
-            | ';'
-            | '!'
-            | '|'
-            | '\''
-            | '"'
-            | '^'
-            | '_'
-            | '~'
-            | ' '
-            | '\n'
-            | '\t'
-    ) || unicode_to_latex_map().contains_key(&ch)
-}
-
-/// Convert a string of text to LaTeX, applying per-character mappings.
-/// Characters without mappings are left as-is.
-///
-/// Returns `(latex_string, fraction_of_chars_that_were_known)`.
-pub fn text_to_latex_chars(text: &str) -> (String, f32) {
-    let map = unicode_to_latex_map();
-    let mut result = String::with_capacity(text.len() * 2);
-    let mut total_nonws = 0usize;
-    let mut known = 0usize;
-
-    for ch in text.chars() {
-        if ch.is_whitespace() {
-            result.push(ch);
-            continue;
-        }
-        total_nonws += 1;
-
-        if let Some(latex) = map.get(&ch) {
-            // Add space before LaTeX commands that start with backslash
-            // to prevent them from merging with preceding text
-            if latex.starts_with('\\') && !result.is_empty() && !result.ends_with(' ') {
-                // Only add space if the last char is alphanumeric (to avoid "x \alpha" but allow "( \alpha")
-                let last = result.chars().last().unwrap();
-                if last.is_alphanumeric() || last == '}' {
-                    result.push(' ');
-                }
-            }
-            result.push_str(latex);
-            // Add trailing space after LaTeX commands so next char doesn't merge
-            if latex.starts_with('\\') && !latex.ends_with('}') && !latex.ends_with('\'') {
-                result.push(' ');
-            }
-            known += 1;
-        } else if is_known_math_char(ch) {
-            result.push(ch);
-            known += 1;
-        } else {
-            // Unknown character — keep it but it lowers confidence
-            result.push(ch);
-        }
-    }
-
-    let frac = if total_nonws == 0 {
-        1.0
-    } else {
-        known as f32 / total_nonws as f32
-    };
-    (result, frac)
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn test_greek_lowercase() {
-        assert_eq!(char_to_latex('\u{03B1}'), Some(r"\alpha"));
-        assert_eq!(char_to_latex('\u{03B2}'), Some(r"\beta"));
-        assert_eq!(char_to_latex('\u{03B3}'), Some(r"\gamma"));
-        assert_eq!(char_to_latex('\u{03B4}'), Some(r"\delta"));
-        assert_eq!(char_to_latex('\u{03B5}'), Some(r"\varepsilon"));
-        assert_eq!(char_to_latex('\u{03B6}'), Some(r"\zeta"));
-        assert_eq!(char_to_latex('\u{03B7}'), Some(r"\eta"));
-        assert_eq!(char_to_latex('\u{03B8}'), Some(r"\theta"));
-        assert_eq!(char_to_latex('\u{03D1}'), Some(r"\vartheta"));
-        assert_eq!(char_to_latex('\u{03B9}'), Some(r"\iota"));
-        assert_eq!(char_to_latex('\u{03BA}'), Some(r"\kappa"));
-        assert_eq!(char_to_latex('\u{03BB}'), Some(r"\lambda"));
-        assert_eq!(char_to_latex('\u{03BC}'), Some(r"\mu"));
-        assert_eq!(char_to_latex('\u{03BD}'), Some(r"\nu"));
-        assert_eq!(char_to_latex('\u{03BE}'), Some(r"\xi"));
-        assert_eq!(char_to_latex('\u{03C0}'), Some(r"\pi"));
-        assert_eq!(char_to_latex('\u{03C1}'), Some(r"\rho"));
-        assert_eq!(char_to_latex('\u{03C3}'), Some(r"\sigma"));
-        assert_eq!(char_to_latex('\u{03C4}'), Some(r"\tau"));
-        assert_eq!(char_to_latex('\u{03C9}'), Some(r"\omega"));
-    }
-
-    #[test]
-    fn test_greek_uppercase() {
-        assert_eq!(char_to_latex('\u{0393}'), Some(r"\Gamma"));
-        assert_eq!(char_to_latex('\u{0394}'), Some(r"\Delta"));
-        assert_eq!(char_to_latex('\u{0398}'), Some(r"\Theta"));
-        assert_eq!(char_to_latex('\u{039B}'), Some(r"\Lambda"));
-        assert_eq!(char_to_latex('\u{03A3}'), Some(r"\Sigma"));
-        assert_eq!(char_to_latex('\u{03A6}'), Some(r"\Phi"));
-        assert_eq!(char_to_latex('\u{03A9}'), Some(r"\Omega"));
-    }
-
-    #[test]
-    fn test_operators() {
-        assert_eq!(char_to_latex('\u{222B}'), Some(r"\int"));
-        assert_eq!(char_to_latex('\u{2211}'), Some(r"\sum"));
-        assert_eq!(char_to_latex('\u{220F}'), Some(r"\prod"));
-        assert_eq!(char_to_latex('\u{221A}'), Some(r"\sqrt"));
-        assert_eq!(char_to_latex('\u{2202}'), Some(r"\partial"));
-        assert_eq!(char_to_latex('\u{2207}'), Some(r"\nabla"));
-    }
-
-    #[test]
-    fn test_relations() {
-        assert_eq!(char_to_latex('\u{2264}'), Some(r"\leq"));
-        assert_eq!(char_to_latex('\u{2265}'), Some(r"\geq"));
-        assert_eq!(char_to_latex('\u{2260}'), Some(r"\neq"));
-        assert_eq!(char_to_latex('\u{2248}'), Some(r"\approx"));
-        assert_eq!(char_to_latex('\u{223C}'), Some(r"\sim"));
-        assert_eq!(char_to_latex('\u{226A}'), Some(r"\ll"));
-        assert_eq!(char_to_latex('\u{226B}'), Some(r"\gg"));
-        assert_eq!(char_to_latex('\u{221E}'), Some(r"\infty"));
-        assert_eq!(char_to_latex('\u{2208}'), Some(r"\in"));
-        assert_eq!(char_to_latex('\u{2209}'), Some(r"\notin"));
-        assert_eq!(char_to_latex('\u{2282}'), Some(r"\subset"));
-    }
-
-    #[test]
-    fn test_misc_symbols() {
-        assert_eq!(char_to_latex('\u{00B1}'), Some(r"\pm"));
-        assert_eq!(char_to_latex('\u{00D7}'), Some(r"\times"));
-        assert_eq!(char_to_latex('\u{00B7}'), Some(r"\cdot"));
-        assert_eq!(char_to_latex('\u{00B0}'), Some(r"^\circ"));
-        assert_eq!(char_to_latex('\u{2192}'), Some(r"\to"));
-        assert_eq!(char_to_latex('\u{21D2}'), Some(r"\Rightarrow"));
-        assert_eq!(char_to_latex('\u{210F}'), Some(r"\hbar"));
-    }
-
-    #[test]
-    fn test_unicode_super_sub_digits() {
-        assert_eq!(char_to_latex('\u{00B2}'), Some("^{2}"));
-        assert_eq!(char_to_latex('\u{00B3}'), Some("^{3}"));
-        assert_eq!(char_to_latex('\u{2082}'), Some("_{2}"));
-        assert_eq!(char_to_latex('\u{2083}'), Some("_{3}"));
-    }
-
-    #[test]
-    fn test_is_known_math_char() {
-        // ASCII math
-        assert!(is_known_math_char('+'));
-        assert!(is_known_math_char('='));
-        assert!(is_known_math_char('('));
-        assert!(is_known_math_char('x'));
-        assert!(is_known_math_char('0'));
-        // Mapped Unicode
-        assert!(is_known_math_char('\u{03B1}')); // alpha
-        assert!(is_known_math_char('\u{2264}')); // leq
-                                                 // Unknown
-        assert!(!is_known_math_char('\u{E000}')); // PUA
-        assert!(!is_known_math_char('\u{4E00}')); // CJK
-    }
-
-    #[test]
-    fn test_text_to_latex_simple() {
-        let (latex, frac) = text_to_latex_chars("x + y");
-        assert_eq!(latex, "x + y");
-        assert!((frac - 1.0).abs() < 0.01);
-    }
-
-    #[test]
-    fn test_text_to_latex_greek() {
-        let (latex, frac) = text_to_latex_chars("αβγ");
-        assert!(latex.contains(r"\alpha"));
-        assert!(latex.contains(r"\beta"));
-        assert!(latex.contains(r"\gamma"));
-        assert!((frac - 1.0).abs() < 0.01);
-    }
-
-    #[test]
-    fn test_text_to_latex_mixed() {
-        let (latex, frac) = text_to_latex_chars("x ≤ y");
-        assert!(latex.contains(r"\leq"));
-        assert!((frac - 1.0).abs() < 0.01);
-    }
-}

src/lib.rs

@@ -1,3 +1,9 @@
+// Rust 1.95 introduced collapsible_match for `if` inside match arms.
+// The content-stream parsers use this pattern extensively (match on operator
+// name, then check `in_text_block && !op.operands.is_empty()`). Collapsing
+// these into match guards would hurt readability. Allow crate-wide.
+#![allow(clippy::collapsible_match)]
+
 //! Smart PDF detection and text extraction using lopdf
 //!
 //! # Quick start
@@ -28,7 +34,6 @@ pub mod python;
 pub mod adobe_korea1;
 pub mod detector;
 pub mod extractor;
-pub mod formula_latex;
 pub mod glyph_names;
 pub mod markdown;
 pub mod process_mode;
@@ -463,98 +468,6 @@ pub struct PageRegionResult {
     pub regions: Vec<RegionText>,
 }
 
-/// Shared page extraction state for region-based extraction functions.
-struct RegionExtractionData {
-    items_by_page: HashMap<u32, Vec<TextItem>>,
-    page_heights: HashMap<u32, f32>,
-    #[allow(dead_code)]
-    gid_pages: HashSet<u32>,
-    page_thresholds: HashMap<u32, f32>,
-    rotated_pages: HashSet<u32>,
-}
-
-/// Extract text items, page heights, and metadata for the pages needed by region queries.
-///
-/// This is the shared boilerplate for `extract_text_in_regions_mem`,
-/// `extract_tables_in_regions_mem`, and `extract_formulas_in_regions_mem`.
-fn prepare_region_extraction(
-    buffer: &[u8],
-    page_regions: &[(u32, Vec<[f32; 4]>)],
-) -> Result<RegionExtractionData, PdfError> {
-    validate_pdf_bytes(buffer)?;
-    let (doc, _page_count) = load_document_from_mem(buffer)?;
-    let pages = doc.get_pages();
-
-    let needed_pages: HashSet<u32> = page_regions.iter().map(|(p, _)| p + 1).collect();
-
-    // Fast mode: skip expensive TrueType font fallback parsing.
-    // Fonts that can't be decoded from ToUnicode alone will produce empty/garbage
-    // text, triggering needs_ocr=true → GPU OCR fallback in the pipeline.
-    let font_cmaps = FontCMaps::from_doc_pages_fast(&doc, Some(&needed_pages));
-
-    let mut items_by_page: HashMap<u32, Vec<TextItem>> = HashMap::new();
-    let mut page_heights: HashMap<u32, f32> = HashMap::new();
-    let mut gid_pages: HashSet<u32> = HashSet::new();
-    let mut page_thresholds: HashMap<u32, f32> = HashMap::new();
-    let mut rotated_pages: HashSet<u32> = HashSet::new();
-
-    for (page_num, &page_id) in pages.iter() {
-        if !needed_pages.contains(page_num) {
-            continue;
-        }
-
-        let height = get_page_height(&doc, page_id).unwrap_or(792.0);
-        page_heights.insert(*page_num, height);
-
-        let ((mut items, _rects, _lines), has_gid, coords_rotated) =
-            extractor::content_stream::extract_page_text_items(
-                &doc,
-                page_id,
-                *page_num,
-                &font_cmaps,
-                false,
-            )?;
-        let threshold = text_utils::fix_letterspaced_items(&mut items);
-        if threshold > 0.10 {
-            page_thresholds.insert(*page_num, threshold);
-        }
-        if has_gid {
-            gid_pages.insert(*page_num);
-        }
-        if coords_rotated {
-            rotated_pages.insert(*page_num);
-        }
-        items_by_page.insert(*page_num, items);
-    }
-
-    Ok(RegionExtractionData {
-        items_by_page,
-        page_heights,
-        gid_pages,
-        page_thresholds,
-        rotated_pages,
-    })
-}
-
-/// Resolve per-page coord space and adaptive threshold for a given page.
-fn page_region_context(
-    data: &RegionExtractionData,
-    page_1idx: u32,
-) -> (f32, f32, RegionCoordSpace) {
-    let page_h = data.page_heights.get(&page_1idx).copied().unwrap_or(792.0);
-    let adaptive_threshold = data
-        .page_thresholds
-        .get(&page_1idx)
-        .copied()
-        .unwrap_or(0.10);
-    let coords = if data.rotated_pages.contains(&page_1idx) {
-        RegionCoordSpace::Rotated90Ccw
-    } else {
-        RegionCoordSpace::Standard
-    };
-    (page_h, adaptive_threshold, coords)
-}
-
 /// Extract text within bounding-box regions from a PDF in memory.
 ///
 /// This is designed for hybrid OCR pipelines: a layout model detects regions
@@ -578,14 +491,70 @@ pub fn extract_text_in_regions_mem(
     buffer: &[u8],
     page_regions: &[(u32, Vec<[f32; 4]>)],
 ) -> Result<Vec<PageRegionResult>, PdfError> {
-    let data = prepare_region_extraction(buffer, page_regions)?;
+    validate_pdf_bytes(buffer)?;
+    let (doc, _page_count) = load_document_from_mem(buffer)?;
+    let pages = doc.get_pages();
 
+    // Build a set of pages we need to extract (1-indexed for lopdf)
+    let needed_pages: HashSet<u32> = page_regions.iter().map(|(p, _)| p + 1).collect();
+
+    // Fast mode: skip expensive TrueType font fallback parsing.
+    // Fonts that can't be decoded from ToUnicode alone will produce empty/garbage
+    // text, triggering needs_ocr=true → GPU OCR fallback in the pipeline.
+    let font_cmaps = FontCMaps::from_doc_pages_fast(&doc, Some(&needed_pages));
+
+    // Extract text items for needed pages only
+    let mut items_by_page: HashMap<u32, Vec<TextItem>> = HashMap::new();
+    let mut page_heights: HashMap<u32, f32> = HashMap::new();
+    let mut gid_pages: HashSet<u32> = HashSet::new();
+    let mut page_thresholds: HashMap<u32, f32> = HashMap::new();
+    let mut rotated_pages: HashSet<u32> = HashSet::new();
+
+    for (page_num, &page_id) in pages.iter() {
+        if !needed_pages.contains(page_num) {
+            continue;
+        }
+
+        // Get page height from MediaBox for coordinate flip
+        let height = get_page_height(&doc, page_id).unwrap_or(792.0);
+        page_heights.insert(*page_num, height);
+
+        // Extract text items for this page
+        let ((mut items, _rects, _lines), has_gid, coords_rotated) =
+            extractor::content_stream::extract_page_text_items(
+                &doc,
+                page_id,
+                *page_num,
+                &font_cmaps,
+                false,
+            )?;
+        let threshold = text_utils::fix_letterspaced_items(&mut items);
+        if threshold > 0.10 {
+            page_thresholds.insert(*page_num, threshold);
+        }
+        if has_gid {
+            gid_pages.insert(*page_num);
+        }
+        if coords_rotated {
+            rotated_pages.insert(*page_num);
+        }
+        items_by_page.insert(*page_num, items);
+    }
+
+    // For each page's regions, filter and assemble text
     let mut results = Vec::with_capacity(page_regions.len());
 
     for (page_0idx, regions) in page_regions {
         let page_1idx = page_0idx + 1;
-        let items = data.items_by_page.get(&page_1idx);
-        let (page_h, adaptive_threshold, coords) = page_region_context(&data, page_1idx);
+        let items = items_by_page.get(&page_1idx);
+        let page_h = page_heights.get(&page_1idx).copied().unwrap_or(792.0);
+        let _page_has_gid = gid_pages.contains(&page_1idx);
+        let adaptive_threshold = page_thresholds.get(&page_1idx).copied().unwrap_or(0.10);
+        let coords = if rotated_pages.contains(&page_1idx) {
+            RegionCoordSpace::Rotated90Ccw
+        } else {
+            RegionCoordSpace::Standard
+        };
 
         let mut page_results = Vec::with_capacity(regions.len());
 
@@ -639,20 +608,74 @@ pub fn extract_tables_in_regions_mem(
     buffer: &[u8],
     page_regions: &[(u32, Vec<[f32; 4]>)],
 ) -> Result<Vec<PageRegionResult>, PdfError> {
-    let data = prepare_region_extraction(buffer, page_regions)?;
+    validate_pdf_bytes(buffer)?;
+    let (doc, _page_count) = load_document_from_mem(buffer)?;
+    let pages = doc.get_pages();
+
+    let needed_pages: HashSet<u32> = page_regions.iter().map(|(p, _)| p + 1).collect();
+    let font_cmaps = FontCMaps::from_doc_pages_fast(&doc, Some(&needed_pages));
+
+    let mut items_by_page: HashMap<u32, Vec<TextItem>> = HashMap::new();
+    let mut page_heights: HashMap<u32, f32> = HashMap::new();
+    let mut gid_pages: HashSet<u32> = HashSet::new();
+    let mut page_thresholds: HashMap<u32, f32> = HashMap::new();
+    let mut rotated_pages: HashSet<u32> = HashSet::new();
+
+    for (page_num, &page_id) in pages.iter() {
+        if !needed_pages.contains(page_num) {
+            continue;
+        }
+        let height = get_page_height(&doc, page_id).unwrap_or(792.0);
+        page_heights.insert(*page_num, height);
+
+        let ((mut items, _rects, _lines), has_gid, coords_rotated) =
+            extractor::content_stream::extract_page_text_items(
+                &doc,
+                page_id,
+                *page_num,
+                &font_cmaps,
+                false,
+            )?;
+        let threshold = text_utils::fix_letterspaced_items(&mut items);
+        if threshold > 0.10 {
+            page_thresholds.insert(*page_num, threshold);
+        }
+        if has_gid {
+            gid_pages.insert(*page_num);
+        }
+        if coords_rotated {
+            rotated_pages.insert(*page_num);
+        }
+        items_by_page.insert(*page_num, items);
+    }
 
     let mut results = Vec::with_capacity(page_regions.len());
 
     for (page_0idx, regions) in page_regions {
         let page_1idx = page_0idx + 1;
-        let items = data.items_by_page.get(&page_1idx);
-        let (page_h, _adaptive_threshold, coords) = page_region_context(&data, page_1idx);
+        let items = items_by_page.get(&page_1idx);
+        let page_h = page_heights.get(&page_1idx).copied().unwrap_or(792.0);
+        let _page_has_gid = gid_pages.contains(&page_1idx);
+        let coords = if rotated_pages.contains(&page_1idx) {
+            RegionCoordSpace::Rotated90Ccw
+        } else {
+            RegionCoordSpace::Standard
+        };
 
         let mut page_results = Vec::with_capacity(regions.len());
 
         for rect in regions {
             let [rx1, ry1, rx2, ry2] = *rect;
 
+            // Note: we intentionally DO NOT bail on page_has_gid here.
+            // The GID flag means some font on the page uses unresolvable
+            // glyph IDs, but that font may only appear in a logo or
+            // header — not in the table region. Instead we let the
+            // per-region text quality checks (is_garbage_text, is_cid_garbage,
+            // detect_encoding_issues) reject based on the actual extracted
+            // content. This avoids rejecting clean tables just because an
+            // unrelated decorative font on the same page is GID-encoded.
+
             let matched: Vec<TextItem> = match items {
                 Some(items) => {
                     let bounds = region_bounds(rx1, ry1, rx2, ry2, page_h, coords);
@@ -733,164 +756,6 @@ pub fn extract_tables_in_regions_mem(
     Ok(results)
 }
 
-/// Extract formula text within bounding-box regions from a PDF in memory.
-///
-/// Similar to [`extract_text_in_regions_mem`] but uses formula-specific quality
-/// checks. Formula text is legitimately symbol-heavy (Greek letters, math
-/// operators, subscripts) so the generic `is_garbage_text` check — which rejects
-/// text with <50% alphanumeric characters — would false-positive on valid
-/// formula regions.
-///
-/// When the extracted text decodes cleanly, `needs_ocr` is `false` and the
-/// caller can skip GPU OCR. When extraction fails (empty, PUA-heavy, encoding
-/// issues), `needs_ocr` is `true` for OCR fallback.
-pub fn extract_formulas_in_regions_mem(
-    buffer: &[u8],
-    page_regions: &[(u32, Vec<[f32; 4]>)],
-) -> Result<Vec<PageRegionResult>, PdfError> {
-    let data = prepare_region_extraction(buffer, page_regions)?;
-
-    let mut results = Vec::with_capacity(page_regions.len());
-
-    for (page_0idx, regions) in page_regions {
-        let page_1idx = page_0idx + 1;
-        let items = data.items_by_page.get(&page_1idx);
-        let (page_h, adaptive_threshold, coords) = page_region_context(&data, page_1idx);
-
-        let mut page_results = Vec::with_capacity(regions.len());
-
-        for rect in regions {
-            let [rx1, ry1, rx2, ry2] = *rect;
-
-            let text = match items {
-                Some(items) => collect_text_in_region_with_options(
-                    items,
-                    rx1,
-                    ry1,
-                    rx2,
-                    ry2,
-                    page_h,
-                    coords,
-                    adaptive_threshold,
-                ),
-                None => String::new(),
-            };
-
-            // Formula-specific quality checks:
-            // - Skip is_garbage_text (formulas are legitimately symbol-heavy)
-            // - Keep CID/encoding checks (broken font decode is still broken)
-            // - Add PUA check (extensible delimiters that didn't decode)
-            let needs_ocr = text.trim().is_empty()
-                || is_cid_garbage(&text)
-                || detect_encoding_issues(&text)
-                || is_formula_garbage(&text);
-
-            page_results.push(RegionText { text, needs_ocr });
-        }
-
-        results.push(PageRegionResult {
-            page: *page_0idx,
-            regions: page_results,
-        });
-    }
-
-    Ok(results)
-}
-
-/// Extract formula text within bounding-box regions and reconstruct LaTeX.
-///
-/// For each formula bbox, this function:
-/// 1. Gets positioned text items inside the bbox
-/// 2. Analyzes font sizes and vertical positions to detect sub/superscripts
-/// 3. Detects simple fractions from vertically stacked text
-/// 4. Converts Unicode math symbols to LaTeX commands
-/// 5. Reconstructs structured LaTeX from the positioned items
-///
-/// Each result includes a `confidence` score (0.0–1.0) indicating how reliable
-/// the LaTeX reconstruction is. Low-confidence results should be sent to GPU OCR.
-///
-/// The `needs_ocr` flag is set only when extraction fails entirely (empty text,
-/// encoding garbage), NOT based on confidence — the caller decides the threshold.
-pub fn extract_formulas_in_regions_as_latex(
-    buffer: &[u8],
-    page_regions: &[(u32, Vec<[f32; 4]>)],
-) -> Result<Vec<formula_latex::PageFormulaResult>, PdfError> {
-    let data = prepare_region_extraction(buffer, page_regions)?;
-
-    let mut results = Vec::with_capacity(page_regions.len());
-
-    for (page_0idx, regions) in page_regions {
-        let page_1idx = page_0idx + 1;
-        let items = data.items_by_page.get(&page_1idx);
-        let (page_h, _adaptive_threshold, coords) = page_region_context(&data, page_1idx);
-
-        let mut page_results = Vec::with_capacity(regions.len());
-
-        for rect in regions {
-            let [rx1, ry1, rx2, ry2] = *rect;
-
-            let result = match items {
-                Some(items) => {
-                    // Convert the bbox from top-left origin to the item coordinate space
-                    let bounds = region_bounds(rx1, ry1, rx2, ry2, page_h, coords);
-
-                    // Filter items whose center falls inside the bbox
-                    let matched = formula_latex::filter_items_in_bbox(
-                        items,
-                        bounds.x_min,
-                        bounds.y_min,
-                        bounds.x_max,
-                        bounds.y_max,
-                    );
-
-                    if matched.is_empty() {
-                        formula_latex::FormulaResult {
-                            latex: String::new(),
-                            raw_text: String::new(),
-                            confidence: 0.0,
-                            needs_ocr: true,
-                            confidence_breakdown: Vec::new(),
-                        }
-                    } else {
-                        let (latex, raw_text, confidence, confidence_breakdown) =
-                            formula_latex::reconstruct_latex(&matched);
-
-                        // Same formula-specific quality checks as extract_formulas_in_regions_mem
-                        let needs_ocr = raw_text.trim().is_empty()
-                            || is_cid_garbage(&raw_text)
-                            || detect_encoding_issues(&raw_text)
-                            || is_formula_garbage(&raw_text);
-
-                        formula_latex::FormulaResult {
-                            latex,
-                            raw_text,
-                            confidence,
-                            needs_ocr,
-                            confidence_breakdown,
-                        }
-                    }
-                }
-                None => formula_latex::FormulaResult {
-                    latex: String::new(),
-                    raw_text: String::new(),
-                    confidence: 0.0,
-                    needs_ocr: true,
-                    confidence_breakdown: Vec::new(),
-                },
-            };
-
-            page_results.push(result);
-        }
-
-        results.push(formula_latex::PageFormulaResult {
-            page: *page_0idx,
-            regions: page_results,
-        });
-    }
-
-    Ok(results)
-}
-
 /// Get page height in points from MediaBox.
 fn get_page_height(doc: &Document, page_id: lopdf::ObjectId) -> Option<f32> {
     let page_dict = doc.get_dictionary(page_id).ok()?;
@@ -1488,49 +1353,6 @@ fn is_cid_garbage(text: &str) -> bool {
     high_latin * 5 >= total * 2 && ascii_letters * 3 < total
 }
 
-/// Detect formula text that is unlikely to be usable despite passing generic checks.
-///
-/// Formula text (Greek letters, math operators, variables) is legitimately
-/// symbol-heavy, so `is_garbage_text` would false-positive. This check instead
-/// catches:
-///
-/// 1. **Private Use Area (PUA) characters** — TeX extensible delimiter glyphs
-///    (large brackets from CMEX fonts) often map to PUA U+E000–F8FF when the
-///    ToUnicode CMap is missing. >10% PUA means significant undecoded content.
-///
-/// 2. **Control characters** — C0 controls (U+0000–001F excluding whitespace)
-///    indicate broken font encoding, not formula content. >30% is rejected.
-fn is_formula_garbage(text: &str) -> bool {
-    let mut total = 0usize;
-    let mut pua = 0usize;
-    let mut control = 0usize;
-    for ch in text.chars() {
-        if ch.is_whitespace() {
-            continue;
-        }
-        total += 1;
-        if ('\u{E000}'..='\u{F8FF}').contains(&ch) {
-            pua += 1;
-        }
-        let cp = ch as u32;
-        if cp < 0x20 {
-            control += 1;
-        }
-    }
-    if total < 3 {
-        return false;
-    }
-    // >10% PUA — significant undecoded extensible delimiters
-    if pua * 10 > total {
-        return true;
-    }
-    // >30% control chars — broken encoding
-    if control * 10 > total * 3 {
-        return true;
-    }
-    false
-}
-
 /// Detect markdown tables with suspicious structure that suggest the heuristic
 /// missed/mangled rows or columns. Returns true when the caller should treat
 /// the result as `needs_ocr` and fall back to GPU OCR.
@@ -2248,58 +2070,4 @@ mod tests {
             "Valid Japanese text should not be flagged as garbage"
         );
     }
-
-    #[test]
-    fn test_is_formula_garbage_accepts_math_text() {
-        // Greek letters, math operators, variables — typical formula text
-        let formula = "Φ(ν) = ∫ ∞ dze −it r1 − iνr2 sinh z";
-        assert!(
-            !is_formula_garbage(formula),
-            "Valid formula text should not be flagged as garbage"
-        );
-
-        // Dense operator text
-        let operators = "α + β − γ × δ ÷ ε ≤ ζ ≥ η ≈ θ ≠ ι ± κ";
-        assert!(
-            !is_formula_garbage(operators),
-            "Math operator text should not be flagged as garbage"
-        );
-
-        // Short formula (e.g. single equation variable)
-        let short = "αβ";
-        assert!(
-            !is_formula_garbage(short),
-            "Short formula text should not be flagged"
-        );
-    }
-
-    #[test]
-    fn test_is_formula_garbage_rejects_pua_heavy() {
-        // Simulates extensible delimiters from CMEX fonts mapping to PUA
-        let pua_heavy = "x \u{F8EB} \u{F8EC} \u{F8ED} \u{F8F6} \u{F8F7} \u{F8F8} y";
-        assert!(
-            is_formula_garbage(pua_heavy),
-            "PUA-heavy text should be flagged as formula garbage"
-        );
-    }
-
-    #[test]
-    fn test_is_formula_garbage_rejects_control_chars() {
-        // Control characters indicate broken encoding
-        let control_heavy = "a\x01b\x02c\x03d\x04e\x05f\x06g\x07h\x08i";
-        assert!(
-            is_formula_garbage(control_heavy),
-            "Control-char-heavy text should be flagged as formula garbage"
-        );
-    }
-
-    #[test]
-    fn test_is_formula_garbage_accepts_few_pua() {
-        // A few PUA chars among many valid chars is fine (<10% threshold)
-        let mostly_good = "Φ(ν) = ∫ dze r1 − iνr2 sinh z α β γ δ ε ζ η θ \u{F8EB}";
-        assert!(
-            !is_formula_garbage(mostly_good),
-            "Mostly-good text with rare PUA should pass"
-        );
-    }
 }

src/tables/detect_heuristic.rs

@@ -653,15 +653,20 @@ fn detect_table_in_region(items: &[(usize, &TextItem)], mode: TableDetectionMode
         return None;
     }
 
-    // Validation 8: Check for Table of Contents pattern
-    if is_table_of_contents(&cells) {
-        log::debug!("  validation 8 fail: table of contents");
+    // Validation 8: Reject paragraph-like content falsely detected as tables
+    if is_paragraph_content(&cells) {
+        log::debug!("  validation 9 fail: paragraph content");
         return None;
     }
 
-    // Validation 9: Reject paragraph-like content falsely detected as tables
-    if is_paragraph_content(&cells) {
-        log::debug!("  validation 9 fail: paragraph content");
+    // Validation 9: Reject wide "index" layouts where every cell carries a
+    // full "label ... page" fragment (back-of-book IRS-style indices).
+    // These render poorly in any structured form; text flow is the best
+    // fallback.  Narrow dot-leader TOCs (2-3 cols) are kept so format.rs
+    // can emit them as a per-row flat list with titles tab-joined to page
+    // numbers.
+    if is_inline_leader_index(&cells) {
+        log::debug!("  validation 9 fail: inline-leader index");
         return None;
     }
 
@@ -898,77 +903,247 @@ fn looks_like_number(s: &str) -> bool {
         && s.chars().any(|c| c.is_ascii_digit())
 }
 
-/// Check if this looks like a Table of Contents
-/// TOCs have characteristic patterns: leader dots, page numbers, section names
-fn is_table_of_contents(cells: &[Vec<String>]) -> bool {
+/// Check if this looks like a Table of Contents (either style).
+///
+/// Used by format.rs to render TOCs as flat lists instead of markdown tables.
+pub(super) fn is_table_of_contents(cells: &[Vec<String>]) -> bool {
+    is_dot_leader_toc(cells) || is_tabular_toc(cells)
+}
+
+/// Dot-leader TOC: any "Chapter 1 ........ 42" style with explicit leader
+/// dots.  Covers both narrow 2-3 col TOCs (where the leader is a dedicated
+/// cell) and wide indices (where each cell encodes a full "label ... page"
+/// fragment).  Used by format.rs to render as a flat list.
+pub(super) fn is_dot_leader_toc(cells: &[Vec<String>]) -> bool {
+    has_structural_dot_leader(cells) || is_inline_leader_index(cells)
+}
+
+/// Rows with a dedicated dots-only cell flanked by label + number (2-3 col
+/// TOC layout).  Format.rs handles these well via per-row flat-list
+/// rendering; they should NOT be rejected at detect time.
+fn has_structural_dot_leader(cells: &[Vec<String>]) -> bool {
     if cells.is_empty() {
         return false;
     }
+    let structural_rows = cells.iter().filter(|row| row_has_dot_leader(row)).count();
+    structural_rows as f32 / cells.len() as f32 >= 0.3
+}
 
-    let num_cols = cells[0].len();
-    let mut dot_cells = 0;
-    let mut page_number_cells = 0;
-    let mut total_cells = 0;
-    // Track which columns contain dots vs numbers to distinguish
-    // TOC (dots span middle, page number at end) from data tables
-    // (dots only in label column, many number columns).
-    let mut dot_cols = vec![0u32; num_cols];
-    let mut numeric_cols = vec![0u32; num_cols];
-
+/// Wide index layout: each cell holds a full "label ... page" fragment
+/// because the column detector kept multi-column indices as single cells.
+/// These render poorly both as markdown tables (column boundaries are
+/// arbitrary) and as flat lists (each row holds 3+ separate index
+/// entries).  Reject these at detect time so they fall back to the page's
+/// normal text flow.
+pub(super) fn is_inline_leader_index(cells: &[Vec<String>]) -> bool {
+    let mut inline_cells = 0;
+    let mut total_nonempty = 0;
     for row in cells {
-        for (ci, cell) in row.iter().enumerate() {
+        for cell in row {
             let trimmed = cell.trim();
             if trimmed.is_empty() {
                 continue;
             }
-            total_cells += 1;
-
-            // Check for leader dots (sequences of periods)
-            // TOCs often have "........" or ". . . ." patterns
-            let dot_count = trimmed.chars().filter(|&c| c == '.').count();
-            let is_mostly_dots = dot_count > trimmed.len() / 2 && dot_count >= 3;
-            if is_mostly_dots {
-                dot_cells += 1;
-                if ci < num_cols {
-                    dot_cols[ci] += 1;
-                }
-            }
-
-            // Check for standalone page numbers (1-4 digits, possibly with spaces)
-            let digits_only: String = trimmed.chars().filter(|c| !c.is_whitespace()).collect();
-            if digits_only.len() <= 4
-                && !digits_only.is_empty()
-                && digits_only.chars().all(|c| c.is_ascii_digit())
-            {
-                page_number_cells += 1;
-                if ci < num_cols {
-                    numeric_cols[ci] += 1;
-                }
+            total_nonempty += 1;
+            if cell_is_inline_leader(trimmed) {
+                inline_cells += 1;
             }
         }
     }
+    total_nonempty >= 4 && inline_cells as f32 / total_nonempty as f32 >= 0.25
+}
 
-    if total_cells == 0 {
+/// A row with a dot-leader.  Accepts two layouts:
+///   1. A dedicated dots-only cell ("....") with a text label somewhere
+///      to its left and a page number somewhere to its right.
+///   2. A "title ... " cell (trailing leader dots glued to the title)
+///      with a page number elsewhere in the same row.
+fn row_has_dot_leader(row: &[String]) -> bool {
+    let has_page_number = row.iter().any(|c| row_cell_is_page_number(c));
+
+    for (ci, cell) in row.iter().enumerate() {
+        let trimmed = cell.trim();
+
+        // Pattern 1: dedicated dots-only cell.
+        let dot_count = trimmed.chars().filter(|&c| c == '.').count();
+        let is_mostly_dots = dot_count >= 3
+            && dot_count > trimmed.len() / 2
+            && trimmed.chars().all(|c| c == '.' || c.is_whitespace());
+        if is_mostly_dots {
+            let has_label_left = row[..ci].iter().any(|c| {
+                let t = c.trim();
+                !t.is_empty() && t.chars().any(|ch| ch.is_alphabetic())
+            });
+            if has_label_left && has_page_number {
+                return true;
+            }
+            continue;
+        }
+
+        // Pattern 2: cell ends with a trailing " ... " run after a label.
+        if has_page_number && cell_has_trailing_leader(trimmed) {
+            return true;
+        }
+    }
+    false
+}
+
+/// Cell ends with a run of ≥3 dots preceded by alphabetic text and a
+/// space — the "Title ... " layout where the leader is glued to the name.
+/// Alphabetic (not alphanumeric) so that data-table row labels like
+/// "1973 ... " do not register as titles.
+fn cell_has_trailing_leader(cell: &str) -> bool {
+    let trimmed = cell.trim_end();
+    if !trimmed.ends_with('.') {
+        return false;
+    }
+    let without_dots = trimmed.trim_end_matches('.');
+    let dot_run = trimmed.len() - without_dots.len();
+    if dot_run < 3 {
+        return false;
+    }
+    // Require a space before the dot run (rules out "etc..." / "Mr...") and
+    // at least one alphabetic char (rules out "1973 ... " data-row labels).
+    without_dots.ends_with(' ') && without_dots.trim().chars().any(|c| c.is_alphabetic())
+}
+
+/// Page-number shape: single ≤4-digit integer, a ", "-separated list of
+/// ≤4-digit integers ("18, 36, 107"), or a dashed section-page ID
+/// ("A-1", "5-21").  Rejects decimal cells ("4. 0"), thousands-separated
+/// values ("189,164"), and other long numeric data that appears in
+/// statistical tables.
+fn row_cell_is_page_number(cell: &str) -> bool {
+    let t = cell.trim();
+    if t.is_empty() {
+        return false;
+    }
+    if looks_like_section_page_id(t) {
+        return true;
+    }
+    // Page list: ", " separator (with space) distinguishes real page lists
+    // from thousands-separated numbers like "189,164".
+    let parts: Vec<&str> = t.split(", ").collect();
+    parts
+        .iter()
+        .all(|p| !p.is_empty() && p.len() <= 4 && p.chars().all(|c| c.is_ascii_digit()))
+}
+
+/// A cell shaped like an index leader fragment.  Accepts two forms:
+///   - "text ... number" — label + dots + page number in one cell
+///   - "... number"      — bare leader + number (row where the label
+///     landed in a separate column)
+///
+/// Both only count if followed by pure numeric content (optionally
+/// comma-separated page lists like "127, 213").
+fn cell_is_inline_leader(cell: &str) -> bool {
+    let cell = cell.trim();
+
+    // Find the first "..." run.  Surrounding-whitespace checks below
+    // reject intra-word ellipses ("etc...").
+    let idx = match cell.match_indices("...").next() {
+        Some((i, _)) => i,
+        None => return false,
+    };
+
+    let before = &cell[..idx];
+    let after_dots = &cell[idx + 3..];
+    // Allow extra dots (e.g. "....") by skipping any additional '.'
+    let after = after_dots.trim_start_matches('.');
+
+    // Require space (or start-of-cell) before the dots and space/digit
+    // after — blocks intra-word ellipses.
+    let before_ok = before.is_empty() || before.ends_with(' ');
+    let after_ok = after.starts_with(' ') || after.is_empty();
+    if !before_ok || !after_ok {
         return false;
     }
 
-    // Data tables with dot leaders (e.g. "1973....") have dots concentrated
-    // in one column (the label column) while many other columns contain numbers.
-    // True TOCs have dots spanning the middle and one page-number column at the end.
-    // If dots are confined to ≤1 column AND there are ≥3 columns with numbers,
-    // this is a data table, not a TOC.
-    let cols_with_dots = dot_cols.iter().filter(|&&c| c >= 2).count();
-    let cols_with_numbers = numeric_cols.iter().filter(|&&c| c >= 2).count();
-    if cols_with_dots <= 1 && cols_with_numbers >= 3 {
+    let after_trim = after.trim();
+    if after_trim.is_empty() {
+        return false;
+    }
+    // Tail must be purely numeric/page-list content.
+    let tail_numeric = after_trim
+        .chars()
+        .all(|c| c.is_ascii_digit() || matches!(c, ',' | ' ' | '.' | '-' | '$'))
+        && after_trim.chars().any(|c| c.is_ascii_digit());
+    if !tail_numeric {
         return false;
     }
 
-    // If a significant portion of cells are dots or page numbers, it's likely a TOC
-    let dot_ratio = dot_cells as f32 / total_cells as f32;
-    let page_num_ratio = page_number_cells as f32 / total_cells as f32;
+    // Either we have a label before, or the leader is bare (starts the cell)
+    // — both are legitimate index fragments.
+    before.chars().any(|c| c.is_alphabetic()) || before.trim().is_empty()
+}
 
-    // TOC typically has >15% dot cells and >10% page number cells
-    dot_ratio > 0.15 || (dot_ratio > 0.05 && page_num_ratio > 0.15)
+/// Dot-less tabular TOC: tagged PDFs emit entries as rows where the first
+/// column starts with a dotted section number (e.g. "4.3.1 Something") and
+/// the last column is one or more page numbers.  These have no leader dots
+/// and benefit from flat-list formatting (page numbers aligned to titles).
+pub(super) fn is_tabular_toc(cells: &[Vec<String>]) -> bool {
+    if cells.is_empty() {
+        return false;
+    }
+    let num_cols = cells[0].len();
+    if num_cols < 2 || cells.len() < 4 {
+        return false;
+    }
+
+    let section_rows = cells
+        .iter()
+        .filter(|row| {
+            row.iter()
+                .find(|c| !c.trim().is_empty())
+                .is_some_and(|c| starts_with_section_number(c.trim()))
+        })
+        .count();
+
+    let last_col = num_cols - 1;
+    let (last_filled, last_page_num) = cells.iter().fold((0u32, 0u32), |(f, n), row| {
+        let cell = row.get(last_col).map(|s| s.trim()).unwrap_or("");
+        if cell.is_empty() {
+            return (f, n);
+        }
+        let is_page_nums = cell
+            .split_whitespace()
+            .all(|tok| !tok.is_empty() && tok.chars().all(|c| c.is_ascii_digit()));
+        (f + 1, n + if is_page_nums { 1 } else { 0 })
+    });
+
+    let section_ratio = section_rows as f32 / cells.len() as f32;
+    let page_num_last_ratio = if last_filled > 0 {
+        last_page_num as f32 / last_filled as f32
+    } else {
+        0.0
+    };
+
+    section_ratio >= 0.6 && last_filled >= 3 && page_num_last_ratio >= 0.7
+}
+
+/// Matches dashed section-page identifiers used in technical manuals:
+/// "5-21", "A-1", "B--3", "TC-2".  At least one ASCII digit is required.
+fn looks_like_section_page_id(s: &str) -> bool {
+    let ok = s
+        .chars()
+        .all(|c| c.is_ascii_digit() || c.is_ascii_uppercase() || c == '-');
+    ok && s.chars().any(|c| c.is_ascii_digit())
+}
+
+/// Returns true when the leading token looks like a dotted section number:
+/// "1", "1.2", "1.2.3", "4.3.1.2" — integer components joined by dots,
+/// with at least one dot (single-number prefixes are too ambiguous).
+fn starts_with_section_number(s: &str) -> bool {
+    let Some(first) = s.split_whitespace().next() else {
+        return false;
+    };
+    let first = first.trim_end_matches('.');
+    let parts: Vec<&str> = first.split('.').collect();
+    if parts.len() < 2 || parts.len() > 6 {
+        return false;
+    }
+    parts
+        .iter()
+        .all(|p| !p.is_empty() && p.len() <= 3 && p.chars().all(|c| c.is_ascii_digit()))
 }
 
 /// Check if detected "table" cells are actually paragraph text fragments.
@@ -1425,4 +1600,227 @@ mod tests {
             "data table with dot-leader labels should not be rejected as TOC"
         );
     }
+
+    #[test]
+    fn is_table_of_contents_rejects_dotless_toc() {
+        // Tabular TOC without leader dots: first column starts with dotted
+        // section numbers, last column is page numbers.  Pattern from
+        // Mythos system card pages 6-8.
+        let cells = vec![
+            vec![
+                "4.3 Case studies and targeted evaluations".to_string(),
+                String::new(),
+                "86".to_string(),
+            ],
+            vec![
+                "4.3.1 Destructive or reckless actions".to_string(),
+                "4.3.1.1 Synthetic-backend evaluation".to_string(),
+                "86 86".to_string(),
+            ],
+            vec![
+                "4.3.2 Adherence to constitution".to_string(),
+                "4.3.2.1 Overview".to_string(),
+                "89 89".to_string(),
+            ],
+            vec![
+                "4.3.3 Honesty and hallucinations".to_string(),
+                "4.3.3.1 Factual hallucinations".to_string(),
+                "93 94".to_string(),
+            ],
+            vec![
+                "4.4 Capability evaluations".to_string(),
+                String::new(),
+                "101".to_string(),
+            ],
+        ];
+        assert!(
+            is_table_of_contents(&cells),
+            "dot-less TOC with section numbers + page numbers should be rejected"
+        );
+    }
+
+    #[test]
+    fn dot_leader_toc_accepts_short_inline_leaders() {
+        // Index-style cells where the full "label ... number" pattern is
+        // preserved in a single cell (IRS Publication 17 back-of-book index).
+        let cells = vec![
+            vec!["Child tax credit ... 235".to_string(), String::new()],
+            vec!["Church employee ... 252".to_string(), String::new()],
+            vec!["Citizens outside the U.S ... 6".to_string(), String::new()],
+            vec![
+                "Claim for refund ... 18, 36, 107".to_string(),
+                String::new(),
+            ],
+            vec!["Clergy ... 7, 52".to_string(), String::new()],
+        ];
+        assert!(is_dot_leader_toc(&cells));
+    }
+
+    #[test]
+    fn dot_leader_toc_allows_ellipsis_data_table() {
+        // Data tables using "..." as a row-omission marker must not be
+        // mistaken for dot-leader TOCs.  Based on MCF5235RM QSPI RAM layout.
+        let cells = vec![
+            vec![
+                "0x00".to_string(),
+                "QTR0".to_string(),
+                "Transmit RAM".to_string(),
+            ],
+            vec!["0x01".to_string(), "QTR1".to_string(), String::new()],
+            vec![
+                "...".to_string(),
+                "...".to_string(),
+                "16 bits wide".to_string(),
+            ],
+            vec!["0x0F".to_string(), "QTR15".to_string(), String::new()],
+            vec![
+                "0x10".to_string(),
+                "QRR0".to_string(),
+                "Receive RAM".to_string(),
+            ],
+            vec!["0x11".to_string(), "QRR1".to_string(), String::new()],
+            vec![
+                "...".to_string(),
+                "...".to_string(),
+                "16 bits wide".to_string(),
+            ],
+            vec!["0x1F".to_string(), "QRR15".to_string(), String::new()],
+        ];
+        assert!(
+            !is_dot_leader_toc(&cells),
+            "ellipsis markers in a data table should not match TOC detection"
+        );
+    }
+
+    #[test]
+    fn dot_leader_toc_rejects_year_row_data_table() {
+        // ERP-2025 economic data tables: year labels with trailing " ... ",
+        // a final " ... " column, and decimal-looking numeric cells.  The
+        // detection previously classified these as dot-leader TOCs and
+        // routed them through flat-list formatting, destroying the grid.
+        let cells = vec![
+            vec![
+                "1973 ... ".to_string(),
+                "4. 0".to_string(),
+                "1. 8".to_string(),
+                "0. 4".to_string(),
+                "3. 2".to_string(),
+                " ... ".to_string(),
+            ],
+            vec![
+                "1974 ... ".to_string(),
+                "–1. 9".to_string(),
+                "–1. 6".to_string(),
+                "–5. 6".to_string(),
+                "2. 4".to_string(),
+                " ... ".to_string(),
+            ],
+            vec![
+                "1975 ... ".to_string(),
+                "2. 6".to_string(),
+                "5. 1".to_string(),
+                "6. 1".to_string(),
+                "4. 1".to_string(),
+                " ... ".to_string(),
+            ],
+            vec![
+                "1976 ... ".to_string(),
+                "4. 3".to_string(),
+                "5. 4".to_string(),
+                "6. 4".to_string(),
+                "4. 5".to_string(),
+                " ... ".to_string(),
+            ],
+        ];
+        assert!(
+            !is_dot_leader_toc(&cells),
+            "year-indexed data tables with decimal cells must not match TOC detection"
+        );
+    }
+
+    #[test]
+    fn dot_leader_toc_rejects_monthly_data_table() {
+        // ERP-2025 Table B-22: monthly labor-force rows with "Jan ... ",
+        // "Feb ... " labels and thousands-separated cells ("189,164").
+        // Previously matched TOC detection because "Jan ..." has alphabetic
+        // text and "189,164" passed the page-number shape check.
+        let cells = vec![
+            vec![
+                "2023: Jan ... ".to_string(),
+                "265,962".to_string(),
+                "165,871".to_string(),
+                "160,152".to_string(),
+                "62. 4".to_string(),
+            ],
+            vec![
+                "Feb ... ".to_string(),
+                "266,112".to_string(),
+                "166,263".to_string(),
+                "160,301".to_string(),
+                "62. 5".to_string(),
+            ],
+            vec![
+                "Mar ... ".to_string(),
+                "266,272".to_string(),
+                "166,690".to_string(),
+                "160,824".to_string(),
+                "62. 6".to_string(),
+            ],
+            vec![
+                "Apr ... ".to_string(),
+                "266,443".to_string(),
+                "166,678".to_string(),
+                "160,962".to_string(),
+                "62. 6".to_string(),
+            ],
+        ];
+        assert!(
+            !is_dot_leader_toc(&cells),
+            "monthly labor-force rows with thousands-separated data must not match TOC detection"
+        );
+    }
+
+    #[test]
+    fn tabular_toc_requires_section_numbers_and_pages() {
+        // Dot-less tabular TOC matches is_tabular_toc but not dot-leader.
+        let cells = vec![
+            vec![
+                "4.3 Case studies".to_string(),
+                String::new(),
+                "86".to_string(),
+            ],
+            vec![
+                "4.3.1 Destructive actions".to_string(),
+                String::new(),
+                "86".to_string(),
+            ],
+            vec![
+                "4.3.2 Adherence".to_string(),
+                String::new(),
+                "89".to_string(),
+            ],
+            vec!["4.3.3 Honesty".to_string(), String::new(), "93".to_string()],
+        ];
+        assert!(is_tabular_toc(&cells));
+        assert!(!is_dot_leader_toc(&cells));
+    }
+
+    #[test]
+    fn starts_with_section_number_matches_dotted() {
+        assert!(starts_with_section_number("1.2"));
+        assert!(starts_with_section_number("4.3.1"));
+        assert!(starts_with_section_number("4.3.1.2"));
+        assert!(starts_with_section_number("4.3 Case studies"));
+        assert!(starts_with_section_number("2.2.5.1 Expert red teaming"));
+    }
+
+    #[test]
+    fn starts_with_section_number_rejects_non_sections() {
+        assert!(!starts_with_section_number("Chapter 1"));
+        assert!(!starts_with_section_number("1973"));
+        assert!(!starts_with_section_number("1.5M"));
+        assert!(!starts_with_section_number("10.0%"));
+        assert!(!starts_with_section_number(""));
+        assert!(!starts_with_section_number("Hello world"));
+    }
 }

src/tables/format.rs

@@ -1,5 +1,6 @@
 //! Table-to-markdown formatting and cell cleanup.
 
+use super::detect_heuristic::is_table_of_contents;
 use super::Table;
 
 pub fn table_to_markdown(table: &Table) -> String {
@@ -7,6 +8,22 @@ pub fn table_to_markdown(table: &Table) -> String {
         return String::new();
     }
 
+    // Detect TOC on the raw cells: clean_table_cells merges rows in ways
+    // that can make genuine data tables superficially resemble a TOC
+    // (short numeric cells, few columns) — but the raw detection here
+    // preserves the original multi-column structure and only matches the
+    // true TOC pattern.
+    //
+    // Tables of contents render poorly as markdown tables — emit a flat
+    // per-row text list instead so the page numbers stay aligned with
+    // their section titles rather than drifting to a separate column.
+    // Format from raw cells: continuation-row merging collapses separate
+    // TOC entries (e.g. "6.2 Contamination" + "6.2.1 SWE-bench") into a
+    // single line because sub-entries leave column 0 empty.
+    if is_table_of_contents(&table.cells) {
+        return format_toc_as_list(&table.cells, &[]);
+    }
+
     // Clean up the table: merge continuation rows, extract footnotes, remove empty rows
     let (cleaned_cells, footnotes) = clean_table_cells(&table.cells);
 
@@ -49,6 +66,101 @@ pub fn table_to_markdown(table: &Table) -> String {
     output
 }
 
+/// Render a table-of-contents as a flat per-row text block.
+///
+/// Each row becomes one line: non-empty cells joined with spaces, and the
+/// last cell (typically a page number) is separated by a tab so the page
+/// numbers stay aligned with their titles instead of being pulled into a
+/// separate column by the column-aware reader.
+fn format_toc_as_list(cells: &[Vec<String>], footnotes: &[String]) -> String {
+    let mut output = String::new();
+
+    for row in cells {
+        let trimmed: Vec<&str> = row.iter().map(|c| c.trim()).collect();
+        let last_idx = trimmed.iter().rposition(|c| !c.is_empty());
+        let Some(last_idx) = last_idx else {
+            continue;
+        };
+
+        let last_cell = trimmed[last_idx];
+        let last_is_page = is_page_number_cell(last_cell);
+
+        let (title_cells, trailing) = if last_is_page && last_idx > 0 {
+            (&trimmed[..last_idx], Some(last_cell))
+        } else {
+            (&trimmed[..=last_idx], None)
+        };
+
+        // Skip dots-only cells when joining the title — in a detected TOC
+        // layout, a "...." cell is a leader separator, not part of the
+        // entry name.
+        let title = title_cells
+            .iter()
+            .filter(|c| !c.is_empty() && !is_dots_only(c))
+            .copied()
+            .collect::<Vec<_>>()
+            .join(" ");
+
+        if title.is_empty() && trailing.is_none() {
+            continue;
+        }
+
+        if !title.is_empty() {
+            output.push_str(&title);
+        }
+        if let Some(page) = trailing {
+            if !title.is_empty() {
+                output.push('\t');
+            }
+            output.push_str(page);
+        }
+        output.push('\n');
+    }
+
+    if !footnotes.is_empty() {
+        output.push('\n');
+        for footnote in footnotes {
+            output.push_str(footnote);
+            output.push('\n');
+        }
+    }
+
+    output
+}
+
+/// True when the cell looks like a page number.  Accepts:
+///   - plain digit tokens: "42", "86 86"
+///   - dashed section-page IDs: "5-21", "A-1", "B--3", "TC-2" (common in
+///     technical manuals)
+fn is_page_number_cell(cell: &str) -> bool {
+    let tokens: Vec<&str> = cell.split_whitespace().collect();
+    if tokens.is_empty() {
+        return false;
+    }
+    tokens.iter().all(|t| {
+        if t.is_empty() || t.len() > 8 {
+            return false;
+        }
+        let all_digits = t.chars().all(|c| c.is_ascii_digit());
+        if all_digits {
+            return t.len() <= 4;
+        }
+        // Section-page form: uppercase letters, digits, dashes; at least
+        // one digit present.
+        t.chars()
+            .all(|c| c.is_ascii_digit() || c.is_ascii_uppercase() || c == '-')
+            && t.chars().any(|c| c.is_ascii_digit())
+    })
+}
+
+/// True when the cell is purely leader dots (any length ≥ 3) with optional
+/// whitespace.
+fn is_dots_only(cell: &str) -> bool {
+    let t = cell.trim();
+    let dots = t.chars().filter(|&c| c == '.').count();
+    dots >= 3 && t.chars().all(|c| c == '.' || c.is_whitespace())
+}
+
 /// Clean up table cells: merge continuation rows, extract footnotes, remove empty rows
 fn clean_table_cells(cells: &[Vec<String>]) -> (Vec<Vec<String>>, Vec<String>) {
     let mut cleaned: Vec<Vec<String>> = Vec::new();
@@ -432,4 +544,42 @@ mod tests {
         };
         assert_eq!(table_to_markdown(&table), "");
     }
+
+    #[test]
+    fn test_table_to_markdown_toc_renders_as_flat_list() {
+        // A TOC-shaped table with section numbers in col 0 and page numbers
+        // in the last column should render as a flat list, not a markdown
+        // table, so the page numbers stay on the same line as their titles.
+        let table = Table {
+            columns: vec![50.0, 80.0, 300.0],
+            rows: vec![500.0; 5],
+            cells: vec![
+                vec![
+                    "4.3".into(),
+                    "Case studies and targeted evaluations".into(),
+                    "86".into(),
+                ],
+                vec![
+                    "4.3.1".into(),
+                    "Destructive or reckless actions".into(),
+                    "86".into(),
+                ],
+                vec![
+                    "4.3.2".into(),
+                    "Adherence to its constitution".into(),
+                    "89".into(),
+                ],
+                vec!["4.4".into(), "Capability evaluations".into(), "101".into()],
+                vec!["4.5".into(), "White-box analyses".into(), "113".into()],
+            ],
+            item_indices: vec![],
+        };
+        let md = table_to_markdown(&table);
+        assert!(
+            !md.contains("|---|"),
+            "TOC should not render as a markdown table: {md}"
+        );
+        assert!(md.contains("4.3 Case studies and targeted evaluations\t86"));
+        assert!(md.contains("4.5 White-box analyses\t113"));
+    }
 }