fix: don't reclassify wrapped bold list leads as headings

When a numbered/bulleted list item's bold lead phrase wraps onto a second visual line, that line is all_bold + standalone, which scored above the rarity heading threshold and was emitted as #### in the middle of the item. That reset in_list, so the body continuation below picked up a stray `- ` bullet via the struct-tree LI path, shattering a single item into heading + stray bullets. Guard the font heuristic: when already inside a list, skip heading classification for lines at the list continuation indent with a Y gap within para_threshold. Structure-tree headings still win. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
update eval instructions
2026-04-21 08:43:04 -07:00 · 2026-04-20 22:50:57 -07:00 · 2026-04-20 22:47:34 -07:00 · 2026-04-20 21:59:53 -07:00 · 2026-04-20 17:19:03 -07:00 · 2026-04-20 14:04:50 -07:00
31 changed files with 2124 additions and 2230 deletions
@@ -46,6 +46,8 @@ jobs:
            target: x86_64-unknown-linux-gnu
          - os: macos-14
            target: aarch64-apple-darwin
+          - os: windows-latest
+            target: x86_64-pc-windows-msvc
    steps:
      - uses: actions/checkout@v4

@@ -35,3 +35,8 @@ scripts/
 # Test output
 test_output/

+# Python
+__pycache__/
+*.pyc
+.pytest_cache/
+
@@ -0,0 +1,79 @@
+# pdf-inspector
+
+Fast PDF text extraction to structured Markdown. CLI binary: `pdf2md`. Detection binary: `detect-pdf`.
+
+## Build & Test
+
+```bash
+cargo fmt                                    # format
+cargo clippy -- -D warnings                  # lint (enforced, zero warnings)
+cargo test                                   # unit + integration tests (267+ unit, 73+ integration)
+cargo build --release                        # release binary for benchmarks
+```
+
+All three must pass before committing.
+
+## Binaries
+
+- `pdf2md` — extract PDF → Markdown. Supports `--json` for structured output.
+- `detect-pdf` — classify PDF type (TextBased/Scanned/Mixed/ImageBased). Supports `--analyze --json`.
+
+## Architecture
+
+```
+src/
+  lib.rs                        – public API, process_pdf_with_options, encoding issue detection
+  detector.rs                   – PDF type classification, tiled-scan detection, page sampling
+  types.rs                      – TextItem, TextLine, PdfRect, PdfLine
+  tounicode.rs                  – CMap/ToUnicode parsing, CID decoding
+  text_utils.rs                 – CJK/RTL handling, Otsu threshold, ligature expansion, NFKC
+  extractor/
+    mod.rs                      – top-level extraction orchestrator
+    content_stream.rs           – PDF operator state machine (Tj/TJ/Td/Tm/q/Q)
+    fonts.rs                    – font width/encoding, CMapDecisionCache, TrueType cmap fallback
+    layout.rs                   – column detection (histogram), newspaper/tabular classification,
+                                  spanning-line pre-masking, sidebar detection
+  tables/
+    detect_rects.rs             – rect-based table detection (union-find clustering)
+    detect_heuristic.rs         – heuristic table detection (gap-histogram, body-font tables)
+    detect_lines.rs             – line-based table detection (H/V line grids)
+    grid.rs                     – column/row boundaries, cell assignment
+    format.rs                   – table→Markdown formatting, continuation row merging
+  markdown/
+    convert.rs                  – core line→Markdown loop, struct-tree role support
+    analysis.rs                 – font stats, heading tiers, paragraph thresholds
+    classify.rs                 – line classification (header, list, code, caption)
+    preprocess.rs               – drop cap merging, heading line merging
+    postprocess.rs              – dot leaders, hyphenation, page numbers, URL formatting
+```
+
+## Key design decisions
+
+- **Primary audience is AI agents.** Output optimized for token efficiency and semantic quality, not visual formatting. No cosmetic padding.
+- **Three table detection strategies** run in priority order: rect-based → line-based → heuristic. First valid result wins.
+- **Column detection** uses horizontal projection histograms with valley detection. Multi-item spanning lines (titles, headers) are pre-masked using column-aware thresholds before column assignment.
+- **Newspaper vs tabular** classification determines reading order: newspaper reads columns sequentially, tabular Y-interleaves them.
+- **Tiled-scan detection** catches scanned PDFs with JBIG2/strip images where no single tile exceeds the template threshold but aggregate area does (≥2M pixels).
+- **Garbage text upgrade** reclassifies Mixed PDFs as Scanned when extracted text is <50% alphanumeric.
+- **Tagged PDF support** uses structure tree roles (H1-H6, P, L, Code, BlockQuote) when available, falling back to font-size heuristics.
+
+## Testing
+
+- **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.
+- **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.
+- **Regression suite**: sibling repo `pdf-evals` with 179+ snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing.
+
+## Debugging
+
+```bash
+RUST_LOG=pdf_inspector::extractor::layout=debug cargo run --bin pdf2md -- file.pdf
+RUST_LOG=pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf
+RUST_LOG=pdf_inspector::detector=debug cargo run --release --bin detect-pdf -- file.pdf
+```
+
+## Conventions
+
+- Clippy: use `is_some_and(...)` not `map_or(false, ...)`
+- lopdf quirk: `ParseError` is private — match by string for `InvalidFileHeader`
+- Column limit for tables: 25 (wide statistical tables)
+- `propagate_merged_cells` skipped for >10 columns (spanning rects = background fills)
@@ -61,7 +61,8 @@ src/

 - **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.
 - **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.
- **Regression suite**: sibling repo `pdf-evals` with 179+ snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing.
+- **Regression suite**: sibling repo `pdf-evals` with 187+ snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing.
+- **Semantic quality**: run `bench.py score` in `pdf-evals` for the semantic verdict (TEDS + MHS + reading order + char/word + list preservation, composited). Character-level diff alone misclassifies structural improvements (e.g., column-detection rewrites) as regressions — `score` is the tie-breaker. See `pdf-evals/CLAUDE.md` "Semantic scoring".

 ## Debugging

@@ -55,12 +55,12 @@ print(result.markdown)   # Markdown string or None
 ### Node.js

 ```bash
-npm install @firecrawl/pdf-inspector-js
+npm install @firecrawl/pdf-inspector
 ```

 ```javascript
 import { readFileSync } from 'fs';
-import { processPdf, classifyPdf } from '@firecrawl/pdf-inspector-js';
+import { processPdf, classifyPdf } from '@firecrawl/pdf-inspector';

 const result = processPdf(readFileSync('document.pdf'));
 console.log(result.pdfType);   // "TextBased", "Scanned", "ImageBased", "Mixed"
@@ -42,6 +42,14 @@ text = pdf_inspector.extract_text("document.pdf")
 items = pdf_inspector.extract_text_with_positions("document.pdf")
 for item in items[:5]:
    print(f"'{item.text}' at ({item.x:.0f}, {item.y:.0f}) size={item.font_size}")
+
+# Per-page markdown (one Markdown string per page, plus layout metadata)
+result = pdf_inspector.extract_pages_markdown("document.pdf")
+for page in result.pages:
+    print(f"Page {page.page}: {len(page.markdown)} chars, needs_ocr={page.needs_ocr}")
+
+# Restrict to specific 0-indexed pages (preserves caller order)
+result = pdf_inspector.extract_pages_markdown("document.pdf", pages=[0, 2])
 ```

 ## API reference
@@ -60,6 +68,8 @@ for item in items[:5]:
 | `extract_text_with_positions_bytes(data, pages=None)` | Text with positions from bytes |
 | `extract_text_in_regions(path, page_regions)` | Extract text in bounding-box regions |
 | `extract_text_in_regions_bytes(data, page_regions)` | Region extraction from bytes |
+| `extract_pages_markdown(path, pages=None)` | Per-page Markdown + layout metadata (all pages by default) |
+| `extract_pages_markdown_bytes(data, pages=None)` | Per-page Markdown from bytes |

 ## Types

@@ -72,3 +82,7 @@ for item in items[:5]:
 **`RegionText` fields:** `text`, `needs_ocr`

 **`PageRegionTexts` fields:** `page` (0-indexed), `regions` (list of RegionText)
+
+**`PageMarkdown` fields:** `page` (0-indexed), `markdown`, `needs_ocr`
+
+**`PagesExtractionResult` fields:** `pages` (list of PageMarkdown), `pages_with_tables` (1-indexed), `pages_with_columns` (1-indexed), `pages_needing_ocr` (1-indexed), `is_complex`
@@ -79,6 +79,27 @@ let bytes = std::fs::read("document.pdf")?;
 let result = process_pdf_mem(&bytes)?;
 ```

+Extract per-page Markdown (one string per page, plus document-wide layout
+metadata):
+
+```rust
+use pdf_inspector::extract_pages_markdown;
+
+// Pass `None` for every page in document order, or a slice of 0-indexed
+// pages to restrict the output (caller-supplied order is preserved).
+let result = extract_pages_markdown("document.pdf", None)?;
+
+for page in &result.pages {
+    if page.needs_ocr {
+        // Route this page to OCR
+    } else {
+        println!("Page {}: {}", page.page, page.markdown);
+    }
+}
+
+println!("Complex layout? {}", result.is_complex);
+```
+
 ## Processing modes

 | Mode | What it does | Returns |
@@ -102,6 +123,8 @@ let result = process_pdf_mem(&bytes)?;
 | `to_markdown(text, options)` | Convert plain text to Markdown |
 | `to_markdown_from_items(items, options)` | Markdown from pre-extracted `TextItem`s |
 | `to_markdown_from_items_with_rects(items, options, rects)` | Markdown with rectangle-based table detection |
+| `extract_pages_markdown(path, pages)` | Per-page Markdown + layout metadata (file) |
+| `extract_pages_markdown_mem(bytes, pages)` | Per-page Markdown from bytes |

 Low-level detection functions are also available via the `detector` module (`detect_pdf_type`, `detect_pdf_type_with_config`, etc.) for callers who need `PdfTypeResult` instead of `PdfProcessResult`.

@@ -119,4 +142,6 @@ Low-level detection functions are also available via the `detector` module (`det
 | `LayoutComplexity` | Layout analysis: is_complex, pages_with_tables, pages_with_columns |
 | `TextItem` | Text with position, font info, and page number |
 | `MarkdownOptions` | Configuration for Markdown formatting (page numbers, etc.) |
+| `PageMarkdown` | Per-page result: page (0-indexed), markdown, needs_ocr |
+| `PagesExtractionResult` | Per-page output + 1-indexed pages_with_tables / pages_with_columns / pages_needing_ocr, is_complex |
 | `PdfError` | `Io`, `Parse`, `Encrypted`, `InvalidStructure`, `NotAPdf` |
@@ -1,4 +1,4 @@
-# firecrawl-pdf-inspector
+# PDF Inspector

 Fast PDF classification and region-based text extraction for Node.js/Bun. Native Rust performance via [napi-rs](https://napi.rs).

@@ -7,9 +7,9 @@ Built by [Firecrawl](https://firecrawl.dev) for hybrid OCR pipelines — extract
 ## Install

 ```bash
-npm install firecrawl-pdf-inspector
+npm install @firecrawl/pdf-inspector
 # or
-bun add firecrawl-pdf-inspector
+bun add @firecrawl/pdf-inspector
 ```

 Prebuilt binaries included for **linux-x64** and **macOS ARM64**. No Rust toolchain needed.
@@ -21,7 +21,7 @@ Prebuilt binaries included for **linux-x64** and **macOS ARM64**. No Rust toolch
 Classify a PDF as TextBased, Scanned, Mixed, or ImageBased (~10-50ms). Returns which pages need OCR.

 ```typescript
-import { classifyPdf } from 'firecrawl-pdf-inspector'
+import { classifyPdf } from '@firecrawl/pdf-inspector'
 import { readFileSync } from 'fs'

 const pdf = readFileSync('document.pdf')
@@ -40,7 +40,7 @@ Extract text within bounding-box regions from a PDF. Designed for hybrid OCR pip
 Each region result includes a `needsOcr` flag that signals unreliable extraction (empty text, GID-encoded fonts, garbage text, encoding issues).

 ```typescript
-import { extractTextInRegions } from 'firecrawl-pdf-inspector'
+import { extractTextInRegions } from '@firecrawl/pdf-inspector'

 const result = extractTextInRegions(pdf, [
  {
@@ -0,0 +1,131 @@
+#!/usr/bin/env node
+
+import { readFileSync, writeFileSync } from "fs";
+import { createRequire } from "module";
+
+const require = createRequire(import.meta.url);
+const { version } = require("../package.json");
+
+const HELP = `pdf-inspector v${version} — Fast PDF text extraction to Markdown
+
+Usage:
+  pdf-inspector <file>                  Extract markdown (default)
+  pdf-inspector detect <file>           Classify PDF type
+
+Options:
+  --json                  Output as JSON
+  --pages <pages>         Comma-separated page numbers (e.g. 1,3,5)
+  -o, --output <file>     Write output to file instead of stdout
+  -h, --help              Show this help
+  -v, --version           Show version
+
+Examples:
+  pdf-inspector document.pdf
+  pdf-inspector document.pdf --json
+  pdf-inspector document.pdf --pages 1,2,3
+  pdf-inspector detect document.pdf --json
+  cat document.pdf | pdf-inspector -`;
+
+function die(msg) {
+  process.stderr.write(`error: ${msg}\n`);
+  process.exit(1);
+}
+
+function parseArgs(argv) {
+  const opts = { json: false, pages: null, output: null, file: null, command: "extract" };
+  let i = 0;
+
+  // Check for subcommand
+  if (argv[0] === "detect") {
+    opts.command = "detect";
+    i = 1;
+  }
+
+  while (i < argv.length) {
+    const arg = argv[i];
+    if (arg === "-h" || arg === "--help") {
+      process.stdout.write(HELP + "\n");
+      process.exit(0);
+    } else if (arg === "-v" || arg === "--version") {
+      process.stdout.write(`${version}\n`);
+      process.exit(0);
+    } else if (arg === "--json") {
+      opts.json = true;
+    } else if (arg === "--pages") {
+      i++;
+      if (!argv[i]) die("--pages requires a value (e.g. 1,3,5)");
+      opts.pages = argv[i].split(",").map((p) => {
+        const n = parseInt(p.trim(), 10);
+        if (Number.isNaN(n) || n < 1) die(`invalid page number: ${p}`);
+        return n;
+      });
+    } else if (arg === "-o" || arg === "--output") {
+      i++;
+      if (!argv[i]) die("-o requires a filename");
+      opts.output = argv[i];
+    } else if (arg === "-" || !arg.startsWith("-")) {
+      if (opts.file) die(`unexpected argument: ${arg}`);
+      opts.file = arg;
+    } else {
+      die(`unknown option: ${arg}`);
+    }
+    i++;
+  }
+
+  return opts;
+}
+
+function readInput(file) {
+  if (file === "-") {
+    return readFileSync(0); // stdin fd
+  }
+  try {
+    return readFileSync(file);
+  } catch (err) {
+    if (err.code === "ENOENT") die(`file not found: ${file}`);
+    die(err.message);
+  }
+}
+
+function output(text, outputPath) {
+  if (outputPath) {
+    writeFileSync(outputPath, text);
+  } else {
+    process.stdout.write(text);
+  }
+}
+
+// ---- main ----
+
+const opts = parseArgs(process.argv.slice(2));
+
+if (!opts.file) {
+  // Check if stdin is piped
+  if (process.stdin.isTTY !== false) {
+    process.stderr.write(HELP + "\n");
+    process.exit(1);
+  }
+  opts.file = "-";
+}
+
+const { processPdf, classifyPdf } = await import("../index.js");
+const buffer = readInput(opts.file);
+
+if (opts.command === "detect") {
+  const result = classifyPdf(buffer);
+  if (opts.json) {
+    output(JSON.stringify(result, null, 2) + "\n", opts.output);
+  } else {
+    const ocr = result.pagesNeedingOcr.length > 0
+      ? `, ${result.pagesNeedingOcr.length} pages need OCR`
+      : "";
+    output(`${result.pdfType} (${result.pageCount} pages, confidence: ${result.confidence.toFixed(2)}${ocr})\n`, opts.output);
+  }
+} else {
+  const result = processPdf(buffer, opts.pages ?? undefined);
+  if (opts.json) {
+    output(JSON.stringify(result, null, 2) + "\n", opts.output);
+  } else {
+    output((result.markdown ?? "") + "\n", opts.output);
+  }
+}
@@ -1,9 +1,12 @@
 {
-  "name": "firecrawl-pdf-inspector",
-  "version": "0.7.5",
+  "name": "@firecrawl/pdf-inspector",
+  "version": "1.4.0",
  "description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
  "main": "index.js",
  "types": "index.d.ts",
+  "bin": {
+    "pdf-inspector": "bin/pdf-inspector.mjs"
+  },
  "license": "MIT",
  "keywords": [
    "pdf",
@@ -20,6 +23,7 @@
    "index.js",
    "index.d.ts",
    "*.node",
+    "bin/",
    "README.md"
  ],
  "repository": {
@@ -34,7 +38,8 @@
    "binaryName": "pdf-inspector",
    "targets": [
      "x86_64-unknown-linux-gnu",
-      "aarch64-apple-darwin"
+      "aarch64-apple-darwin",
+      "x86_64-pc-windows-msvc"
    ],
    "package": {
      "name": "@firecrawl/pdf-inspector-js"
@@ -99,26 +99,6 @@ pub struct PageRegionTexts {
    pub regions: Vec<RegionText>,
 }

-/// LaTeX reconstruction result for a single formula region.
-#[napi(object)]
-pub struct FormulaLatexResult {
-    /// Reconstructed LaTeX string.
-    pub latex: String,
-    /// The linearized raw text (before LaTeX reconstruction).
-    pub raw_text: String,
-    /// Heuristic confidence in the LaTeX output (0.0–1.0).
-    pub confidence: f64,
-    /// `true` when extraction failed entirely and GPU OCR is needed.
-    pub needs_ocr: bool,
-}
-
-/// LaTeX reconstruction results for one page's formula regions.
-#[napi(object)]
-pub struct PageFormulaLatexResults {
-    pub page: u32,
-    pub regions: Vec<FormulaLatexResult>,
-}
-
 // ---------------------------------------------------------------------------
 // Helpers
 // ---------------------------------------------------------------------------
@@ -337,70 +317,6 @@ pub fn extract_tables_in_regions(
    })
 }

-/// Extract formula text within bounding-box regions from a PDF.
-///
-/// Like `extractTextInRegions` but uses formula-specific quality checks.
-/// Formula text is legitimately symbol-heavy (Greek letters, math operators)
-/// so the generic garbage-text check is relaxed. When the text decodes
-/// cleanly, `needsOcr` is `false` — the caller can skip GPU OCR.
-///
-/// Coordinates are PDF points with top-left origin.
-#[napi]
-pub fn extract_formulas_in_regions(
-    buffer: Buffer,
-    page_regions: Vec<PageRegions>,
-) -> Result<Vec<PageRegionTexts>> {
-    let bytes: Vec<u8> = buffer.to_vec();
-    let regions = parse_page_regions(&page_regions);
-
-    catch_panic("extract_formulas_in_regions", move || {
-        let results = pdf_inspector::extract_formulas_in_regions_mem(&bytes, &regions)
-            .map_err(|e| to_napi_err(e, "extract_formulas_in_regions"))?;
-        Ok(to_page_region_texts(results))
-    })
-}
-
-/// Extract formula text within bounding-box regions and reconstruct LaTeX.
-///
-/// Like `extractFormulasInRegions` but additionally reconstructs LaTeX from
-/// the positioned text items. Each result includes the raw text, reconstructed
-/// LaTeX, a confidence score, and the `needsOcr` flag.
-///
-/// The confidence score (0.0–1.0) indicates how reliable the heuristic LaTeX
-/// reconstruction is. The caller should use this to decide whether to trust
-/// the LaTeX or fall back to GPU OCR.
-///
-/// Coordinates are PDF points with top-left origin.
-#[napi]
-pub fn extract_formulas_in_regions_as_latex(
-    buffer: Buffer,
-    page_regions: Vec<PageRegions>,
-) -> Result<Vec<PageFormulaLatexResults>> {
-    let bytes: Vec<u8> = buffer.to_vec();
-    let regions = parse_page_regions(&page_regions);
-
-    catch_panic("extract_formulas_in_regions_as_latex", move || {
-        let results = pdf_inspector::extract_formulas_in_regions_as_latex(&bytes, &regions)
-            .map_err(|e| to_napi_err(e, "extract_formulas_in_regions_as_latex"))?;
-        Ok(results
-            .into_iter()
-            .map(|page_result| PageFormulaLatexResults {
-                page: page_result.page,
-                regions: page_result
-                    .regions
-                    .into_iter()
-                    .map(|r| FormulaLatexResult {
-                        latex: r.latex,
-                        raw_text: r.raw_text,
-                        confidence: r.confidence as f64,
-                        needs_ocr: r.needs_ocr,
-                    })
-                    .collect(),
-            })
-            .collect())
-    })
-}
-
 /// Per-page markdown extraction result.
 #[napi(object)]
 pub struct PageMarkdownResult {
@@ -427,21 +343,26 @@ pub struct PagesExtractionResult {
    pub is_complex: bool,
 }

-/// Extract formatted markdown for specific pages of a PDF, with layout
-/// classification metadata.
+/// Extract formatted markdown for pages of a PDF, with layout classification
+/// metadata.
 ///
 /// Returns per-page markdown and classification data (tables, columns,
 /// OCR needs) from a single parse. Font statistics are computed from the
 /// full document so header detection is consistent across pages.
+///
+/// Omit `pages` (or pass `undefined`) to return every page in document
+/// order. Pass an array of 0-indexed page numbers to restrict output to
+/// those pages, in caller-supplied order.
 #[napi]
 pub fn extract_pages_markdown(
    buffer: Buffer,
-    pages: Vec<u32>,
+    pages: Option<Vec<u32>>,
 ) -> Result<PagesExtractionResult> {
    let bytes: Vec<u8> = buffer.to_vec();
    catch_panic("extract_pages_markdown", move || {
-        let result = pdf_inspector::extract_pages_markdown_mem(&bytes, &pages)
-            .map_err(|e| to_napi_err(e, "extract_pages_markdown"))?;
+        let result =
+            pdf_inspector::extract_pages_markdown_mem(&bytes, pages.as_deref())
+                .map_err(|e| to_napi_err(e, "extract_pages_markdown"))?;
        Ok(PagesExtractionResult {
            pages: result
                .pages
@@ -7,6 +7,7 @@ import {
  extractText,
  extractTextWithPositions,
  extractTextInRegions,
+  extractPagesMarkdown,
 } from './index.js';

 const fixture = readFileSync('../tests/fixtures/thermo-freon12.pdf');
@@ -89,6 +90,28 @@ assert.equal(typeof regionResults[0].regions[0].text, 'string');
 assert.equal(typeof regionResults[0].regions[0].needsOcr, 'boolean');
 console.log('  extractTextInRegions: OK');

+// --- extractPagesMarkdown ---
+console.log('Testing extractPagesMarkdown...');
+
+// omit pages → every page in document order
+const allPages = extractPagesMarkdown(fixture);
+assert.equal(allPages.pages.length, 3);
+assert.deepEqual(allPages.pages.map(p => p.page), [0, 1, 2]);
+assert.ok(typeof allPages.pages[0].markdown === 'string');
+assert.equal(typeof allPages.pages[0].needsOcr, 'boolean');
+assert.ok(Array.isArray(allPages.pagesWithTables));
+assert.ok(Array.isArray(allPages.pagesWithColumns));
+assert.ok(Array.isArray(allPages.pagesNeedingOcr));
+assert.equal(typeof allPages.isComplex, 'boolean');
+console.log('  extractPagesMarkdown (no pages arg): OK');
+
+// selected pages preserve caller order
+const picked = extractPagesMarkdown(fixture, [2, 0]);
+assert.equal(picked.pages.length, 2);
+assert.equal(picked.pages[0].page, 2);
+assert.equal(picked.pages[1].page, 0);
+console.log('  extractPagesMarkdown with pages: OK');
+
 // --- Error handling ---
 console.log('Testing error handling...');
 assert.throws(() => processPdf(Buffer.from('not a pdf')), /process_pdf/);
@@ -52,6 +52,28 @@ class PageRegionTexts:
    """0-indexed page number."""
    regions: list[RegionText]

+class PageMarkdown:
+    """Per-page markdown extraction result."""
+    page: int
+    """0-indexed page number."""
+    markdown: str
+    """Formatted markdown for this page (empty string when needs_ocr is True)."""
+    needs_ocr: bool
+    """True when text on this page is unreliable and OCR should be used instead."""
+
+class PagesExtractionResult:
+    """Per-page markdown output with document-wide layout classification."""
+    pages: list[PageMarkdown]
+    """Per-page markdown results, in the order requested."""
+    pages_with_tables: list[int]
+    """1-indexed pages where tables were detected."""
+    pages_with_columns: list[int]
+    """1-indexed pages where multi-column layout was detected."""
+    pages_needing_ocr: list[int]
+    """1-indexed pages that need OCR."""
+    is_complex: bool
+    """True if any page has tables or multi-column layout."""
+
 def process_pdf(path: str, pages: Optional[list[int]] = None) -> PdfResult:
    """Process a PDF: detect type, extract text, convert to Markdown."""
    ...
@@ -115,3 +137,31 @@ def extract_text_in_regions_bytes(
        page_regions: List of (page_0indexed, [[x1, y1, x2, y2], ...]) tuples.
    """
    ...
+
+def extract_pages_markdown(
+    path: str,
+    pages: Optional[list[int]] = None,
+) -> PagesExtractionResult:
+    """Extract formatted markdown for pages of a PDF, with layout classification.
+
+    Args:
+        path: Path to the PDF file.
+        pages: Optional list of 0-indexed pages. When ``None`` (default), every
+            page is returned in document order. Otherwise, output matches the
+            caller-supplied order.
+
+    Returns:
+        PagesExtractionResult with per-page markdown and document-wide layout
+        classification (tables, columns, OCR needs).
+    """
+    ...
+
+def extract_pages_markdown_bytes(
+    data: bytes,
+    pages: Optional[list[int]] = None,
+) -> PagesExtractionResult:
+    """Extract formatted markdown for pages of a PDF from bytes.
+
+    See :func:`extract_pages_markdown` for details.
+    """
+    ...
@@ -216,6 +216,7 @@ pub(crate) fn extract_page_text_items(
    let mut marked_content_stack: Vec<MarkedContentEntry> = Vec::new();
    let mut suppress_glyph_extraction = false;
    let mut actual_text_start_tm: Option<[f32; 6]> = None; // text matrix at BDC entry
+    let mut actual_text_glyph_tm: Option<[f32; 6]> = None; // text matrix at first glyph inside BDC
    /// Get the innermost MCID from the marked content stack.
    fn current_mcid(stack: &[MarkedContentEntry]) -> Option<i64> {
        stack.iter().rev().find_map(|e| e.mcid)
@@ -349,8 +350,15 @@ pub(crate) fn extract_page_text_items(
                            )
                        })
                    });
-                    // ActualText: suppress glyph extraction, just advance text matrix
+                    // ActualText: suppress glyph extraction, just advance text matrix.
+                    // Capture the FIRST glyph's text matrix as the rendering position
+                    // for the ActualText item. Td ops between BDC and the first Tj
+                    // may have moved the position to the correct line — the BDC-entry
+                    // position (actual_text_start_tm) can be on the previous line.
                    if suppress_glyph_extraction {
+                        if actual_text_glyph_tm.is_none() {
+                            actual_text_glyph_tm = Some(text_matrix);
+                        }
                        if let Some(w_ts) = w_ts_opt {
                            text_matrix[4] += w_ts * text_matrix[0];
                            text_matrix[5] += w_ts * text_matrix[1];
@@ -425,6 +433,10 @@ pub(crate) fn extract_page_text_items(
                        let font_info = font_widths.get(&current_font);
                        let is_invisible = (text_rendering_mode == 3 && !include_invisible)
                            || suppress_glyph_extraction;
+                        // Capture first-glyph position for ActualText
+                        if suppress_glyph_extraction && actual_text_glyph_tm.is_none() {
+                            actual_text_glyph_tm = Some(text_matrix);
+                        }

                        // Compute space threshold based on font metrics when available
                        let space_threshold = if let Some(font_info) = font_info {
@@ -700,6 +712,7 @@ pub(crate) fn extract_page_text_items(
                if actual_text.is_some() {
                    suppress_glyph_extraction = true;
                    actual_text_start_tm = Some(text_matrix);
+                    actual_text_glyph_tm = None; // reset — will be captured at first Tj/TJ
                }
                marked_content_stack.push(MarkedContentEntry { actual_text, mcid });
            }
@@ -707,8 +720,13 @@ pub(crate) fn extract_page_text_items(
                // End Marked Content — emit ActualText item with correct width
                if let Some(entry) = marked_content_stack.pop() {
                    if let Some(at) = entry.actual_text {
-                        // Compute width from text matrix advancement during BDC..EMC
-                        if let Some(start_tm) = actual_text_start_tm.take() {
+                        // Use the first-glyph position (if available) instead of the
+                        // BDC-entry position. Td operators between BDC and the first
+                        // Tj may have moved the text position to the correct line —
+                        // the BDC-entry position can be on the previous line.
+                        let glyph_tm = actual_text_glyph_tm.take();
+                        let entry_tm = actual_text_start_tm.take();
+                        if let Some(start_tm) = glyph_tm.or(entry_tm) {
                            let combined = multiply_matrices(&start_tm, &ctm);
                            if combined[0].abs() >= combined[1].abs() {
                                rotation_votes.horizontal += 1;
@@ -626,6 +626,33 @@ fn find_relative_valleys(
    valleys
 }

+/// Detect whether a side of a gutter consists predominantly of list-marker
+/// glyphs (•, ●, ○, ◦, ▪, ▫, ◆, ◇). A column of bullets on the left margin
+/// creates a spurious histogram valley between the bullet and the content.
+/// Treating it as a real column splits each list item's text across two
+/// "columns," so we reject these candidates.
+fn is_list_marker_column(items: &[&&TextItem]) -> bool {
+    const LIST_MARKERS: &[char] = &['•', '●', '○', '◦', '▪', '▫', '◆', '◇', '■', '□'];
+    if items.is_empty() {
+        return false;
+    }
+    let marker_count = items
+        .iter()
+        .filter(|i| {
+            let t = i.text.trim();
+            let mut chars = t.chars();
+            match (chars.next(), chars.next()) {
+                (Some(c), None) => LIST_MARKERS.contains(&c),
+                _ => false,
+            }
+        })
+        .count();
+    // Require ≥80% of items on this side to be standalone markers. A handful
+    // of non-marker items (stray page numbers, footnote refs) shouldn't
+    // defeat the check.
+    marker_count as f32 / items.len() as f32 >= 0.8
+}
+
 /// Validate valley candidates with vertical consistency checks and build column regions.
 ///
 /// When `center_assign` is true, items are assigned to columns based on their
@@ -694,6 +721,19 @@ fn validate_and_build_columns(
            continue;
        }

+        // Reject valleys where the smaller side is just a column of list
+        // markers (bullets aligned at the left margin). This is a common
+        // pattern in PDFs where ● starts each list item: histogram detection
+        // sees the gap between bullet and content as a gutter.
+        let smaller_items: &[&&TextItem] = if left_items.len() <= right_items.len() {
+            &left_items
+        } else {
+            &right_items
+        };
+        if is_list_marker_column(smaller_items) {
+            continue;
+        }
+
        // Check vertical overlap
        if y_range > 0.0 {
            let left_y_min = left_items.iter().map(|i| i.y).fold(f32::INFINITY, f32::min);
@@ -1780,6 +1820,63 @@ mod tests {
        );
    }

+    #[test]
+    fn bullet_marker_column_not_detected_as_column() {
+        // Pattern: every line is `● <content>`, with ● at x=90 and content
+        // starting at x=104. Histogram detection sees a gutter between them
+        // and would split the page into a "bullet column" and "content column",
+        // scrambling every list item.
+        let mut items = Vec::new();
+        for i in 0..15 {
+            let y = 750.0 - i as f32 * 30.0;
+            items.push(make_item(1, 90.0, y, "●"));
+            items.push(make_item(
+                1,
+                104.0,
+                y,
+                "FullContentLineTextHere________________",
+            ));
+        }
+        // Pad with content to satisfy min item count for column detection.
+        for i in 0..15 {
+            let y = 300.0 - i as f32 * 14.0;
+            items.push(make_item(1, 72.0, y, "FootnoteText_____________________"));
+        }
+
+        let cols = detect_columns(&items, 1, false);
+        assert_eq!(
+            cols.len(),
+            1,
+            "Bullet markers aligned at left margin should not be treated as their own column"
+        );
+    }
+
+    #[test]
+    fn is_list_marker_column_detects_bullets() {
+        let items = vec![
+            make_item(1, 90.0, 100.0, "●"),
+            make_item(1, 90.0, 114.0, "●"),
+            make_item(1, 90.0, 128.0, "●"),
+            make_item(1, 90.0, 142.0, "●"),
+        ];
+        let refs: Vec<&TextItem> = items.iter().collect();
+        let wrapped: Vec<&&TextItem> = refs.iter().collect();
+        assert!(is_list_marker_column(&wrapped));
+    }
+
+    #[test]
+    fn is_list_marker_column_rejects_prose() {
+        let items = vec![
+            make_item(1, 30.0, 100.0, "Regular prose line"),
+            make_item(1, 30.0, 114.0, "Another sentence"),
+            make_item(1, 30.0, 128.0, "Third line"),
+            make_item(1, 30.0, 142.0, "Fourth line"),
+        ];
+        let refs: Vec<&TextItem> = items.iter().collect();
+        let wrapped: Vec<&&TextItem> = refs.iter().collect();
+        assert!(!is_list_marker_column(&wrapped));
+    }
+
    #[test]
    fn premask_narrow_line_not_masked() {
        // Items that form a line spanning only ~40% of column width → not masked
@@ -1,418 +0,0 @@
-//! Unicode to LaTeX character mapping.
-//!
-//! Maps Unicode math symbols, Greek letters, operators, and relations to their
-//! LaTeX command equivalents. Only includes characters that commonly appear in
-//! PDF text extraction of mathematical formulas.
-
-use std::collections::HashMap;
-use std::sync::OnceLock;
-
-/// Returns a reference to the global Unicode → LaTeX mapping table.
-pub fn unicode_to_latex_map() -> &'static HashMap<char, &'static str> {
-    static MAP: OnceLock<HashMap<char, &'static str>> = OnceLock::new();
-    MAP.get_or_init(build_map)
-}
-
-fn build_map() -> HashMap<char, &'static str> {
-    let entries: &[(char, &str)] = &[
-        // ── Greek lowercase ─────────────────────────────────────────
-        ('\u{03B1}', r"\alpha"),
-        ('\u{03B2}', r"\beta"),
-        ('\u{03B3}', r"\gamma"),
-        ('\u{03B4}', r"\delta"),
-        ('\u{03B5}', r"\varepsilon"),
-        ('\u{03F5}', r"\epsilon"),
-        ('\u{03B6}', r"\zeta"),
-        ('\u{03B7}', r"\eta"),
-        ('\u{03B8}', r"\theta"),
-        ('\u{03D1}', r"\vartheta"),
-        ('\u{03B9}', r"\iota"),
-        ('\u{03BA}', r"\kappa"),
-        ('\u{03BB}', r"\lambda"),
-        ('\u{03BC}', r"\mu"),
-        ('\u{03BD}', r"\nu"),
-        ('\u{03BE}', r"\xi"),
-        ('\u{03C0}', r"\pi"),
-        ('\u{03D6}', r"\varpi"),
-        ('\u{03C1}', r"\rho"),
-        ('\u{03C2}', r"\varsigma"),
-        ('\u{03C3}', r"\sigma"),
-        ('\u{03C4}', r"\tau"),
-        ('\u{03C5}', r"\upsilon"),
-        ('\u{03C6}', r"\varphi"),
-        ('\u{03D5}', r"\phi"),
-        ('\u{03C7}', r"\chi"),
-        ('\u{03C8}', r"\psi"),
-        ('\u{03C9}', r"\omega"),
-        // ── Greek uppercase ─────────────────────────────────────────
-        ('\u{0393}', r"\Gamma"),
-        ('\u{0394}', r"\Delta"),
-        ('\u{0398}', r"\Theta"),
-        ('\u{039B}', r"\Lambda"),
-        ('\u{039E}', r"\Xi"),
-        ('\u{03A0}', r"\Pi"),
-        ('\u{03A3}', r"\Sigma"),
-        ('\u{03A5}', r"\Upsilon"),
-        ('\u{03A6}', r"\Phi"),
-        ('\u{03A8}', r"\Psi"),
-        ('\u{03A9}', r"\Omega"),
-        // ── Large operators ─────────────────────────────────────────
-        ('\u{222B}', r"\int"),
-        ('\u{222C}', r"\iint"),
-        ('\u{222D}', r"\iiint"),
-        ('\u{222E}', r"\oint"),
-        ('\u{2211}', r"\sum"),
-        ('\u{220F}', r"\prod"),
-        ('\u{2210}', r"\coprod"),
-        // ── Roots / radicals ────────────────────────────────────────
-        ('\u{221A}', r"\sqrt"),
-        // ── Calculus / differential ─────────────────────────────────
-        ('\u{2202}', r"\partial"),
-        ('\u{2207}', r"\nabla"),
-        // ── Binary operators ────────────────────────────────────────
-        ('\u{00B1}', r"\pm"),
-        ('\u{2213}', r"\mp"),
-        ('\u{00D7}', r"\times"),
-        ('\u{00F7}', r"\div"),
-        ('\u{2217}', r"\ast"),
-        ('\u{22C6}', r"\star"),
-        ('\u{00B7}', r"\cdot"),
-        ('\u{2219}', r"\bullet"),
-        ('\u{2218}', r"\circ"),
-        ('\u{2020}', r"\dagger"),
-        ('\u{2021}', r"\ddagger"),
-        ('\u{2295}', r"\oplus"),
-        ('\u{2297}', r"\otimes"),
-        ('\u{2227}', r"\wedge"),
-        ('\u{2228}', r"\vee"),
-        ('\u{2229}', r"\cap"),
-        ('\u{222A}', r"\cup"),
-        // ── Relations ───────────────────────────────────────────────
-        ('\u{2264}', r"\leq"),
-        ('\u{2265}', r"\geq"),
-        ('\u{2260}', r"\neq"),
-        ('\u{2248}', r"\approx"),
-        ('\u{223C}', r"\sim"),
-        ('\u{2243}', r"\simeq"),
-        ('\u{2261}', r"\equiv"),
-        ('\u{226A}', r"\ll"),
-        ('\u{226B}', r"\gg"),
-        ('\u{221D}', r"\propto"),
-        ('\u{2208}', r"\in"),
-        ('\u{2209}', r"\notin"),
-        ('\u{220B}', r"\ni"),
-        ('\u{2282}', r"\subset"),
-        ('\u{2283}', r"\supset"),
-        ('\u{2286}', r"\subseteq"),
-        ('\u{2287}', r"\supseteq"),
-        ('\u{22A2}', r"\vdash"),
-        ('\u{22A3}', r"\dashv"),
-        ('\u{22A4}', r"\top"),
-        ('\u{22A5}', r"\bot"),
-        ('\u{2225}', r"\parallel"),
-        ('\u{22A5}', r"\perp"),
-        // ── Arrows ──────────────────────────────────────────────────
-        ('\u{2190}', r"\leftarrow"),
-        ('\u{2192}', r"\to"),
-        ('\u{2191}', r"\uparrow"),
-        ('\u{2193}', r"\downarrow"),
-        ('\u{2194}', r"\leftrightarrow"),
-        ('\u{21D0}', r"\Leftarrow"),
-        ('\u{21D2}', r"\Rightarrow"),
-        ('\u{21D4}', r"\Leftrightarrow"),
-        ('\u{21A6}', r"\mapsto"),
-        ('\u{2197}', r"\nearrow"),
-        ('\u{2198}', r"\searrow"),
-        // ── Miscellaneous symbols ───────────────────────────────────
-        ('\u{221E}', r"\infty"),
-        ('\u{2200}', r"\forall"),
-        ('\u{2203}', r"\exists"),
-        ('\u{2204}', r"\nexists"),
-        ('\u{2205}', r"\emptyset"),
-        ('\u{00AC}', r"\neg"),
-        ('\u{00B0}', r"^\circ"),
-        ('\u{2032}', r"'"),  // prime (common in physics: x')
-        ('\u{2033}', r"''"), // double prime
-        ('\u{210F}', r"\hbar"),
-        ('\u{2113}', r"\ell"),
-        ('\u{211C}', r"\Re"),
-        ('\u{2111}', r"\Im"),
-        ('\u{2118}', r"\wp"),
-        ('\u{2135}', r"\aleph"),
-        // ── Dots ────────────────────────────────────────────────────
-        ('\u{22EF}', r"\cdots"),
-        ('\u{22EE}', r"\vdots"),
-        ('\u{22F1}', r"\ddots"),
-        ('\u{2026}', r"\ldots"),
-        // ── Delimiters / brackets ───────────────────────────────────
-        ('\u{27E8}', r"\langle"),
-        ('\u{27E9}', r"\rangle"),
-        ('\u{2308}', r"\lceil"),
-        ('\u{2309}', r"\rceil"),
-        ('\u{230A}', r"\lfloor"),
-        ('\u{230B}', r"\rfloor"),
-        ('\u{2016}', r"\|"),
-        // ── Accents / decorations (as standalone chars) ─────────────
-        ('\u{0302}', r"\hat{}"),
-        ('\u{0303}', r"\tilde{}"),
-        ('\u{0304}', r"\bar{}"),
-        ('\u{0307}', r"\dot{}"),
-        ('\u{0308}', r"\ddot{}"),
-        ('\u{20D7}', r"\vec{}"),
-        // Hat/tilde as standalone characters (sometimes extracted separately)
-        ('\u{02C6}', r"\hat{}"),
-        ('\u{02DC}', r"\tilde{}"),
-        // ── Subscript/superscript digits (Unicode) ──────────────────
-        ('\u{2070}', "^{0}"),
-        ('\u{00B9}', "^{1}"),
-        ('\u{00B2}', "^{2}"),
-        ('\u{00B3}', "^{3}"),
-        ('\u{2074}', "^{4}"),
-        ('\u{2075}', "^{5}"),
-        ('\u{2076}', "^{6}"),
-        ('\u{2077}', "^{7}"),
-        ('\u{2078}', "^{8}"),
-        ('\u{2079}', "^{9}"),
-        ('\u{207A}', "^{+}"),
-        ('\u{207B}', "^{-}"),
-        ('\u{2080}', "_{0}"),
-        ('\u{2081}', "_{1}"),
-        ('\u{2082}', "_{2}"),
-        ('\u{2083}', "_{3}"),
-        ('\u{2084}', "_{4}"),
-        ('\u{2085}', "_{5}"),
-        ('\u{2086}', "_{6}"),
-        ('\u{2087}', "_{7}"),
-        ('\u{2088}', "_{8}"),
-        ('\u{2089}', "_{9}"),
-        ('\u{208A}', "_{+}"),
-        ('\u{208B}', "_{-}"),
-        // ── Math italic letters (sometimes used in PDF fonts) ───────
-        // These map back to plain ASCII in LaTeX (math mode handles italics)
-    ];
-
-    let mut map = HashMap::with_capacity(entries.len());
-    for &(ch, latex) in entries {
-        map.insert(ch, latex);
-    }
-    map
-}
-
-/// Convert a single character to its LaTeX representation.
-///
-/// Returns `Some(latex_str)` if the character has a known mapping,
-/// or `None` if it should be kept as-is.
-pub fn char_to_latex(ch: char) -> Option<&'static str> {
-    unicode_to_latex_map().get(&ch).copied()
-}
-
-/// Returns true if a character is a "known math character" — either ASCII
-/// alphanumeric, basic punctuation used in math, or a mapped Unicode symbol.
-pub fn is_known_math_char(ch: char) -> bool {
-    if ch.is_ascii_alphanumeric() {
-        return true;
-    }
-    // Common ASCII math characters
-    matches!(
-        ch,
-        '+' | '-'
-            | '*'
-            | '/'
-            | '='
-            | '<'
-            | '>'
-            | '('
-            | ')'
-            | '['
-            | ']'
-            | '{'
-            | '}'
-            | ','
-            | '.'
-            | ':'
-            | ';'
-            | '!'
-            | '|'
-            | '\''
-            | '"'
-            | '^'
-            | '_'
-            | '~'
-            | ' '
-            | '\n'
-            | '\t'
-    ) || unicode_to_latex_map().contains_key(&ch)
-}
-
-/// Convert a string of text to LaTeX, applying per-character mappings.
-/// Characters without mappings are left as-is.
-///
-/// Returns `(latex_string, fraction_of_chars_that_were_known)`.
-pub fn text_to_latex_chars(text: &str) -> (String, f32) {
-    let map = unicode_to_latex_map();
-    let mut result = String::with_capacity(text.len() * 2);
-    let mut total_nonws = 0usize;
-    let mut known = 0usize;
-
-    for ch in text.chars() {
-        if ch.is_whitespace() {
-            result.push(ch);
-            continue;
-        }
-        total_nonws += 1;
-
-        if let Some(latex) = map.get(&ch) {
-            // Add space before LaTeX commands that start with backslash
-            // to prevent them from merging with preceding text
-            if latex.starts_with('\\') && !result.is_empty() && !result.ends_with(' ') {
-                // Only add space if the last char is alphanumeric (to avoid "x \alpha" but allow "( \alpha")
-                let last = result.chars().last().unwrap();
-                if last.is_alphanumeric() || last == '}' {
-                    result.push(' ');
-                }
-            }
-            result.push_str(latex);
-            // Add trailing space after LaTeX commands so next char doesn't merge
-            if latex.starts_with('\\') && !latex.ends_with('}') && !latex.ends_with('\'') {
-                result.push(' ');
-            }
-            known += 1;
-        } else if is_known_math_char(ch) {
-            result.push(ch);
-            known += 1;
-        } else {
-            // Unknown character — keep it but it lowers confidence
-            result.push(ch);
-        }
-    }
-
-    let frac = if total_nonws == 0 {
-        1.0
-    } else {
-        known as f32 / total_nonws as f32
-    };
-    (result, frac)
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn test_greek_lowercase() {
-        assert_eq!(char_to_latex('\u{03B1}'), Some(r"\alpha"));
-        assert_eq!(char_to_latex('\u{03B2}'), Some(r"\beta"));
-        assert_eq!(char_to_latex('\u{03B3}'), Some(r"\gamma"));
-        assert_eq!(char_to_latex('\u{03B4}'), Some(r"\delta"));
-        assert_eq!(char_to_latex('\u{03B5}'), Some(r"\varepsilon"));
-        assert_eq!(char_to_latex('\u{03B6}'), Some(r"\zeta"));
-        assert_eq!(char_to_latex('\u{03B7}'), Some(r"\eta"));
-        assert_eq!(char_to_latex('\u{03B8}'), Some(r"\theta"));
-        assert_eq!(char_to_latex('\u{03D1}'), Some(r"\vartheta"));
-        assert_eq!(char_to_latex('\u{03B9}'), Some(r"\iota"));
-        assert_eq!(char_to_latex('\u{03BA}'), Some(r"\kappa"));
-        assert_eq!(char_to_latex('\u{03BB}'), Some(r"\lambda"));
-        assert_eq!(char_to_latex('\u{03BC}'), Some(r"\mu"));
-        assert_eq!(char_to_latex('\u{03BD}'), Some(r"\nu"));
-        assert_eq!(char_to_latex('\u{03BE}'), Some(r"\xi"));
-        assert_eq!(char_to_latex('\u{03C0}'), Some(r"\pi"));
-        assert_eq!(char_to_latex('\u{03C1}'), Some(r"\rho"));
-        assert_eq!(char_to_latex('\u{03C3}'), Some(r"\sigma"));
-        assert_eq!(char_to_latex('\u{03C4}'), Some(r"\tau"));
-        assert_eq!(char_to_latex('\u{03C9}'), Some(r"\omega"));
-    }
-
-    #[test]
-    fn test_greek_uppercase() {
-        assert_eq!(char_to_latex('\u{0393}'), Some(r"\Gamma"));
-        assert_eq!(char_to_latex('\u{0394}'), Some(r"\Delta"));
-        assert_eq!(char_to_latex('\u{0398}'), Some(r"\Theta"));
-        assert_eq!(char_to_latex('\u{039B}'), Some(r"\Lambda"));
-        assert_eq!(char_to_latex('\u{03A3}'), Some(r"\Sigma"));
-        assert_eq!(char_to_latex('\u{03A6}'), Some(r"\Phi"));
-        assert_eq!(char_to_latex('\u{03A9}'), Some(r"\Omega"));
-    }
-
-    #[test]
-    fn test_operators() {
-        assert_eq!(char_to_latex('\u{222B}'), Some(r"\int"));
-        assert_eq!(char_to_latex('\u{2211}'), Some(r"\sum"));
-        assert_eq!(char_to_latex('\u{220F}'), Some(r"\prod"));
-        assert_eq!(char_to_latex('\u{221A}'), Some(r"\sqrt"));
-        assert_eq!(char_to_latex('\u{2202}'), Some(r"\partial"));
-        assert_eq!(char_to_latex('\u{2207}'), Some(r"\nabla"));
-    }
-
-    #[test]
-    fn test_relations() {
-        assert_eq!(char_to_latex('\u{2264}'), Some(r"\leq"));
-        assert_eq!(char_to_latex('\u{2265}'), Some(r"\geq"));
-        assert_eq!(char_to_latex('\u{2260}'), Some(r"\neq"));
-        assert_eq!(char_to_latex('\u{2248}'), Some(r"\approx"));
-        assert_eq!(char_to_latex('\u{223C}'), Some(r"\sim"));
-        assert_eq!(char_to_latex('\u{226A}'), Some(r"\ll"));
-        assert_eq!(char_to_latex('\u{226B}'), Some(r"\gg"));
-        assert_eq!(char_to_latex('\u{221E}'), Some(r"\infty"));
-        assert_eq!(char_to_latex('\u{2208}'), Some(r"\in"));
-        assert_eq!(char_to_latex('\u{2209}'), Some(r"\notin"));
-        assert_eq!(char_to_latex('\u{2282}'), Some(r"\subset"));
-    }
-
-    #[test]
-    fn test_misc_symbols() {
-        assert_eq!(char_to_latex('\u{00B1}'), Some(r"\pm"));
-        assert_eq!(char_to_latex('\u{00D7}'), Some(r"\times"));
-        assert_eq!(char_to_latex('\u{00B7}'), Some(r"\cdot"));
-        assert_eq!(char_to_latex('\u{00B0}'), Some(r"^\circ"));
-        assert_eq!(char_to_latex('\u{2192}'), Some(r"\to"));
-        assert_eq!(char_to_latex('\u{21D2}'), Some(r"\Rightarrow"));
-        assert_eq!(char_to_latex('\u{210F}'), Some(r"\hbar"));
-    }
-
-    #[test]
-    fn test_unicode_super_sub_digits() {
-        assert_eq!(char_to_latex('\u{00B2}'), Some("^{2}"));
-        assert_eq!(char_to_latex('\u{00B3}'), Some("^{3}"));
-        assert_eq!(char_to_latex('\u{2082}'), Some("_{2}"));
-        assert_eq!(char_to_latex('\u{2083}'), Some("_{3}"));
-    }
-
-    #[test]
-    fn test_is_known_math_char() {
-        // ASCII math
-        assert!(is_known_math_char('+'));
-        assert!(is_known_math_char('='));
-        assert!(is_known_math_char('('));
-        assert!(is_known_math_char('x'));
-        assert!(is_known_math_char('0'));
-        // Mapped Unicode
-        assert!(is_known_math_char('\u{03B1}')); // alpha
-        assert!(is_known_math_char('\u{2264}')); // leq
-                                                 // Unknown
-        assert!(!is_known_math_char('\u{E000}')); // PUA
-        assert!(!is_known_math_char('\u{4E00}')); // CJK
-    }
-
-    #[test]
-    fn test_text_to_latex_simple() {
-        let (latex, frac) = text_to_latex_chars("x + y");
-        assert_eq!(latex, "x + y");
-        assert!((frac - 1.0).abs() < 0.01);
-    }
-
-    #[test]
-    fn test_text_to_latex_greek() {
-        let (latex, frac) = text_to_latex_chars("αβγ");
-        assert!(latex.contains(r"\alpha"));
-        assert!(latex.contains(r"\beta"));
-        assert!(latex.contains(r"\gamma"));
-        assert!((frac - 1.0).abs() < 0.01);
-    }
-
-    #[test]
-    fn test_text_to_latex_mixed() {
-        let (latex, frac) = text_to_latex_chars("x ≤ y");
-        assert!(latex.contains(r"\leq"));
-        assert!((frac - 1.0).abs() < 0.01);
-    }
-}
@@ -1,3 +1,9 @@
+// Rust 1.95 introduced collapsible_match for `if` inside match arms.
+// The content-stream parsers use this pattern extensively (match on operator
+// name, then check `in_text_block && !op.operands.is_empty()`). Collapsing
+// these into match guards would hurt readability. Allow crate-wide.
+#![allow(clippy::collapsible_match)]
+
 //! Smart PDF detection and text extraction using lopdf
 //!
 //! # Quick start
@@ -28,7 +34,6 @@ pub mod python;
 pub mod adobe_korea1;
 pub mod detector;
 pub mod extractor;
-pub mod formula_latex;
 pub mod glyph_names;
 pub mod markdown;
 pub mod process_mode;
@@ -331,13 +336,17 @@ pub struct PagesExtractionResult {
    pub is_complex: bool,
 }

-/// Extract formatted markdown for specific pages of a PDF, with layout
+/// Extract formatted markdown for pages of a PDF, with layout
 /// classification metadata.
 ///
 /// Unlike [`process_pdf_mem`] which returns one concatenated markdown string,
 /// this returns per-page markdown so callers can mix direct extraction
 /// (for simple text pages) with GPU OCR (for complex/scanned pages).
 ///
+/// When `pages` is `None`, every page (0-indexed, in document order) is
+/// returned. When `Some(&[...])`, only the listed 0-indexed pages are
+/// returned, in the caller's order.
+///
 /// Font statistics are computed from the full document so header
 /// detection thresholds are consistent regardless of which pages are
 /// requested. Per-page `needs_ocr` is set when the page has GID-encoded
@@ -347,7 +356,7 @@ pub struct PagesExtractionResult {
 /// at near-zero cost since the items/rects/lines are already in memory.
 pub fn extract_pages_markdown_mem(
    buffer: &[u8],
-    pages: &[u32],
+    pages: Option<&[u32]>,
 ) -> Result<PagesExtractionResult, PdfError> {
    validate_pdf_bytes(buffer)?;
    let (doc, page_count) = load_document_from_mem(buffer)?;
@@ -363,10 +372,20 @@ pub fn extract_pages_markdown_mem(
    // Compute font stats from full document (cross-page consistency).
    let font_stats = markdown::analysis::calculate_font_stats_from_items(&all_items);

-    let mut results = Vec::with_capacity(pages.len());
+    // When caller doesn't specify pages, return every page in document order.
+    let all_pages: Vec<u32>;
+    let pages_slice: &[u32] = match pages {
+        Some(p) => p,
+        None => {
+            all_pages = (0..page_count).collect();
+            &all_pages
+        }
+    };
+
+    let mut results = Vec::with_capacity(pages_slice.len());
    let mut pages_needing_ocr = Vec::new();

-    for &page_0idx in pages {
+    for &page_0idx in pages_slice {
        // Out-of-range pages → empty + needs_ocr
        if page_0idx >= page_count {
            pages_needing_ocr.push(page_0idx + 1);
@@ -439,6 +458,20 @@ pub fn extract_pages_markdown_mem(
    })
 }

+/// Path-based wrapper for [`extract_pages_markdown_mem`].
+///
+/// Reads the PDF from disk and extracts per-page markdown. Pass `None` for
+/// `pages` to return every page in document order, or `Some(&[...])` to
+/// restrict to specific 0-indexed pages (in caller-supplied order).
+pub fn extract_pages_markdown<P: AsRef<Path>>(
+    path: P,
+    pages: Option<&[u32]>,
+) -> Result<PagesExtractionResult, PdfError> {
+    validate_pdf_file(&path)?;
+    let buffer = std::fs::read(path.as_ref())?;
+    extract_pages_markdown_mem(&buffer, pages)
+}
+
 // =========================================================================
 // Region-based text extraction (for hybrid OCR pipelines)
 // =========================================================================
@@ -463,98 +496,6 @@ pub struct PageRegionResult {
    pub regions: Vec<RegionText>,
 }

-/// Shared page extraction state for region-based extraction functions.
-struct RegionExtractionData {
-    items_by_page: HashMap<u32, Vec<TextItem>>,
-    page_heights: HashMap<u32, f32>,
-    #[allow(dead_code)]
-    gid_pages: HashSet<u32>,
-    page_thresholds: HashMap<u32, f32>,
-    rotated_pages: HashSet<u32>,
-}
-
-/// Extract text items, page heights, and metadata for the pages needed by region queries.
-///
-/// This is the shared boilerplate for `extract_text_in_regions_mem`,
-/// `extract_tables_in_regions_mem`, and `extract_formulas_in_regions_mem`.
-fn prepare_region_extraction(
-    buffer: &[u8],
-    page_regions: &[(u32, Vec<[f32; 4]>)],
-) -> Result<RegionExtractionData, PdfError> {
-    validate_pdf_bytes(buffer)?;
-    let (doc, _page_count) = load_document_from_mem(buffer)?;
-    let pages = doc.get_pages();
-
-    let needed_pages: HashSet<u32> = page_regions.iter().map(|(p, _)| p + 1).collect();
-
-    // Fast mode: skip expensive TrueType font fallback parsing.
-    // Fonts that can't be decoded from ToUnicode alone will produce empty/garbage
-    // text, triggering needs_ocr=true → GPU OCR fallback in the pipeline.
-    let font_cmaps = FontCMaps::from_doc_pages_fast(&doc, Some(&needed_pages));
-
-    let mut items_by_page: HashMap<u32, Vec<TextItem>> = HashMap::new();
-    let mut page_heights: HashMap<u32, f32> = HashMap::new();
-    let mut gid_pages: HashSet<u32> = HashSet::new();
-    let mut page_thresholds: HashMap<u32, f32> = HashMap::new();
-    let mut rotated_pages: HashSet<u32> = HashSet::new();
-
-    for (page_num, &page_id) in pages.iter() {
-        if !needed_pages.contains(page_num) {
-            continue;
-        }
-
-        let height = get_page_height(&doc, page_id).unwrap_or(792.0);
-        page_heights.insert(*page_num, height);
-
-        let ((mut items, _rects, _lines), has_gid, coords_rotated) =
-            extractor::content_stream::extract_page_text_items(
-                &doc,
-                page_id,
-                *page_num,
-                &font_cmaps,
-                false,
-            )?;
-        let threshold = text_utils::fix_letterspaced_items(&mut items);
-        if threshold > 0.10 {
-            page_thresholds.insert(*page_num, threshold);
-        }
-        if has_gid {
-            gid_pages.insert(*page_num);
-        }
-        if coords_rotated {
-            rotated_pages.insert(*page_num);
-        }
-        items_by_page.insert(*page_num, items);
-    }
-
-    Ok(RegionExtractionData {
-        items_by_page,
-        page_heights,
-        gid_pages,
-        page_thresholds,
-        rotated_pages,
-    })
-}
-
-/// Resolve per-page coord space and adaptive threshold for a given page.
-fn page_region_context(
-    data: &RegionExtractionData,
-    page_1idx: u32,
-) -> (f32, f32, RegionCoordSpace) {
-    let page_h = data.page_heights.get(&page_1idx).copied().unwrap_or(792.0);
-    let adaptive_threshold = data
-        .page_thresholds
-        .get(&page_1idx)
-        .copied()
-        .unwrap_or(0.10);
-    let coords = if data.rotated_pages.contains(&page_1idx) {
-        RegionCoordSpace::Rotated90Ccw
-    } else {
-        RegionCoordSpace::Standard
-    };
-    (page_h, adaptive_threshold, coords)
-}
-
 /// Extract text within bounding-box regions from a PDF in memory.
 ///
 /// This is designed for hybrid OCR pipelines: a layout model detects regions
@@ -578,14 +519,70 @@ pub fn extract_text_in_regions_mem(
    buffer: &[u8],
    page_regions: &[(u32, Vec<[f32; 4]>)],
 ) -> Result<Vec<PageRegionResult>, PdfError> {
-    let data = prepare_region_extraction(buffer, page_regions)?;
+    validate_pdf_bytes(buffer)?;
+    let (doc, _page_count) = load_document_from_mem(buffer)?;
+    let pages = doc.get_pages();

+    // Build a set of pages we need to extract (1-indexed for lopdf)
+    let needed_pages: HashSet<u32> = page_regions.iter().map(|(p, _)| p + 1).collect();
+
+    // Fast mode: skip expensive TrueType font fallback parsing.
+    // Fonts that can't be decoded from ToUnicode alone will produce empty/garbage
+    // text, triggering needs_ocr=true → GPU OCR fallback in the pipeline.
+    let font_cmaps = FontCMaps::from_doc_pages_fast(&doc, Some(&needed_pages));
+
+    // Extract text items for needed pages only
+    let mut items_by_page: HashMap<u32, Vec<TextItem>> = HashMap::new();
+    let mut page_heights: HashMap<u32, f32> = HashMap::new();
+    let mut gid_pages: HashSet<u32> = HashSet::new();
+    let mut page_thresholds: HashMap<u32, f32> = HashMap::new();
+    let mut rotated_pages: HashSet<u32> = HashSet::new();
+
+    for (page_num, &page_id) in pages.iter() {
+        if !needed_pages.contains(page_num) {
+            continue;
+        }
+
+        // Get page height from MediaBox for coordinate flip
+        let height = get_page_height(&doc, page_id).unwrap_or(792.0);
+        page_heights.insert(*page_num, height);
+
+        // Extract text items for this page
+        let ((mut items, _rects, _lines), has_gid, coords_rotated) =
+            extractor::content_stream::extract_page_text_items(
+                &doc,
+                page_id,
+                *page_num,
+                &font_cmaps,
+                false,
+            )?;
+        let threshold = text_utils::fix_letterspaced_items(&mut items);
+        if threshold > 0.10 {
+            page_thresholds.insert(*page_num, threshold);
+        }
+        if has_gid {
+            gid_pages.insert(*page_num);
+        }
+        if coords_rotated {
+            rotated_pages.insert(*page_num);
+        }
+        items_by_page.insert(*page_num, items);
+    }
+
+    // For each page's regions, filter and assemble text
    let mut results = Vec::with_capacity(page_regions.len());

    for (page_0idx, regions) in page_regions {
        let page_1idx = page_0idx + 1;
-        let items = data.items_by_page.get(&page_1idx);
-        let (page_h, adaptive_threshold, coords) = page_region_context(&data, page_1idx);
+        let items = items_by_page.get(&page_1idx);
+        let page_h = page_heights.get(&page_1idx).copied().unwrap_or(792.0);
+        let _page_has_gid = gid_pages.contains(&page_1idx);
+        let adaptive_threshold = page_thresholds.get(&page_1idx).copied().unwrap_or(0.10);
+        let coords = if rotated_pages.contains(&page_1idx) {
+            RegionCoordSpace::Rotated90Ccw
+        } else {
+            RegionCoordSpace::Standard
+        };

        let mut page_results = Vec::with_capacity(regions.len());

@@ -639,20 +636,74 @@ pub fn extract_tables_in_regions_mem(
    buffer: &[u8],
    page_regions: &[(u32, Vec<[f32; 4]>)],
 ) -> Result<Vec<PageRegionResult>, PdfError> {
-    let data = prepare_region_extraction(buffer, page_regions)?;
+    validate_pdf_bytes(buffer)?;
+    let (doc, _page_count) = load_document_from_mem(buffer)?;
+    let pages = doc.get_pages();
+
+    let needed_pages: HashSet<u32> = page_regions.iter().map(|(p, _)| p + 1).collect();
+    let font_cmaps = FontCMaps::from_doc_pages_fast(&doc, Some(&needed_pages));
+
+    let mut items_by_page: HashMap<u32, Vec<TextItem>> = HashMap::new();
+    let mut page_heights: HashMap<u32, f32> = HashMap::new();
+    let mut gid_pages: HashSet<u32> = HashSet::new();
+    let mut page_thresholds: HashMap<u32, f32> = HashMap::new();
+    let mut rotated_pages: HashSet<u32> = HashSet::new();
+
+    for (page_num, &page_id) in pages.iter() {
+        if !needed_pages.contains(page_num) {
+            continue;
+        }
+        let height = get_page_height(&doc, page_id).unwrap_or(792.0);
+        page_heights.insert(*page_num, height);
+
+        let ((mut items, _rects, _lines), has_gid, coords_rotated) =
+            extractor::content_stream::extract_page_text_items(
+                &doc,
+                page_id,
+                *page_num,
+                &font_cmaps,
+                false,
+            )?;
+        let threshold = text_utils::fix_letterspaced_items(&mut items);
+        if threshold > 0.10 {
+            page_thresholds.insert(*page_num, threshold);
+        }
+        if has_gid {
+            gid_pages.insert(*page_num);
+        }
+        if coords_rotated {
+            rotated_pages.insert(*page_num);
+        }
+        items_by_page.insert(*page_num, items);
+    }

    let mut results = Vec::with_capacity(page_regions.len());

    for (page_0idx, regions) in page_regions {
        let page_1idx = page_0idx + 1;
-        let items = data.items_by_page.get(&page_1idx);
-        let (page_h, _adaptive_threshold, coords) = page_region_context(&data, page_1idx);
+        let items = items_by_page.get(&page_1idx);
+        let page_h = page_heights.get(&page_1idx).copied().unwrap_or(792.0);
+        let _page_has_gid = gid_pages.contains(&page_1idx);
+        let coords = if rotated_pages.contains(&page_1idx) {
+            RegionCoordSpace::Rotated90Ccw
+        } else {
+            RegionCoordSpace::Standard
+        };

        let mut page_results = Vec::with_capacity(regions.len());

        for rect in regions {
            let [rx1, ry1, rx2, ry2] = *rect;

+            // Note: we intentionally DO NOT bail on page_has_gid here.
+            // The GID flag means some font on the page uses unresolvable
+            // glyph IDs, but that font may only appear in a logo or
+            // header — not in the table region. Instead we let the
+            // per-region text quality checks (is_garbage_text, is_cid_garbage,
+            // detect_encoding_issues) reject based on the actual extracted
+            // content. This avoids rejecting clean tables just because an
+            // unrelated decorative font on the same page is GID-encoded.
+
            let matched: Vec<TextItem> = match items {
                Some(items) => {
                    let bounds = region_bounds(rx1, ry1, rx2, ry2, page_h, coords);
@@ -733,164 +784,6 @@ pub fn extract_tables_in_regions_mem(
    Ok(results)
 }

-/// Extract formula text within bounding-box regions from a PDF in memory.
-///
-/// Similar to [`extract_text_in_regions_mem`] but uses formula-specific quality
-/// checks. Formula text is legitimately symbol-heavy (Greek letters, math
-/// operators, subscripts) so the generic `is_garbage_text` check — which rejects
-/// text with <50% alphanumeric characters — would false-positive on valid
-/// formula regions.
-///
-/// When the extracted text decodes cleanly, `needs_ocr` is `false` and the
-/// caller can skip GPU OCR. When extraction fails (empty, PUA-heavy, encoding
-/// issues), `needs_ocr` is `true` for OCR fallback.
-pub fn extract_formulas_in_regions_mem(
-    buffer: &[u8],
-    page_regions: &[(u32, Vec<[f32; 4]>)],
-) -> Result<Vec<PageRegionResult>, PdfError> {
-    let data = prepare_region_extraction(buffer, page_regions)?;
-
-    let mut results = Vec::with_capacity(page_regions.len());
-
-    for (page_0idx, regions) in page_regions {
-        let page_1idx = page_0idx + 1;
-        let items = data.items_by_page.get(&page_1idx);
-        let (page_h, adaptive_threshold, coords) = page_region_context(&data, page_1idx);
-
-        let mut page_results = Vec::with_capacity(regions.len());
-
-        for rect in regions {
-            let [rx1, ry1, rx2, ry2] = *rect;
-
-            let text = match items {
-                Some(items) => collect_text_in_region_with_options(
-                    items,
-                    rx1,
-                    ry1,
-                    rx2,
-                    ry2,
-                    page_h,
-                    coords,
-                    adaptive_threshold,
-                ),
-                None => String::new(),
-            };
-
-            // Formula-specific quality checks:
-            // - Skip is_garbage_text (formulas are legitimately symbol-heavy)
-            // - Keep CID/encoding checks (broken font decode is still broken)
-            // - Add PUA check (extensible delimiters that didn't decode)
-            let needs_ocr = text.trim().is_empty()
-                || is_cid_garbage(&text)
-                || detect_encoding_issues(&text)
-                || is_formula_garbage(&text);
-
-            page_results.push(RegionText { text, needs_ocr });
-        }
-
-        results.push(PageRegionResult {
-            page: *page_0idx,
-            regions: page_results,
-        });
-    }
-
-    Ok(results)
-}
-
-/// Extract formula text within bounding-box regions and reconstruct LaTeX.
-///
-/// For each formula bbox, this function:
-/// 1. Gets positioned text items inside the bbox
-/// 2. Analyzes font sizes and vertical positions to detect sub/superscripts
-/// 3. Detects simple fractions from vertically stacked text
-/// 4. Converts Unicode math symbols to LaTeX commands
-/// 5. Reconstructs structured LaTeX from the positioned items
-///
-/// Each result includes a `confidence` score (0.0–1.0) indicating how reliable
-/// the LaTeX reconstruction is. Low-confidence results should be sent to GPU OCR.
-///
-/// The `needs_ocr` flag is set only when extraction fails entirely (empty text,
-/// encoding garbage), NOT based on confidence — the caller decides the threshold.
-pub fn extract_formulas_in_regions_as_latex(
-    buffer: &[u8],
-    page_regions: &[(u32, Vec<[f32; 4]>)],
-) -> Result<Vec<formula_latex::PageFormulaResult>, PdfError> {
-    let data = prepare_region_extraction(buffer, page_regions)?;
-
-    let mut results = Vec::with_capacity(page_regions.len());
-
-    for (page_0idx, regions) in page_regions {
-        let page_1idx = page_0idx + 1;
-        let items = data.items_by_page.get(&page_1idx);
-        let (page_h, _adaptive_threshold, coords) = page_region_context(&data, page_1idx);
-
-        let mut page_results = Vec::with_capacity(regions.len());
-
-        for rect in regions {
-            let [rx1, ry1, rx2, ry2] = *rect;
-
-            let result = match items {
-                Some(items) => {
-                    // Convert the bbox from top-left origin to the item coordinate space
-                    let bounds = region_bounds(rx1, ry1, rx2, ry2, page_h, coords);
-
-                    // Filter items whose center falls inside the bbox
-                    let matched = formula_latex::filter_items_in_bbox(
-                        items,
-                        bounds.x_min,
-                        bounds.y_min,
-                        bounds.x_max,
-                        bounds.y_max,
-                    );
-
-                    if matched.is_empty() {
-                        formula_latex::FormulaResult {
-                            latex: String::new(),
-                            raw_text: String::new(),
-                            confidence: 0.0,
-                            needs_ocr: true,
-                            confidence_breakdown: Vec::new(),
-                        }
-                    } else {
-                        let (latex, raw_text, confidence, confidence_breakdown) =
-                            formula_latex::reconstruct_latex(&matched);
-
-                        // Same formula-specific quality checks as extract_formulas_in_regions_mem
-                        let needs_ocr = raw_text.trim().is_empty()
-                            || is_cid_garbage(&raw_text)
-                            || detect_encoding_issues(&raw_text)
-                            || is_formula_garbage(&raw_text);
-
-                        formula_latex::FormulaResult {
-                            latex,
-                            raw_text,
-                            confidence,
-                            needs_ocr,
-                            confidence_breakdown,
-                        }
-                    }
-                }
-                None => formula_latex::FormulaResult {
-                    latex: String::new(),
-                    raw_text: String::new(),
-                    confidence: 0.0,
-                    needs_ocr: true,
-                    confidence_breakdown: Vec::new(),
-                },
-            };
-
-            page_results.push(result);
-        }
-
-        results.push(formula_latex::PageFormulaResult {
-            page: *page_0idx,
-            regions: page_results,
-        });
-    }
-
-    Ok(results)
-}
-
 /// Get page height in points from MediaBox.
 fn get_page_height(doc: &Document, page_id: lopdf::ObjectId) -> Option<f32> {
    let page_dict = doc.get_dictionary(page_id).ok()?;
@@ -1488,49 +1381,6 @@ fn is_cid_garbage(text: &str) -> bool {
    high_latin * 5 >= total * 2 && ascii_letters * 3 < total
 }

-/// Detect formula text that is unlikely to be usable despite passing generic checks.
-///
-/// Formula text (Greek letters, math operators, variables) is legitimately
-/// symbol-heavy, so `is_garbage_text` would false-positive. This check instead
-/// catches:
-///
-/// 1. **Private Use Area (PUA) characters** — TeX extensible delimiter glyphs
-///    (large brackets from CMEX fonts) often map to PUA U+E000–F8FF when the
-///    ToUnicode CMap is missing. >10% PUA means significant undecoded content.
-///
-/// 2. **Control characters** — C0 controls (U+0000–001F excluding whitespace)
-///    indicate broken font encoding, not formula content. >30% is rejected.
-fn is_formula_garbage(text: &str) -> bool {
-    let mut total = 0usize;
-    let mut pua = 0usize;
-    let mut control = 0usize;
-    for ch in text.chars() {
-        if ch.is_whitespace() {
-            continue;
-        }
-        total += 1;
-        if ('\u{E000}'..='\u{F8FF}').contains(&ch) {
-            pua += 1;
-        }
-        let cp = ch as u32;
-        if cp < 0x20 {
-            control += 1;
-        }
-    }
-    if total < 3 {
-        return false;
-    }
-    // >10% PUA — significant undecoded extensible delimiters
-    if pua * 10 > total {
-        return true;
-    }
-    // >30% control chars — broken encoding
-    if control * 10 > total * 3 {
-        return true;
-    }
-    false
-}
-
 /// Detect markdown tables with suspicious structure that suggest the heuristic
 /// missed/mangled rows or columns. Returns true when the caller should treat
 /// the result as `needs_ocr` and fall back to GPU OCR.
@@ -1960,19 +1810,26 @@ fn compute_layout_complexity(
                markdown::filter_lines_to_band(lines, page, x_lo, x_hi)
            };

+            // TOC pages route through the table detector but render as flat
+            // lists. They aren't tables in any user-facing sense, so don't
+            // count them toward LayoutComplexity (would also trip the
+            // table-page guard in column detection below).
+            let has_data_table =
+                |tables: &[tables::Table]| tables.iter().any(|t| t.kind == tables::TableKind::Data);
+
            let (rect_tables, _) = tables::detect_tables_from_rects(&band_items, &band_rects, page);
-            if !rect_tables.is_empty() {
+            if has_data_table(&rect_tables) {
                found_table = true;
                break;
            }
            let line_tables = tables::detect_tables_from_lines(&band_items, &band_lines, page);
-            if !line_tables.is_empty() {
+            if has_data_table(&line_tables) {
                found_table = true;
                break;
            }
            // Heuristic fallback for borderless tables
            let heuristic_tables = tables::detect_tables(&band_items, base_size, false);
-            if !heuristic_tables.is_empty() {
+            if has_data_table(&heuristic_tables) {
                found_table = true;
                break;
            }
@@ -2248,58 +2105,4 @@ mod tests {
            "Valid Japanese text should not be flagged as garbage"
        );
    }
-
-    #[test]
-    fn test_is_formula_garbage_accepts_math_text() {
-        // Greek letters, math operators, variables — typical formula text
-        let formula = "Φ(ν) = ∫ ∞ dze −it r1 − iνr2 sinh z";
-        assert!(
-            !is_formula_garbage(formula),
-            "Valid formula text should not be flagged as garbage"
-        );
-
-        // Dense operator text
-        let operators = "α + β − γ × δ ÷ ε ≤ ζ ≥ η ≈ θ ≠ ι ± κ";
-        assert!(
-            !is_formula_garbage(operators),
-            "Math operator text should not be flagged as garbage"
-        );
-
-        // Short formula (e.g. single equation variable)
-        let short = "αβ";
-        assert!(
-            !is_formula_garbage(short),
-            "Short formula text should not be flagged"
-        );
-    }
-
-    #[test]
-    fn test_is_formula_garbage_rejects_pua_heavy() {
-        // Simulates extensible delimiters from CMEX fonts mapping to PUA
-        let pua_heavy = "x \u{F8EB} \u{F8EC} \u{F8ED} \u{F8F6} \u{F8F7} \u{F8F8} y";
-        assert!(
-            is_formula_garbage(pua_heavy),
-            "PUA-heavy text should be flagged as formula garbage"
-        );
-    }
-
-    #[test]
-    fn test_is_formula_garbage_rejects_control_chars() {
-        // Control characters indicate broken encoding
-        let control_heavy = "a\x01b\x02c\x03d\x04e\x05f\x06g\x07h\x08i";
-        assert!(
-            is_formula_garbage(control_heavy),
-            "Control-char-heavy text should be flagged as formula garbage"
-        );
-    }
-
-    #[test]
-    fn test_is_formula_garbage_accepts_few_pua() {
-        // A few PUA chars among many valid chars is fine (<10% threshold)
-        let mostly_good = "Φ(ν) = ∫ dze r1 − iνr2 sinh z α β γ δ ε ζ η θ \u{F8EB}";
-        assert!(
-            !is_formula_garbage(mostly_good),
-            "Mostly-good text with rare PUA should pass"
-        );
-    }
 }
@@ -64,6 +64,22 @@ pub(crate) fn is_caption_line(text: &str) -> bool {
    false
 }

+/// Check if text starts with an unambiguous bullet marker (●, •, ○, ◦).
+///
+/// Narrower than [`is_list_item`]: it excludes numbered/lettered patterns
+/// like `1.` or `a)`, which legitimately appear as section headings in many
+/// documents. Used by the heading classifier to reject bullet lines without
+/// also demoting numbered headings.
+pub(crate) fn starts_with_bullet_marker(text: &str) -> bool {
+    let trimmed = text.trim_start();
+    trimmed.starts_with("• ")
+        || trimmed.starts_with("● ")
+        || trimmed.starts_with("○ ")
+        || trimmed.starts_with("◦ ")
+        || trimmed.starts_with("- ")
+        || trimmed.starts_with("* ")
+}
+
 /// Check if text looks like a list item
 pub(crate) fn is_list_item(text: &str) -> bool {
    let trimmed = text.trim_start();
@@ -115,6 +131,16 @@ pub(crate) fn format_list_item(text: &str) -> String {
        if let Some(rest) = trimmed.strip_prefix(*bullet) {
            return format!("- {}", rest.trim_start());
        }
+        // Bullet inside a leading bold/italic run (e.g. "**● Label:** rest").
+        // The run wraps both the marker and the following label because both
+        // use a bold font in the PDF.
+        for wrapper in ["**", "*"] {
+            if let Some(after_open) = trimmed.strip_prefix(wrapper) {
+                if let Some(rest) = after_open.strip_prefix(*bullet) {
+                    return format!("- {}{}", wrapper, rest.trim_start());
+                }
+            }
+        }
    }

    if trimmed.starts_with("- ") || trimmed.starts_with("* ") {
@@ -198,3 +224,42 @@ pub(crate) fn is_monospace_font(font_name: &str) -> bool {

    patterns.iter().any(|p| lower.contains(p))
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn format_list_item_plain_bullet() {
+        assert_eq!(format_list_item("● Item"), "- Item");
+        assert_eq!(format_list_item("• Item"), "- Item");
+    }
+
+    #[test]
+    fn format_list_item_bullet_inside_bold() {
+        // PDF that uses bold font for both the marker and the label produces
+        // a single bold run like "**● Label:** rest"; the bullet must still
+        // be stripped and the bold wrapper preserved on the label.
+        assert_eq!(
+            format_list_item("**● Fraud: Willing cooperation;**"),
+            "- **Fraud: Willing cooperation;**"
+        );
+        assert_eq!(
+            format_list_item("**● Label:** rest of line"),
+            "- **Label:** rest of line"
+        );
+        assert_eq!(format_list_item("*● Italic:* rest"), "- *Italic:* rest");
+    }
+
+    #[test]
+    fn format_list_item_already_dash() {
+        assert_eq!(format_list_item("- existing"), "- existing");
+    }
+
+    #[test]
+    fn is_list_item_with_bullet_space() {
+        assert!(is_list_item("● Item"));
+        assert!(is_list_item("• Item"));
+        assert!(is_list_item("- Item"));
+    }
+}
@@ -9,7 +9,9 @@ use super::analysis::{
    bold_heading_level, calculate_font_stats, compute_heading_tiers, compute_paragraph_threshold,
    detect_header_level, font_size_rarity, has_dot_leaders,
 };
-use super::classify::{format_list_item, is_caption_line, is_list_item, is_monospace_font};
+use super::classify::{
+    format_list_item, is_caption_line, is_list_item, is_monospace_font, starts_with_bullet_marker,
+};
 use super::postprocess::clean_markdown;
 use super::preprocess::{merge_drop_caps, merge_heading_lines};
 use super::MarkdownOptions;
@@ -584,9 +586,30 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
            .as_ref()
            .and_then(struct_role_heading_level)
            .filter(|level| !overused_heading_levels.contains(level));
+
+        // Protect wrapped list items: when inside a list, a visually-continuing
+        // line (same indent, line-wrap spacing) must not be reclassified as a
+        // heading by the font heuristic — PDFs often bold the lead phrase of a
+        // list item across multiple wrap lines, and an all-bold middle line
+        // would otherwise split one item into a heading + stray body text.
+        // We gate on the document's paragraph threshold so genuine section
+        // headings that follow a numbered paragraph (y_gap > para_threshold)
+        // remain detectable.
+        let looks_like_list_continuation = in_list
+            && match (last_list_x, line.items.first().map(|i| i.x)) {
+                (Some(list_x), Some(curr_x)) => {
+                    let x_ok = curr_x >= list_x - 5.0 && curr_x <= list_x + 50.0;
+                    let y_ok = y_gap >= 0.0 && y_gap <= para_threshold;
+                    x_ok && y_ok && !is_list_item(plain_trimmed)
+                }
+                _ => false,
+            };
+
        let heuristic_heading = if options.detect_headers
+            && !looks_like_list_continuation
            && plain_trimmed.len() > 3
            && plain_trimmed.split_whitespace().count() <= 15
+            && !starts_with_bullet_marker(plain_trimmed)
        {
            let line_font_size = line.items.first().map(|i| i.font_size).unwrap_or(base_size);
            detect_header_level(line_font_size, base_size, &heading_tiers).or_else(|| {
@@ -641,11 +664,16 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
            continue;
        }

-        // Structure-tree list item (LI only — LBody is a continuation, not a new item)
+        // Structure-tree list item (LI only — LBody is a continuation, not a new item).
+        // Some tagged PDFs use a "flat" style where every wrapped line in a list item
+        // gets its own MCID tagged directly under LI. When we're already inside a list
+        // and the line has no visible bullet marker, treat it as a continuation (falls
+        // through to the continuation logic below) rather than a new list item.
        if struct_role
            .as_ref()
            .is_some_and(|r| matches!(r, StructRole::LI))
            && !is_list_item(plain_trimmed)
+            && !in_list
        {
            if in_paragraph {
                output.push_str("\n\n");
@@ -1090,6 +1118,59 @@ mod tests {
        );
    }

+    #[test]
+    fn test_struct_role_li_flat_continuation_lines_merge() {
+        // Regression: some tagged PDFs put each wrapped visual line of a list
+        // item under its own MCID, all tagged directly as LI. Continuation
+        // lines (no bullet marker) must merge into the bulleted parent item,
+        // not each become their own list item.
+        let make = |text: &str, mcid: i64, x: f32, y: f32| {
+            let mut item = make_item(text, 1, Some(mcid));
+            item.x = x;
+            item.y = y;
+            item
+        };
+        let lines = vec![
+            make_line(vec![make("● First item that wraps onto", 0, 90.0, 322.0)]),
+            make_line(vec![make("a continuation line.", 1, 108.0, 306.0)]),
+            make_line(vec![make("● Second bullet also wraps", 2, 90.0, 290.0)]),
+            make_line(vec![make("to a second line here.", 3, 108.0, 274.0)]),
+        ];
+
+        let mut page_roles = HashMap::new();
+        for mcid in 0..4 {
+            page_roles.insert(mcid, StructRole::LI);
+        }
+        let mut roles = HashMap::new();
+        roles.insert(1u32, page_roles);
+
+        let md = to_markdown_from_lines_with_tables_and_images(
+            lines,
+            MarkdownOptions::default(),
+            HashMap::new(),
+            HashMap::new(),
+            &std::collections::HashSet::new(),
+            Some(&roles),
+        );
+
+        assert!(
+            md.contains("- First item that wraps onto a continuation line."),
+            "continuation should merge into first bullet: {md}"
+        );
+        assert!(
+            md.contains("- Second bullet also wraps to a second line here."),
+            "continuation should merge into second bullet: {md}"
+        );
+        assert!(
+            !md.contains("- a continuation line."),
+            "continuation line should not get its own bullet: {md}"
+        );
+        assert!(
+            !md.contains("- to a second line here."),
+            "continuation line should not get its own bullet: {md}"
+        );
+    }
+
    #[test]
    fn test_struct_role_blockquote() {
        let lines = vec![make_line(vec![make_item("Quoted text", 1, Some(0))])];
@@ -1392,4 +1473,70 @@ mod tests {
            overused
        );
    }
+
+    #[test]
+    fn test_wrapped_bold_lead_in_list_item_not_heading() {
+        // Regression: numbered-list items whose bold "lead" phrase wraps onto
+        // a second line (e.g. definitions in system cards) must not have the
+        // wrapped line reclassified as a heading. The middle line is
+        // all_bold + standalone (in_paragraph=false while in_list), which
+        // previously tripped the rarity heuristic and emitted #### in the
+        // middle of the item, splitting the body into stray bullets.
+        let make = |text: &str, x: f32, y: f32, bold: bool| {
+            let mut item = make_item(text, 1, None);
+            item.x = x;
+            item.y = y;
+            item.is_bold = bold;
+            item
+        };
+
+        let lines = vec![
+            // "1. **bold lead phrase start**"
+            make_line(vec![
+                make("1. ", 72.0, 700.0, false),
+                make(
+                    "Chemical and biological weapons threat model 1 (CB-1): Non-novel",
+                    90.0,
+                    700.0,
+                    true,
+                ),
+            ]),
+            // wrapped continuation of the bold lead — all_bold, same indent
+            make_line(vec![make(
+                "chemical/biological weapons production capabilities: A model has CB-1",
+                90.0,
+                686.0,
+                true,
+            )]),
+            // body text of the same list item
+            make_line(vec![make(
+                "capabilities if it has the ability to significantly help individuals.",
+                90.0,
+                672.0,
+                false,
+            )]),
+        ];
+
+        let md = to_markdown_from_lines_with_tables_and_images(
+            lines,
+            MarkdownOptions::default(),
+            HashMap::new(),
+            HashMap::new(),
+            &std::collections::HashSet::new(),
+            None,
+        );
+
+        assert!(
+            !md.contains("#### "),
+            "wrapped bold lead must not become a heading: {md}"
+        );
+        assert!(
+            md.lines().filter(|l| l.starts_with("- ")).count() == 0,
+            "continuation body must not become a stray bullet: {md}"
+        );
+        assert!(
+            md.contains("1. ") && md.contains("A model has CB-1"),
+            "numbered list item should remain intact: {md}"
+        );
+    }
 }
@@ -146,6 +146,67 @@ impl PyPageRegionTexts {
 // Text item wrapper
 // ---------------------------------------------------------------------------

+/// Per-page markdown extraction result.
+#[pyclass(name = "PageMarkdown")]
+#[derive(Clone)]
+pub struct PyPageMarkdown {
+    /// 0-indexed page number.
+    #[pyo3(get)]
+    pub page: u32,
+    /// Formatted markdown for this page.
+    #[pyo3(get)]
+    pub markdown: String,
+    /// True when text on this page is unreliable (GID-encoded fonts,
+    /// encoding issues, garbage text, or empty extraction).
+    #[pyo3(get)]
+    pub needs_ocr: bool,
+}
+
+#[pymethods]
+impl PyPageMarkdown {
+    fn __repr__(&self) -> String {
+        format!(
+            "PageMarkdown(page={}, markdown='{}', needs_ocr={})",
+            self.page,
+            self.markdown.chars().take(40).collect::<String>(),
+            self.needs_ocr
+        )
+    }
+}
+
+/// Combined per-page markdown extraction and layout classification result.
+#[pyclass(name = "PagesExtractionResult")]
+#[derive(Clone)]
+pub struct PyPagesExtractionResult {
+    /// Per-page markdown results, in the order requested.
+    #[pyo3(get)]
+    pub pages: Vec<PyPageMarkdown>,
+    /// 1-indexed pages where tables were detected.
+    #[pyo3(get)]
+    pub pages_with_tables: Vec<u32>,
+    /// 1-indexed pages where multi-column layout was detected.
+    #[pyo3(get)]
+    pub pages_with_columns: Vec<u32>,
+    /// 1-indexed pages that need OCR (scanned/image-based or unreliable text).
+    #[pyo3(get)]
+    pub pages_needing_ocr: Vec<u32>,
+    /// True if any page has tables or columns.
+    #[pyo3(get)]
+    pub is_complex: bool,
+}
+
+#[pymethods]
+impl PyPagesExtractionResult {
+    fn __repr__(&self) -> String {
+        format!(
+            "PagesExtractionResult(pages={}, pages_with_tables={:?}, is_complex={})",
+            self.pages.len(),
+            self.pages_with_tables,
+            self.is_complex
+        )
+    }
+}
+
 /// A positioned text item extracted from a PDF.
 #[pyclass(name = "TextItem")]
 #[derive(Clone)]
@@ -280,6 +341,24 @@ fn parse_page_regions(
        .collect()
 }

+fn to_py_pages_result(r: crate::PagesExtractionResult) -> PyPagesExtractionResult {
+    PyPagesExtractionResult {
+        pages: r
+            .pages
+            .into_iter()
+            .map(|p| PyPageMarkdown {
+                page: p.page,
+                markdown: p.markdown,
+                needs_ocr: p.needs_ocr,
+            })
+            .collect(),
+        pages_with_tables: r.pages_with_tables,
+        pages_with_columns: r.pages_with_columns,
+        pages_needing_ocr: r.pages_needing_ocr,
+        is_complex: r.is_complex,
+    }
+}
+
 fn convert_region_results(results: Vec<crate::PageRegionResult>) -> Vec<PyPageRegionTexts> {
    results
        .into_iter()
@@ -442,6 +521,44 @@ fn extract_text_in_regions_bytes(
    Ok(convert_region_results(results))
 }

+/// Extract formatted markdown for pages of a PDF file, with layout
+/// classification metadata.
+///
+/// Returns per-page markdown and classification data (tables, columns,
+/// OCR needs) from a single parse. Font statistics are computed from the
+/// full document so header detection is consistent across pages.
+///
+/// Args:
+///     path: Path to the PDF file.
+///     pages: Optional list of 0-indexed pages. When None (default), every
+///         page is returned in document order. When provided, output
+///         matches the caller-supplied order.
+///
+/// Returns:
+///     PagesExtractionResult with per-page markdown and classification data.
+#[pyfunction]
+#[pyo3(signature = (path, pages=None))]
+fn extract_pages_markdown(
+    path: &str,
+    pages: Option<Vec<u32>>,
+) -> PyResult<PyPagesExtractionResult> {
+    let result = crate::extract_pages_markdown(path, pages.as_deref()).map_err(to_py_err)?;
+    Ok(to_py_pages_result(result))
+}
+
+/// Extract formatted markdown for pages of a PDF from bytes.
+///
+/// See [`extract_pages_markdown`] for details.
+#[pyfunction]
+#[pyo3(signature = (data, pages=None))]
+fn extract_pages_markdown_bytes(
+    data: &[u8],
+    pages: Option<Vec<u32>>,
+) -> PyResult<PyPagesExtractionResult> {
+    let result = crate::extract_pages_markdown_mem(data, pages.as_deref()).map_err(to_py_err)?;
+    Ok(to_py_pages_result(result))
+}
+
 /// Python module definition.
 #[pymodule]
 fn pdf_inspector(m: &Bound<'_, PyModule>) -> PyResult<()> {
@@ -450,6 +567,8 @@ fn pdf_inspector(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_class::<PyTextItem>()?;
    m.add_class::<PyRegionText>()?;
    m.add_class::<PyPageRegionTexts>()?;
+    m.add_class::<PyPageMarkdown>()?;
+    m.add_class::<PyPagesExtractionResult>()?;
    m.add_function(wrap_pyfunction!(process_pdf, m)?)?;
    m.add_function(wrap_pyfunction!(process_pdf_bytes, m)?)?;
    m.add_function(wrap_pyfunction!(detect_pdf, m)?)?;
@@ -462,5 +581,7 @@ fn pdf_inspector(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(extract_text_with_positions_bytes, m)?)?;
    m.add_function(wrap_pyfunction!(extract_text_in_regions, m)?)?;
    m.add_function(wrap_pyfunction!(extract_text_in_regions_bytes, m)?)?;
+    m.add_function(wrap_pyfunction!(extract_pages_markdown, m)?)?;
+    m.add_function(wrap_pyfunction!(extract_pages_markdown_bytes, m)?)?;
    Ok(())
 }
@@ -581,8 +581,14 @@ fn detect_table_in_region(items: &[(usize, &TextItem)], mode: TableDetectionMode
    // Validation 1: some rows should have content in first column.
    // Use a lower threshold (25%) for tables with wrapped cells where
    // continuation lines leave the first column empty.
+    // Skip when cells form a narrow TOC pattern: hierarchical entries indented
+    // across multiple X levels leave the leftmost column sparse (only top-level
+    // chapters land there) but the structure is still a valid TOC. Narrow only
+    // (<=5 cols) — wide multi-column TOCs (e.g. 2-up indices) would render
+    // poorly through format_toc_as_list, which assumes one entry per row.
    let rows_with_first_col = cells.iter().filter(|row| !row[0].is_empty()).count();
-    if rows_with_first_col < rows.len() / 4 {
+    let is_narrow_toc = columns.len() <= 5 && is_table_of_contents(&cells);
+    if rows_with_first_col < rows.len() / 4 && !is_narrow_toc {
        log::debug!(
            "  validation 1 fail: {}/{} rows have first col",
            rows_with_first_col,
@@ -653,15 +659,24 @@ fn detect_table_in_region(items: &[(usize, &TextItem)], mode: TableDetectionMode
        return None;
    }

-    // Validation 8: Check for Table of Contents pattern
-    if is_table_of_contents(&cells) {
-        log::debug!("  validation 8 fail: table of contents");
+    // Validation 8: Reject paragraph-like content falsely detected as tables.
+    // TOC pages with deep indentation (top-level chapters in col 0, subsections
+    // in cols 1-3, page numbers in last col) leave most cells empty and trip
+    // the paragraph heuristic; TOC shape is a safer signal here. Narrow only
+    // — see narrow-TOC rationale at validation 1.
+    if is_paragraph_content(&cells) && !is_narrow_toc {
+        log::debug!("  validation 9 fail: paragraph content");
        return None;
    }

-    // Validation 9: Reject paragraph-like content falsely detected as tables
-    if is_paragraph_content(&cells) {
-        log::debug!("  validation 9 fail: paragraph content");
+    // Validation 9: Reject wide "index" layouts where every cell carries a
+    // full "label ... page" fragment (back-of-book IRS-style indices).
+    // These render poorly in any structured form; text flow is the best
+    // fallback.  Narrow dot-leader TOCs (2-3 cols) are kept so format.rs
+    // can emit them as a per-row flat list with titles tab-joined to page
+    // numbers.
+    if is_inline_leader_index(&cells) {
+        log::debug!("  validation 9 fail: inline-leader index");
        return None;
    }

@@ -672,12 +687,7 @@ fn detect_table_in_region(items: &[(usize, &TextItem)], mode: TableDetectionMode
        item_indices.len()
    );

-    Some(Table {
-        columns,
-        rows,
-        cells,
-        item_indices,
-    })
+    Some(Table::new(columns, rows, cells, item_indices))
 }

 /// Check if this looks like a key-value pair layout rather than a table
@@ -898,77 +908,247 @@ fn looks_like_number(s: &str) -> bool {
        && s.chars().any(|c| c.is_ascii_digit())
 }

-/// Check if this looks like a Table of Contents
-/// TOCs have characteristic patterns: leader dots, page numbers, section names
-fn is_table_of_contents(cells: &[Vec<String>]) -> bool {
+/// Check if this looks like a Table of Contents (either style).
+///
+/// Used by format.rs to render TOCs as flat lists instead of markdown tables.
+pub fn is_table_of_contents(cells: &[Vec<String>]) -> bool {
+    is_dot_leader_toc(cells) || is_tabular_toc(cells)
+}
+
+/// Dot-leader TOC: any "Chapter 1 ........ 42" style with explicit leader
+/// dots.  Covers both narrow 2-3 col TOCs (where the leader is a dedicated
+/// cell) and wide indices (where each cell encodes a full "label ... page"
+/// fragment).  Used by format.rs to render as a flat list.
+pub(super) fn is_dot_leader_toc(cells: &[Vec<String>]) -> bool {
+    has_structural_dot_leader(cells) || is_inline_leader_index(cells)
+}
+
+/// Rows with a dedicated dots-only cell flanked by label + number (2-3 col
+/// TOC layout).  Format.rs handles these well via per-row flat-list
+/// rendering; they should NOT be rejected at detect time.
+fn has_structural_dot_leader(cells: &[Vec<String>]) -> bool {
    if cells.is_empty() {
        return false;
    }
+    let structural_rows = cells.iter().filter(|row| row_has_dot_leader(row)).count();
+    structural_rows as f32 / cells.len() as f32 >= 0.3
+}

-    let num_cols = cells[0].len();
-    let mut dot_cells = 0;
-    let mut page_number_cells = 0;
-    let mut total_cells = 0;
-    // Track which columns contain dots vs numbers to distinguish
-    // TOC (dots span middle, page number at end) from data tables
-    // (dots only in label column, many number columns).
-    let mut dot_cols = vec![0u32; num_cols];
-    let mut numeric_cols = vec![0u32; num_cols];
-
+/// Wide index layout: each cell holds a full "label ... page" fragment
+/// because the column detector kept multi-column indices as single cells.
+/// These render poorly both as markdown tables (column boundaries are
+/// arbitrary) and as flat lists (each row holds 3+ separate index
+/// entries).  Reject these at detect time so they fall back to the page's
+/// normal text flow.
+pub(super) fn is_inline_leader_index(cells: &[Vec<String>]) -> bool {
+    let mut inline_cells = 0;
+    let mut total_nonempty = 0;
    for row in cells {
-        for (ci, cell) in row.iter().enumerate() {
+        for cell in row {
            let trimmed = cell.trim();
            if trimmed.is_empty() {
                continue;
            }
-            total_cells += 1;
-
-            // Check for leader dots (sequences of periods)
-            // TOCs often have "........" or ". . . ." patterns
-            let dot_count = trimmed.chars().filter(|&c| c == '.').count();
-            let is_mostly_dots = dot_count > trimmed.len() / 2 && dot_count >= 3;
-            if is_mostly_dots {
-                dot_cells += 1;
-                if ci < num_cols {
-                    dot_cols[ci] += 1;
-                }
-            }
-
-            // Check for standalone page numbers (1-4 digits, possibly with spaces)
-            let digits_only: String = trimmed.chars().filter(|c| !c.is_whitespace()).collect();
-            if digits_only.len() <= 4
-                && !digits_only.is_empty()
-                && digits_only.chars().all(|c| c.is_ascii_digit())
-            {
-                page_number_cells += 1;
-                if ci < num_cols {
-                    numeric_cols[ci] += 1;
-                }
+            total_nonempty += 1;
+            if cell_is_inline_leader(trimmed) {
+                inline_cells += 1;
            }
        }
    }
+    total_nonempty >= 4 && inline_cells as f32 / total_nonempty as f32 >= 0.25
+}

-    if total_cells == 0 {
+/// A row with a dot-leader.  Accepts two layouts:
+///   1. A dedicated dots-only cell ("....") with a text label somewhere
+///      to its left and a page number somewhere to its right.
+///   2. A "title ... " cell (trailing leader dots glued to the title)
+///      with a page number elsewhere in the same row.
+fn row_has_dot_leader(row: &[String]) -> bool {
+    let has_page_number = row.iter().any(|c| row_cell_is_page_number(c));
+
+    for (ci, cell) in row.iter().enumerate() {
+        let trimmed = cell.trim();
+
+        // Pattern 1: dedicated dots-only cell.
+        let dot_count = trimmed.chars().filter(|&c| c == '.').count();
+        let is_mostly_dots = dot_count >= 3
+            && dot_count > trimmed.len() / 2
+            && trimmed.chars().all(|c| c == '.' || c.is_whitespace());
+        if is_mostly_dots {
+            let has_label_left = row[..ci].iter().any(|c| {
+                let t = c.trim();
+                !t.is_empty() && t.chars().any(|ch| ch.is_alphabetic())
+            });
+            if has_label_left && has_page_number {
+                return true;
+            }
+            continue;
+        }
+
+        // Pattern 2: cell ends with a trailing " ... " run after a label.
+        if has_page_number && cell_has_trailing_leader(trimmed) {
+            return true;
+        }
+    }
+    false
+}
+
+/// Cell ends with a run of ≥3 dots preceded by alphabetic text and a
+/// space — the "Title ... " layout where the leader is glued to the name.
+/// Alphabetic (not alphanumeric) so that data-table row labels like
+/// "1973 ... " do not register as titles.
+fn cell_has_trailing_leader(cell: &str) -> bool {
+    let trimmed = cell.trim_end();
+    if !trimmed.ends_with('.') {
+        return false;
+    }
+    let without_dots = trimmed.trim_end_matches('.');
+    let dot_run = trimmed.len() - without_dots.len();
+    if dot_run < 3 {
+        return false;
+    }
+    // Require a space before the dot run (rules out "etc..." / "Mr...") and
+    // at least one alphabetic char (rules out "1973 ... " data-row labels).
+    without_dots.ends_with(' ') && without_dots.trim().chars().any(|c| c.is_alphabetic())
+}
+
+/// Page-number shape: single ≤4-digit integer, a ", "-separated list of
+/// ≤4-digit integers ("18, 36, 107"), or a dashed section-page ID
+/// ("A-1", "5-21").  Rejects decimal cells ("4. 0"), thousands-separated
+/// values ("189,164"), and other long numeric data that appears in
+/// statistical tables.
+fn row_cell_is_page_number(cell: &str) -> bool {
+    let t = cell.trim();
+    if t.is_empty() {
+        return false;
+    }
+    if looks_like_section_page_id(t) {
+        return true;
+    }
+    // Page list: ", " separator (with space) distinguishes real page lists
+    // from thousands-separated numbers like "189,164".
+    let parts: Vec<&str> = t.split(", ").collect();
+    parts
+        .iter()
+        .all(|p| !p.is_empty() && p.len() <= 4 && p.chars().all(|c| c.is_ascii_digit()))
+}
+
+/// A cell shaped like an index leader fragment.  Accepts two forms:
+///   - "text ... number" — label + dots + page number in one cell
+///   - "... number"      — bare leader + number (row where the label
+///     landed in a separate column)
+///
+/// Both only count if followed by pure numeric content (optionally
+/// comma-separated page lists like "127, 213").
+fn cell_is_inline_leader(cell: &str) -> bool {
+    let cell = cell.trim();
+
+    // Find the first "..." run.  Surrounding-whitespace checks below
+    // reject intra-word ellipses ("etc...").
+    let idx = match cell.match_indices("...").next() {
+        Some((i, _)) => i,
+        None => return false,
+    };
+
+    let before = &cell[..idx];
+    let after_dots = &cell[idx + 3..];
+    // Allow extra dots (e.g. "....") by skipping any additional '.'
+    let after = after_dots.trim_start_matches('.');
+
+    // Require space (or start-of-cell) before the dots and space/digit
+    // after — blocks intra-word ellipses.
+    let before_ok = before.is_empty() || before.ends_with(' ');
+    let after_ok = after.starts_with(' ') || after.is_empty();
+    if !before_ok || !after_ok {
        return false;
    }

-    // Data tables with dot leaders (e.g. "1973....") have dots concentrated
-    // in one column (the label column) while many other columns contain numbers.
-    // True TOCs have dots spanning the middle and one page-number column at the end.
-    // If dots are confined to ≤1 column AND there are ≥3 columns with numbers,
-    // this is a data table, not a TOC.
-    let cols_with_dots = dot_cols.iter().filter(|&&c| c >= 2).count();
-    let cols_with_numbers = numeric_cols.iter().filter(|&&c| c >= 2).count();
-    if cols_with_dots <= 1 && cols_with_numbers >= 3 {
+    let after_trim = after.trim();
+    if after_trim.is_empty() {
+        return false;
+    }
+    // Tail must be purely numeric/page-list content.
+    let tail_numeric = after_trim
+        .chars()
+        .all(|c| c.is_ascii_digit() || matches!(c, ',' | ' ' | '.' | '-' | '$'))
+        && after_trim.chars().any(|c| c.is_ascii_digit());
+    if !tail_numeric {
        return false;
    }

-    // If a significant portion of cells are dots or page numbers, it's likely a TOC
-    let dot_ratio = dot_cells as f32 / total_cells as f32;
-    let page_num_ratio = page_number_cells as f32 / total_cells as f32;
+    // Either we have a label before, or the leader is bare (starts the cell)
+    // — both are legitimate index fragments.
+    before.chars().any(|c| c.is_alphabetic()) || before.trim().is_empty()
+}

-    // TOC typically has >15% dot cells and >10% page number cells
-    dot_ratio > 0.15 || (dot_ratio > 0.05 && page_num_ratio > 0.15)
+/// Dot-less tabular TOC: tagged PDFs emit entries as rows where the first
+/// column starts with a dotted section number (e.g. "4.3.1 Something") and
+/// the last column is one or more page numbers.  These have no leader dots
+/// and benefit from flat-list formatting (page numbers aligned to titles).
+pub(super) fn is_tabular_toc(cells: &[Vec<String>]) -> bool {
+    if cells.is_empty() {
+        return false;
+    }
+    let num_cols = cells[0].len();
+    if num_cols < 2 || cells.len() < 4 {
+        return false;
+    }
+
+    let section_rows = cells
+        .iter()
+        .filter(|row| {
+            row.iter()
+                .find(|c| !c.trim().is_empty())
+                .is_some_and(|c| starts_with_section_number(c.trim()))
+        })
+        .count();
+
+    let last_col = num_cols - 1;
+    let (last_filled, last_page_num) = cells.iter().fold((0u32, 0u32), |(f, n), row| {
+        let cell = row.get(last_col).map(|s| s.trim()).unwrap_or("");
+        if cell.is_empty() {
+            return (f, n);
+        }
+        let is_page_nums = cell
+            .split_whitespace()
+            .all(|tok| !tok.is_empty() && tok.chars().all(|c| c.is_ascii_digit()));
+        (f + 1, n + if is_page_nums { 1 } else { 0 })
+    });
+
+    let section_ratio = section_rows as f32 / cells.len() as f32;
+    let page_num_last_ratio = if last_filled > 0 {
+        last_page_num as f32 / last_filled as f32
+    } else {
+        0.0
+    };
+
+    section_ratio >= 0.6 && last_filled >= 3 && page_num_last_ratio >= 0.7
+}
+
+/// Matches dashed section-page identifiers used in technical manuals:
+/// "5-21", "A-1", "B--3", "TC-2".  At least one ASCII digit is required.
+fn looks_like_section_page_id(s: &str) -> bool {
+    let ok = s
+        .chars()
+        .all(|c| c.is_ascii_digit() || c.is_ascii_uppercase() || c == '-');
+    ok && s.chars().any(|c| c.is_ascii_digit())
+}
+
+/// Returns true when the leading token looks like a dotted section number:
+/// "1", "1.2", "1.2.3", "4.3.1.2" — integer components joined by dots,
+/// with at least one dot (single-number prefixes are too ambiguous).
+fn starts_with_section_number(s: &str) -> bool {
+    let Some(first) = s.split_whitespace().next() else {
+        return false;
+    };
+    let first = first.trim_end_matches('.');
+    let parts: Vec<&str> = first.split('.').collect();
+    if parts.len() < 2 || parts.len() > 6 {
+        return false;
+    }
+    parts
+        .iter()
+        .all(|p| !p.is_empty() && p.len() <= 3 && p.chars().all(|c| c.is_ascii_digit()))
 }

 /// Check if detected "table" cells are actually paragraph text fragments.
@@ -1425,4 +1605,283 @@ mod tests {
            "data table with dot-leader labels should not be rejected as TOC"
        );
    }
+
+    #[test]
+    fn is_table_of_contents_accepts_hierarchical_indented_toc() {
+        // Mythos system card pages 4-5: top-level chapters indent at col 0,
+        // subsections at cols 1-2, leaving col 0 mostly empty (only ~10% of
+        // rows). Validation 1 was rejecting these even though the structure
+        // is unambiguously a TOC.
+        let cells = vec![
+            vec!["Abstract".to_string(), String::new(), "3".to_string()],
+            vec![
+                "1 Introduction".to_string(),
+                String::new(),
+                "10".to_string(),
+            ],
+            vec![
+                String::new(),
+                "1.1 Model training".to_string(),
+                "11".to_string(),
+            ],
+            vec![
+                String::new(),
+                "1.1.1 Training data".to_string(),
+                "11".to_string(),
+            ],
+            vec![
+                String::new(),
+                "1.1.2 Crowd workers".to_string(),
+                "12".to_string(),
+            ],
+            vec![
+                String::new(),
+                "1.2 Release decision".to_string(),
+                "13".to_string(),
+            ],
+            vec![
+                "2 RSP evaluations".to_string(),
+                String::new(),
+                "16".to_string(),
+            ],
+            vec![
+                String::new(),
+                "2.1 RSP risk assessment".to_string(),
+                "16".to_string(),
+            ],
+            vec![String::new(), "2.1.1 Context".to_string(), "16".to_string()],
+            vec![
+                String::new(),
+                "2.2 CB evaluations".to_string(),
+                "20".to_string(),
+            ],
+        ];
+        assert!(
+            is_table_of_contents(&cells),
+            "hierarchical TOC with sparse col 0 should still be detected"
+        );
+    }
+
+    #[test]
+    fn is_table_of_contents_rejects_dotless_toc() {
+        // Tabular TOC without leader dots: first column starts with dotted
+        // section numbers, last column is page numbers.  Pattern from
+        // Mythos system card pages 6-8.
+        let cells = vec![
+            vec![
+                "4.3 Case studies and targeted evaluations".to_string(),
+                String::new(),
+                "86".to_string(),
+            ],
+            vec![
+                "4.3.1 Destructive or reckless actions".to_string(),
+                "4.3.1.1 Synthetic-backend evaluation".to_string(),
+                "86 86".to_string(),
+            ],
+            vec![
+                "4.3.2 Adherence to constitution".to_string(),
+                "4.3.2.1 Overview".to_string(),
+                "89 89".to_string(),
+            ],
+            vec![
+                "4.3.3 Honesty and hallucinations".to_string(),
+                "4.3.3.1 Factual hallucinations".to_string(),
+                "93 94".to_string(),
+            ],
+            vec![
+                "4.4 Capability evaluations".to_string(),
+                String::new(),
+                "101".to_string(),
+            ],
+        ];
+        assert!(
+            is_table_of_contents(&cells),
+            "dot-less TOC with section numbers + page numbers should be rejected"
+        );
+    }
+
+    #[test]
+    fn dot_leader_toc_accepts_short_inline_leaders() {
+        // Index-style cells where the full "label ... number" pattern is
+        // preserved in a single cell (IRS Publication 17 back-of-book index).
+        let cells = vec![
+            vec!["Child tax credit ... 235".to_string(), String::new()],
+            vec!["Church employee ... 252".to_string(), String::new()],
+            vec!["Citizens outside the U.S ... 6".to_string(), String::new()],
+            vec![
+                "Claim for refund ... 18, 36, 107".to_string(),
+                String::new(),
+            ],
+            vec!["Clergy ... 7, 52".to_string(), String::new()],
+        ];
+        assert!(is_dot_leader_toc(&cells));
+    }
+
+    #[test]
+    fn dot_leader_toc_allows_ellipsis_data_table() {
+        // Data tables using "..." as a row-omission marker must not be
+        // mistaken for dot-leader TOCs.  Based on MCF5235RM QSPI RAM layout.
+        let cells = vec![
+            vec![
+                "0x00".to_string(),
+                "QTR0".to_string(),
+                "Transmit RAM".to_string(),
+            ],
+            vec!["0x01".to_string(), "QTR1".to_string(), String::new()],
+            vec![
+                "...".to_string(),
+                "...".to_string(),
+                "16 bits wide".to_string(),
+            ],
+            vec!["0x0F".to_string(), "QTR15".to_string(), String::new()],
+            vec![
+                "0x10".to_string(),
+                "QRR0".to_string(),
+                "Receive RAM".to_string(),
+            ],
+            vec!["0x11".to_string(), "QRR1".to_string(), String::new()],
+            vec![
+                "...".to_string(),
+                "...".to_string(),
+                "16 bits wide".to_string(),
+            ],
+            vec!["0x1F".to_string(), "QRR15".to_string(), String::new()],
+        ];
+        assert!(
+            !is_dot_leader_toc(&cells),
+            "ellipsis markers in a data table should not match TOC detection"
+        );
+    }
+
+    #[test]
+    fn dot_leader_toc_rejects_year_row_data_table() {
+        // ERP-2025 economic data tables: year labels with trailing " ... ",
+        // a final " ... " column, and decimal-looking numeric cells.  The
+        // detection previously classified these as dot-leader TOCs and
+        // routed them through flat-list formatting, destroying the grid.
+        let cells = vec![
+            vec![
+                "1973 ... ".to_string(),
+                "4. 0".to_string(),
+                "1. 8".to_string(),
+                "0. 4".to_string(),
+                "3. 2".to_string(),
+                " ... ".to_string(),
+            ],
+            vec![
+                "1974 ... ".to_string(),
+                "–1. 9".to_string(),
+                "–1. 6".to_string(),
+                "–5. 6".to_string(),
+                "2. 4".to_string(),
+                " ... ".to_string(),
+            ],
+            vec![
+                "1975 ... ".to_string(),
+                "2. 6".to_string(),
+                "5. 1".to_string(),
+                "6. 1".to_string(),
+                "4. 1".to_string(),
+                " ... ".to_string(),
+            ],
+            vec![
+                "1976 ... ".to_string(),
+                "4. 3".to_string(),
+                "5. 4".to_string(),
+                "6. 4".to_string(),
+                "4. 5".to_string(),
+                " ... ".to_string(),
+            ],
+        ];
+        assert!(
+            !is_dot_leader_toc(&cells),
+            "year-indexed data tables with decimal cells must not match TOC detection"
+        );
+    }
+
+    #[test]
+    fn dot_leader_toc_rejects_monthly_data_table() {
+        // ERP-2025 Table B-22: monthly labor-force rows with "Jan ... ",
+        // "Feb ... " labels and thousands-separated cells ("189,164").
+        // Previously matched TOC detection because "Jan ..." has alphabetic
+        // text and "189,164" passed the page-number shape check.
+        let cells = vec![
+            vec![
+                "2023: Jan ... ".to_string(),
+                "265,962".to_string(),
+                "165,871".to_string(),
+                "160,152".to_string(),
+                "62. 4".to_string(),
+            ],
+            vec![
+                "Feb ... ".to_string(),
+                "266,112".to_string(),
+                "166,263".to_string(),
+                "160,301".to_string(),
+                "62. 5".to_string(),
+            ],
+            vec![
+                "Mar ... ".to_string(),
+                "266,272".to_string(),
+                "166,690".to_string(),
+                "160,824".to_string(),
+                "62. 6".to_string(),
+            ],
+            vec![
+                "Apr ... ".to_string(),
+                "266,443".to_string(),
+                "166,678".to_string(),
+                "160,962".to_string(),
+                "62. 6".to_string(),
+            ],
+        ];
+        assert!(
+            !is_dot_leader_toc(&cells),
+            "monthly labor-force rows with thousands-separated data must not match TOC detection"
+        );
+    }
+
+    #[test]
+    fn tabular_toc_requires_section_numbers_and_pages() {
+        // Dot-less tabular TOC matches is_tabular_toc but not dot-leader.
+        let cells = vec![
+            vec![
+                "4.3 Case studies".to_string(),
+                String::new(),
+                "86".to_string(),
+            ],
+            vec![
+                "4.3.1 Destructive actions".to_string(),
+                String::new(),
+                "86".to_string(),
+            ],
+            vec![
+                "4.3.2 Adherence".to_string(),
+                String::new(),
+                "89".to_string(),
+            ],
+            vec!["4.3.3 Honesty".to_string(), String::new(), "93".to_string()],
+        ];
+        assert!(is_tabular_toc(&cells));
+        assert!(!is_dot_leader_toc(&cells));
+    }
+
+    #[test]
+    fn starts_with_section_number_matches_dotted() {
+        assert!(starts_with_section_number("1.2"));
+        assert!(starts_with_section_number("4.3.1"));
+        assert!(starts_with_section_number("4.3.1.2"));
+        assert!(starts_with_section_number("4.3 Case studies"));
+        assert!(starts_with_section_number("2.2.5.1 Expert red teaming"));
+    }
+
+    #[test]
+    fn starts_with_section_number_rejects_non_sections() {
+        assert!(!starts_with_section_number("Chapter 1"));
+        assert!(!starts_with_section_number("1973"));
+        assert!(!starts_with_section_number("1.5M"));
+        assert!(!starts_with_section_number("10.0%"));
+        assert!(!starts_with_section_number(""));
+        assert!(!starts_with_section_number("Hello world"));
+    }
 }
@@ -265,12 +265,12 @@ pub fn detect_tables_from_lines(items: &[TextItem], lines: &[PdfLine], page: u32
        page, num_rows, num_cols, item_indices.len(), page_item_count, non_empty_rows, cols_with_content
    );

-    vec![Table {
-        columns: col_edges,
-        rows: row_edges_desc[..num_rows].to_vec(),
+    vec![Table::new(
+        col_edges,
+        row_edges_desc[..num_rows].to_vec(),
        cells,
        item_indices,
-    }]
+    )]
 }

 #[cfg(test)]
@@ -1037,12 +1037,7 @@ fn try_build_grid(
        (columns, cells)
    };

-    GridResult::Ok(Table {
-        columns,
-        rows,
-        cells,
-        item_indices,
-    })
+    GridResult::Ok(Table::new(columns, rows, cells, item_indices))
 }

 /// Deduplicate nearby edge values within a tolerance, returning sorted unique edges.
@@ -1442,12 +1437,7 @@ fn detect_row_stripe_table(
        content_ratio * 100.0
    );

-    Some(Table {
-        columns: column_centers,
-        rows: row_centers,
-        cells,
-        item_indices,
-    })
+    Some(Table::new(column_centers, row_centers, cells, item_indices))
 }

 /// Detect a table from cell-background rects that failed grid detection.
@@ -1691,12 +1681,7 @@ fn detect_row_stripe_table_from_cell_rects(
        non_empty_cells as f32 / total_cells * 100.0
    );

-    Some(Table {
-        columns: column_centers,
-        rows: row_centers,
-        cells,
-        item_indices,
-    })
+    Some(Table::new(column_centers, row_centers, cells, item_indices))
 }

 /// Detect a table by merging all cluster rects into one group.
@@ -1875,12 +1860,7 @@ fn detect_merged_cluster_table(
        content_ratio * 100.0
    );

-    Some(Table {
-        columns: column_centers,
-        rows: row_centers,
-        cells,
-        item_indices,
-    })
+    Some(Table::new(column_centers, row_centers, cells, item_indices))
 }

 /// Cluster text item X positions into column centers with a given minimum threshold.
@@ -188,12 +188,12 @@ pub fn detect_tables_from_struct_tree(
        all_item_indices.sort_unstable();
        all_item_indices.dedup();

-        tables.push(Table {
-            columns: col_positions,
-            rows: row_positions,
+        tables.push(Table::new(
+            col_positions,
+            row_positions,
            cells,
-            item_indices: all_item_indices,
-        });
+            all_item_indices,
+        ));
    }

    tables
@@ -1,12 +1,22 @@
 //! Table-to-markdown formatting and cell cleanup.

-use super::Table;
+use super::{Table, TableKind};

 pub fn table_to_markdown(table: &Table) -> String {
    if table.cells.is_empty() || table.cells[0].is_empty() {
        return String::new();
    }

+    // TOCs render poorly as markdown tables — emit a flat per-row text list
+    // instead so the page numbers stay aligned with their section titles
+    // rather than drifting to a separate column. Format from raw cells
+    // because continuation-row merging in clean_table_cells collapses
+    // separate TOC entries (e.g. "6.2 Contamination" + "6.2.1 SWE-bench")
+    // into one line where sub-entries leave column 0 empty.
+    if table.kind == TableKind::Toc {
+        return format_toc_as_list(&table.cells, &[]);
+    }
+
    // Clean up the table: merge continuation rows, extract footnotes, remove empty rows
    let (cleaned_cells, footnotes) = clean_table_cells(&table.cells);

@@ -49,6 +59,101 @@ pub fn table_to_markdown(table: &Table) -> String {
    output
 }

+/// Render a table-of-contents as a flat per-row text block.
+///
+/// Each row becomes one line: non-empty cells joined with spaces, and the
+/// last cell (typically a page number) is separated by a tab so the page
+/// numbers stay aligned with their titles instead of being pulled into a
+/// separate column by the column-aware reader.
+fn format_toc_as_list(cells: &[Vec<String>], footnotes: &[String]) -> String {
+    let mut output = String::new();
+
+    for row in cells {
+        let trimmed: Vec<&str> = row.iter().map(|c| c.trim()).collect();
+        let last_idx = trimmed.iter().rposition(|c| !c.is_empty());
+        let Some(last_idx) = last_idx else {
+            continue;
+        };
+
+        let last_cell = trimmed[last_idx];
+        let last_is_page = is_page_number_cell(last_cell);
+
+        let (title_cells, trailing) = if last_is_page && last_idx > 0 {
+            (&trimmed[..last_idx], Some(last_cell))
+        } else {
+            (&trimmed[..=last_idx], None)
+        };
+
+        // Skip dots-only cells when joining the title — in a detected TOC
+        // layout, a "...." cell is a leader separator, not part of the
+        // entry name.
+        let title = title_cells
+            .iter()
+            .filter(|c| !c.is_empty() && !is_dots_only(c))
+            .copied()
+            .collect::<Vec<_>>()
+            .join(" ");
+
+        if title.is_empty() && trailing.is_none() {
+            continue;
+        }
+
+        if !title.is_empty() {
+            output.push_str(&title);
+        }
+        if let Some(page) = trailing {
+            if !title.is_empty() {
+                output.push('\t');
+            }
+            output.push_str(page);
+        }
+        output.push('\n');
+    }
+
+    if !footnotes.is_empty() {
+        output.push('\n');
+        for footnote in footnotes {
+            output.push_str(footnote);
+            output.push('\n');
+        }
+    }
+
+    output
+}
+
+/// True when the cell looks like a page number.  Accepts:
+///   - plain digit tokens: "42", "86 86"
+///   - dashed section-page IDs: "5-21", "A-1", "B--3", "TC-2" (common in
+///     technical manuals)
+fn is_page_number_cell(cell: &str) -> bool {
+    let tokens: Vec<&str> = cell.split_whitespace().collect();
+    if tokens.is_empty() {
+        return false;
+    }
+    tokens.iter().all(|t| {
+        if t.is_empty() || t.len() > 8 {
+            return false;
+        }
+        let all_digits = t.chars().all(|c| c.is_ascii_digit());
+        if all_digits {
+            return t.len() <= 4;
+        }
+        // Section-page form: uppercase letters, digits, dashes; at least
+        // one digit present.
+        t.chars()
+            .all(|c| c.is_ascii_digit() || c.is_ascii_uppercase() || c == '-')
+            && t.chars().any(|c| c.is_ascii_digit())
+    })
+}
+
+/// True when the cell is purely leader dots (any length ≥ 3) with optional
+/// whitespace.
+fn is_dots_only(cell: &str) -> bool {
+    let t = cell.trim();
+    let dots = t.chars().filter(|&c| c == '.').count();
+    dots >= 3 && t.chars().all(|c| c == '.' || c.is_whitespace())
+}
+
 /// Clean up table cells: merge continuation rows, extract footnotes, remove empty rows
 fn clean_table_cells(cells: &[Vec<String>]) -> (Vec<Vec<String>>, Vec<String>) {
    let mut cleaned: Vec<Vec<String>> = Vec::new();
@@ -358,6 +463,7 @@ mod tests {
                vec!["Bob".into(), "25".into()],
            ],
            item_indices: vec![],
+            kind: TableKind::Data,
        };
        let md = table_to_markdown(&table);
        assert!(md.contains("|Name|"));
@@ -373,6 +479,7 @@ mod tests {
            rows: vec![500.0],
            cells: vec![vec!["Only".into(), "Row".into()]],
            item_indices: vec![],
+            kind: TableKind::Data,
        };
        let md = table_to_markdown(&table);
        assert!(md.contains("|Only|"));
@@ -386,6 +493,7 @@ mod tests {
            rows: vec![],
            cells: vec![],
            item_indices: vec![],
+            kind: TableKind::Data,
        };
        assert_eq!(table_to_markdown(&table), "");
    }
@@ -401,6 +509,7 @@ mod tests {
                vec!["(1)".into(), "Footnote text".into()],
            ],
            item_indices: vec![],
+            kind: TableKind::Data,
        };
        let md = table_to_markdown(&table);
        assert!(md.contains("(1) Footnote text"));
@@ -416,6 +525,7 @@ mod tests {
                vec!["太郎".into(), "25".into()],
            ],
            item_indices: vec![],
+            kind: TableKind::Data,
        };
        let md = table_to_markdown(&table);
        assert!(md.contains("名前"));
@@ -429,7 +539,47 @@ mod tests {
            rows: vec![500.0],
            cells: vec![vec![]],
            item_indices: vec![],
+            kind: TableKind::Data,
        };
        assert_eq!(table_to_markdown(&table), "");
    }
+
+    #[test]
+    fn test_table_to_markdown_toc_renders_as_flat_list() {
+        // A TOC-shaped table with section numbers in col 0 and page numbers
+        // in the last column should render as a flat list, not a markdown
+        // table, so the page numbers stay on the same line as their titles.
+        let table = Table::new(
+            vec![50.0, 80.0, 300.0],
+            vec![500.0; 5],
+            vec![
+                vec![
+                    "4.3".into(),
+                    "Case studies and targeted evaluations".into(),
+                    "86".into(),
+                ],
+                vec![
+                    "4.3.1".into(),
+                    "Destructive or reckless actions".into(),
+                    "86".into(),
+                ],
+                vec![
+                    "4.3.2".into(),
+                    "Adherence to its constitution".into(),
+                    "89".into(),
+                ],
+                vec!["4.4".into(), "Capability evaluations".into(), "101".into()],
+                vec!["4.5".into(), "White-box analyses".into(), "113".into()],
+            ],
+            vec![],
+        );
+        assert_eq!(table.kind, TableKind::Toc);
+        let md = table_to_markdown(&table);
+        assert!(
+            !md.contains("|---|"),
+            "TOC should not render as a markdown table: {md}"
+        );
+        assert!(md.contains("4.3 Case studies and targeted evaluations\t86"));
+        assert!(md.contains("4.5 White-box analyses\t113"));
+    }
 }
@@ -499,6 +499,7 @@ pub(crate) fn recover_header_row(
 #[cfg(test)]
 mod tests {
    use super::*;
+    use crate::tables::TableKind;
    use crate::types::ItemType;

    fn make_item(text: &str, x: f32, y: f32, font_size: f32) -> TextItem {
@@ -756,6 +757,7 @@ mod tests {
            rows: vec![500.0, 480.0],
            cells: vec![vec!["A".into(), "B".into()], vec!["C".into(), "D".into()]],
            item_indices: vec![2, 3],
+            kind: TableKind::Data,
        };

        recover_header_row(&mut table, &all_items, 9.0);
@@ -774,6 +776,7 @@ mod tests {
            rows: vec![500.0],
            cells: vec![vec!["A".into(), "B".into()]],
            item_indices: vec![0, 1],
+            kind: TableKind::Data,
        };

        let rows_before = table.rows.len();
@@ -794,6 +797,7 @@ mod tests {
            rows: vec![500.0, 480.0],
            cells: vec![vec!["A".into(), "B".into()], vec!["C".into(), "D".into()]],
            item_indices: vec![2, 3],
+            kind: TableKind::Data,
        };

        let rows_before = table.rows.len();
@@ -814,6 +818,7 @@ mod tests {
            rows: vec![500.0],
            cells: vec![vec!["A".into(), "B".into()]],
            item_indices: vec![1, 2],
+            kind: TableKind::Data,
        };

        let rows_before = table.rows.len();
@@ -829,6 +834,7 @@ mod tests {
            rows: vec![],
            cells: vec![],
            item_indices: vec![],
+            kind: TableKind::Data,
        };

        recover_header_row(&mut table, &all_items, 9.0);
@@ -11,6 +11,7 @@ mod format;
 mod grid;

 pub use detect_heuristic::detect_tables;
+pub(crate) use detect_heuristic::is_table_of_contents;
 pub use detect_lines::detect_tables_from_lines;
 pub(crate) use detect_rects::cluster_rects;
 pub use detect_rects::{detect_tables_from_rects, RectHintRegion};
@@ -166,12 +167,12 @@ pub(crate) fn try_build_rect_guided_table(
    used_indices.sort_unstable();
    used_indices.dedup();

-    Some(Table {
-        columns: col_boundaries,
-        rows: row_boundaries,
+    Some(Table::new(
+        col_boundaries,
+        row_boundaries,
        cells,
-        item_indices: used_indices,
-    })
+        used_indices,
+    ))
 }

 /// Split a TextItem whose text contains multiple whitespace-separated tokens
@@ -541,12 +542,22 @@ pub(crate) fn try_build_table_from_columns(items: &[TextItem], page: u32) -> Opt
        multi_col_rows
    );

-    Some(Table {
-        columns: col_xs,
-        rows: row_ys,
-        cells,
-        item_indices,
-    })
+    Some(Table::new(col_xs, row_ys, cells, item_indices))
+}
+
+/// What kind of structure a detected `Table` represents. Classification is
+/// computed once at construction so consumers don't have to re-analyze the
+/// cells (and stay consistent across detection backends).
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum TableKind {
+    /// A real data table — renders as markdown table syntax.
+    #[default]
+    Data,
+    /// A table of contents — renders as a flat list with tab-aligned page
+    /// numbers via `format_toc_as_list`. Detected through the table pipeline
+    /// because TOCs share row/column structure with tables, but they are not
+    /// data tables and shouldn't appear in `pages_with_tables` etc.
+    Toc,
 }

 /// A detected table.
@@ -560,6 +571,31 @@ pub struct Table {
    pub cells: Vec<Vec<String>>,
    /// Items that belong to this table
    pub item_indices: Vec<usize>,
+    /// Data table vs TOC. Set by `Table::new` from `cells`.
+    pub kind: TableKind,
+}
+
+impl Table {
+    /// Build a table and classify it (data vs TOC) from its cells.
+    pub fn new(
+        columns: Vec<f32>,
+        rows: Vec<f32>,
+        cells: Vec<Vec<String>>,
+        item_indices: Vec<usize>,
+    ) -> Self {
+        let kind = if is_table_of_contents(&cells) {
+            TableKind::Toc
+        } else {
+            TableKind::Data
+        };
+        Self {
+            columns,
+            rows,
+            cells,
+            item_indices,
+            kind,
+        }
+    }
 }

 #[cfg(test)]
@@ -642,6 +678,7 @@ mod tests {
                vec!["Cell 1".into(), "Cell 2".into()],
            ],
            item_indices: vec![],
+            kind: TableKind::Data,
        };

        let md = table_to_markdown(&table);
@@ -1002,6 +1039,7 @@ mod tests {
                vec!["3".into(), "5/2".into(), "Item C".into(), "300".into()],
            ],
            item_indices: vec![],
+            kind: TableKind::Data,
        };

        let md = table_to_markdown(&table);
@@ -520,6 +520,18 @@ impl ToUnicodeCMap {
        }
    }

+    /// Get the maximum source CID across all mappings (char_map + ranges).
+    fn max_source_cid(&self) -> Option<u16> {
+        let char_max = self.char_map.keys().copied().max();
+        let range_max = self.ranges.iter().map(|&(_, end, _)| end).max();
+        match (char_max, range_max) {
+            (Some(a), Some(b)) => Some(a.max(b)),
+            (a @ Some(_), None) => a,
+            (None, b @ Some(_)) => b,
+            (None, None) => None,
+        }
+    }
+
    /// Remap a CMap that references pre-subsetting GIDs to sequential post-subsetting GIDs.
    /// Collects all source CIDs, sorts them, and reassigns to 1, 2, 3, ...
    pub fn remap_to_sequential(&self) -> ToUnicodeCMap {
@@ -657,6 +669,81 @@ fn get_w_array_start_cid(cid_font_dict: &lopdf::Dictionary, doc: &Document) -> O
    }
 }

+/// Return true if the CIDFont's W (widths) array explicitly covers the given CID.
+///
+/// The W array uses two formats (PDF 32000-1:2008, §9.7.4.3):
+///   1. `c [w1 w2 ... wn]` — widths for CIDs c, c+1, ..., c+n-1
+///   2. `c_first c_last w` — CIDs c_first..c_last all have width w
+fn w_array_covers_cid(cid_font_dict: &lopdf::Dictionary, doc: &Document, target: u16) -> bool {
+    let Ok(w_obj) = cid_font_dict.get(b"W") else {
+        return false;
+    };
+    let arr = match w_obj {
+        Object::Array(arr) => arr,
+        Object::Reference(r) => match doc.get_object(*r) {
+            Ok(Object::Array(arr)) => arr,
+            _ => return false,
+        },
+        _ => return false,
+    };
+
+    let resolve_int = |o: &Object| -> Option<i64> {
+        match o {
+            Object::Integer(n) => Some(*n),
+            Object::Reference(r) => match doc.get_object(*r) {
+                Ok(Object::Integer(n)) => Some(*n),
+                _ => None,
+            },
+            _ => None,
+        }
+    };
+
+    let resolve_arr = |o: &Object| -> Option<Vec<Object>> {
+        match o {
+            Object::Array(a) => Some(a.clone()),
+            Object::Reference(r) => match doc.get_object(*r) {
+                Ok(Object::Array(a)) => Some(a.clone()),
+                _ => None,
+            },
+            _ => None,
+        }
+    };
+
+    let target = target as i64;
+    let mut i = 0usize;
+    while i < arr.len() {
+        let Some(first) = resolve_int(&arr[i]) else {
+            break;
+        };
+        i += 1;
+        if i >= arr.len() {
+            break;
+        }
+        // Peek at arr[i] to decide format.
+        if let Some(widths) = resolve_arr(&arr[i]) {
+            // Format 1: c [w1 ... wn]
+            let last = first + widths.len() as i64 - 1;
+            if target >= first && target <= last {
+                return true;
+            }
+            i += 1;
+        } else if let Some(last) = resolve_int(&arr[i]) {
+            // Format 2: c_first c_last w
+            i += 1;
+            if i < arr.len() {
+                i += 1; // skip the width value
+            }
+            if target >= first && target <= last {
+                return true;
+            }
+        } else {
+            // Unknown token — abort parsing safely
+            break;
+        }
+    }
+    false
+}
+
 /// Extract CIDToGIDMap as a vector of GIDs (u16) indexed by CID.
 fn get_cid_to_gid_map(cid_font_dict: &lopdf::Dictionary, doc: &Document) -> Option<Vec<u16>> {
    let obj = cid_font_dict.get(b"CIDToGIDMap").ok()?;
@@ -752,6 +839,20 @@ fn try_remap_subset_cmap(
        _ => return (cmap, None),
    };

+    // If the W array actually covers the CMap's max source CID, the CMap is
+    // aligned with the font — no sequential renumbering happened. A sparse W
+    // array starting at CID 0 (for .notdef) with additional high-CID entries
+    // matching the CMap is the normal subset layout, not a mismatch.
+    if let Some(max_cid) = cmap.max_source_cid() {
+        if w_array_covers_cid(cid_font_dict, doc, max_cid) {
+            debug!(
+                "Subset remap skipped for obj={}: W array covers CMap max CID {}",
+                obj_num, max_cid
+            );
+            return (cmap, None);
+        }
+    }
+
    debug!(
        "Subset GID mismatch detected for obj={}: W starts at CID {}, CMap min CID {}. Remapping to sequential.",
        obj_num, w_start, min_cid
@@ -2717,4 +2818,187 @@ endbfchar
        assert_eq!(remapped.unwrap().char_map.len(), 50);
        assert_eq!(fallback.unwrap().char_map.len(), 10);
    }
+
+    #[test]
+    fn test_max_source_cid() {
+        let cmap_content = r#"
+1 begincodespacerange
+<0000><FFFF>
+endcodespacerange
+2 beginbfchar
+<0003> <0020>
+<0031> <004E>
+endbfchar
+1 beginbfrange
+<0208> <0227> <0430>
+endbfrange
+"#;
+        let cmap = ToUnicodeCMap::parse(cmap_content.as_bytes()).unwrap();
+        assert_eq!(cmap.min_source_cid(), Some(0x0003));
+        assert_eq!(cmap.max_source_cid(), Some(0x0227));
+    }
+
+    /// Helper: build a minimal CIDFont dict with a W array and check coverage.
+    fn cid_font_dict_with_w(w_items: Vec<lopdf::Object>) -> lopdf::Dictionary {
+        let mut d = lopdf::Dictionary::new();
+        d.set("W", lopdf::Object::Array(w_items));
+        d
+    }
+
+    #[test]
+    fn test_w_array_covers_cid_format1() {
+        // Format 1: `c [w1 w2 ... wn]` — widths for CIDs c..c+n-1.
+        // Mimics the 16.pdf Tahoma W array: 0[1000] 3[313] 5[401] 11[383 383] 16[363 303 382]
+        let doc = Document::new();
+        let d = cid_font_dict_with_w(vec![
+            lopdf::Object::Integer(0),
+            lopdf::Object::Array(vec![lopdf::Object::Integer(1000)]),
+            lopdf::Object::Integer(3),
+            lopdf::Object::Array(vec![lopdf::Object::Integer(313)]),
+            lopdf::Object::Integer(5),
+            lopdf::Object::Array(vec![lopdf::Object::Integer(401)]),
+            lopdf::Object::Integer(11),
+            lopdf::Object::Array(vec![
+                lopdf::Object::Integer(383),
+                lopdf::Object::Integer(383),
+            ]),
+            lopdf::Object::Integer(16),
+            lopdf::Object::Array(vec![
+                lopdf::Object::Integer(363),
+                lopdf::Object::Integer(303),
+                lopdf::Object::Integer(382),
+            ]),
+            lopdf::Object::Integer(570),
+            lopdf::Object::Array(vec![lopdf::Object::Integer(667); 26]),
+        ]);
+
+        assert!(w_array_covers_cid(&d, &doc, 0));
+        assert!(w_array_covers_cid(&d, &doc, 3));
+        assert!(w_array_covers_cid(&d, &doc, 5));
+        assert!(w_array_covers_cid(&d, &doc, 11));
+        assert!(w_array_covers_cid(&d, &doc, 12));
+        assert!(w_array_covers_cid(&d, &doc, 16));
+        assert!(w_array_covers_cid(&d, &doc, 18));
+        assert!(w_array_covers_cid(&d, &doc, 570));
+        assert!(w_array_covers_cid(&d, &doc, 595));
+        // Gaps are NOT covered
+        assert!(!w_array_covers_cid(&d, &doc, 1));
+        assert!(!w_array_covers_cid(&d, &doc, 4));
+        assert!(!w_array_covers_cid(&d, &doc, 19));
+        assert!(!w_array_covers_cid(&d, &doc, 596));
+    }
+
+    #[test]
+    fn test_w_array_covers_cid_format2() {
+        // Format 2: `c_first c_last w` — CIDs c_first..c_last all have width w.
+        let doc = Document::new();
+        let d = cid_font_dict_with_w(vec![
+            lopdf::Object::Integer(100),
+            lopdf::Object::Integer(120),
+            lopdf::Object::Integer(500),
+        ]);
+
+        assert!(w_array_covers_cid(&d, &doc, 100));
+        assert!(w_array_covers_cid(&d, &doc, 110));
+        assert!(w_array_covers_cid(&d, &doc, 120));
+        assert!(!w_array_covers_cid(&d, &doc, 99));
+        assert!(!w_array_covers_cid(&d, &doc, 121));
+    }
+
+    #[test]
+    fn test_w_array_covers_cid_missing_w() {
+        let doc = Document::new();
+        let d = lopdf::Dictionary::new();
+        assert!(!w_array_covers_cid(&d, &doc, 3));
+    }
+
+    #[test]
+    fn test_try_remap_skipped_when_w_covers_cmap() {
+        // Simulates 16.pdf: CMap's max source CID (0x0279 = 633) is explicitly
+        // in the W array, so no subset-renumbering happened — remap must NOT fire.
+        let cmap_content = r#"
+1 begincodespacerange
+<0000><FFFF>
+endcodespacerange
+2 beginbfchar
+<0003> <0020>
+<0031> <004E>
+endbfchar
+2 beginbfrange
+<023A> <0253> <0410>
+<0255> <0279> <042B>
+endbfrange
+"#;
+        let cmap = ToUnicodeCMap::parse(cmap_content.as_bytes()).unwrap();
+
+        let mut doc = Document::new();
+        // Build a CIDFont dict with Identity CIDToGIDMap and a W array that
+        // covers CID 633 via `597 [widths...]`.
+        let mut cid_font = lopdf::Dictionary::new();
+        cid_font.set("CIDToGIDMap", lopdf::Object::Name(b"Identity".to_vec()));
+        cid_font.set(
+            "W",
+            lopdf::Object::Array(vec![
+                lopdf::Object::Integer(0),
+                lopdf::Object::Array(vec![lopdf::Object::Integer(750)]),
+                lopdf::Object::Integer(597),
+                lopdf::Object::Array(vec![lopdf::Object::Integer(500); 37]), // 597..633
+            ]),
+        );
+        let cid_font_id = doc.add_object(cid_font);
+
+        // Build the Type0 font dict with Identity-H + DescendantFonts ref.
+        let mut font_dict = lopdf::Dictionary::new();
+        font_dict.set("Encoding", lopdf::Object::Name(b"Identity-H".to_vec()));
+        font_dict.set(
+            "DescendantFonts",
+            lopdf::Object::Array(vec![lopdf::Object::Reference(cid_font_id)]),
+        );
+
+        let (primary, remapped) = try_remap_subset_cmap(cmap, &font_dict, &doc, 123);
+        assert!(
+            remapped.is_none(),
+            "Remap must be skipped when W covers CMap max CID (this is 16.pdf)"
+        );
+        assert_eq!(primary.lookup(0x0003), Some(" ".to_string()));
+    }
+
+    #[test]
+    fn test_try_remap_fires_for_true_subset_mismatch() {
+        // True mismatch: CMap has high CIDs (512-544) but W only lists low sequential CIDs.
+        let cmap_content = r#"
+1 begincodespacerange
+<0000><FFFF>
+endcodespacerange
+1 beginbfrange
+<0200> <0220> <0410>
+endbfrange
+"#;
+        let cmap = ToUnicodeCMap::parse(cmap_content.as_bytes()).unwrap();
+
+        let mut doc = Document::new();
+        let mut cid_font = lopdf::Dictionary::new();
+        cid_font.set("CIDToGIDMap", lopdf::Object::Name(b"Identity".to_vec()));
+        cid_font.set(
+            "W",
+            lopdf::Object::Array(vec![
+                lopdf::Object::Integer(0),
+                lopdf::Object::Array(vec![lopdf::Object::Integer(500); 34]), // 0..33
+            ]),
+        );
+        let cid_font_id = doc.add_object(cid_font);
+
+        let mut font_dict = lopdf::Dictionary::new();
+        font_dict.set("Encoding", lopdf::Object::Name(b"Identity-H".to_vec()));
+        font_dict.set(
+            "DescendantFonts",
+            lopdf::Object::Array(vec![lopdf::Object::Reference(cid_font_id)]),
+        );
+
+        let (_primary, remapped) = try_remap_subset_cmap(cmap, &font_dict, &doc, 456);
+        assert!(
+            remapped.is_some(),
+            "Remap must fire when CMap's CIDs are outside W array coverage"
+        );
+    }
 }
@@ -4,10 +4,10 @@ use pdf_inspector::detector::{DetectionConfig, ScanStrategy};
 use pdf_inspector::extractor::group_into_lines;
 use pdf_inspector::types::TextLine;
 use pdf_inspector::{
-    detect_pdf_type, extract_pages_markdown_mem, extract_tables_in_regions_mem, extract_text,
-    extract_text_in_regions_mem, extract_text_with_positions, process_pdf_mem,
-    process_pdf_with_options, to_markdown, MarkdownOptions, PdfError, PdfOptions, PdfType,
-    TextItem,
+    detect_pdf_type, extract_pages_markdown, extract_pages_markdown_mem,
+    extract_tables_in_regions_mem, extract_text, extract_text_in_regions_mem,
+    extract_text_with_positions, process_pdf_mem, process_pdf_with_options, to_markdown,
+    MarkdownOptions, PdfError, PdfOptions, PdfType, TextItem,
 };
 use std::collections::HashSet;

@@ -1481,7 +1481,7 @@ fn test_bits_pilani_page8_table_detection() {
 #[test]
 fn test_extract_pages_markdown_basic() {
    let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
-    let result = extract_pages_markdown_mem(&buf, &[0, 1]).unwrap();
+    let result = extract_pages_markdown_mem(&buf, Some(&[0, 1])).unwrap();

    assert_eq!(result.pages.len(), 2);
    assert_eq!(result.pages[0].page, 0);
@@ -1495,7 +1495,7 @@ fn test_extract_pages_markdown_basic() {
 fn test_extract_pages_markdown_page_ordering() {
    let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
    // Request pages in non-sequential order
-    let result = extract_pages_markdown_mem(&buf, &[1, 0]).unwrap();
+    let result = extract_pages_markdown_mem(&buf, Some(&[1, 0])).unwrap();

    assert_eq!(result.pages.len(), 2);
    // Results should match input order, not document order
@@ -1506,7 +1506,7 @@ fn test_extract_pages_markdown_page_ordering() {
 #[test]
 fn test_extract_pages_markdown_out_of_range() {
    let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
-    let result = extract_pages_markdown_mem(&buf, &[9999]).unwrap();
+    let result = extract_pages_markdown_mem(&buf, Some(&[9999])).unwrap();

    assert_eq!(result.pages.len(), 1);
    assert_eq!(result.pages[0].page, 9999);
@@ -1518,14 +1518,14 @@ fn test_extract_pages_markdown_out_of_range() {
 #[test]
 fn test_extract_pages_markdown_empty_pages_list() {
    let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
-    let result = extract_pages_markdown_mem(&buf, &[]).unwrap();
+    let result = extract_pages_markdown_mem(&buf, Some(&[])).unwrap();
    assert!(result.pages.is_empty());
 }

 #[test]
 fn test_extract_pages_markdown_single_page() {
    let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
-    let result = extract_pages_markdown_mem(&buf, &[0]).unwrap();
+    let result = extract_pages_markdown_mem(&buf, Some(&[0])).unwrap();

    assert_eq!(result.pages.len(), 1);
    assert_eq!(result.pages[0].page, 0);
@@ -1535,7 +1535,7 @@ fn test_extract_pages_markdown_single_page() {

 #[test]
 fn test_extract_pages_markdown_invalid_buffer() {
-    let result = extract_pages_markdown_mem(b"not a pdf", &[0]);
+    let result = extract_pages_markdown_mem(b"not a pdf", Some(&[0]));
    assert!(result.is_err());
 }

@@ -1543,7 +1543,7 @@ fn test_extract_pages_markdown_invalid_buffer() {
 fn test_extract_pages_markdown_gid_pages_need_ocr() {
    // shinagawa_identity_h.pdf has GID-encoded fonts
    let buf = std::fs::read("tests/fixtures/shinagawa_identity_h.pdf").unwrap();
-    let result = extract_pages_markdown_mem(&buf, &[0]).unwrap();
+    let result = extract_pages_markdown_mem(&buf, Some(&[0])).unwrap();

    assert_eq!(result.pages.len(), 1);
    assert!(result.pages[0].needs_ocr);
@@ -1556,7 +1556,7 @@ fn test_extract_pages_markdown_classification_with_tables() {
    let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
    let page_count = process_pdf_mem(&buf).unwrap().page_count;
    let page_indices: Vec<u32> = (0..page_count).collect();
-    let result = extract_pages_markdown_mem(&buf, &page_indices).unwrap();
+    let result = extract_pages_markdown_mem(&buf, Some(&page_indices)).unwrap();

    assert!(
        !result.pages_with_tables.is_empty(),
@@ -1569,7 +1569,7 @@ fn test_extract_pages_markdown_classification_with_tables() {
 fn test_extract_pages_markdown_simple_pdf_no_complexity() {
    // bare_name_struct.pdf is a simple document with a heading and code block
    let buf = std::fs::read("tests/fixtures/bare_name_struct.pdf").unwrap();
-    let result = extract_pages_markdown_mem(&buf, &[0]).unwrap();
+    let result = extract_pages_markdown_mem(&buf, Some(&[0])).unwrap();

    assert!(result.pages_with_tables.is_empty());
    assert!(result.pages_with_columns.is_empty());
@@ -1582,7 +1582,7 @@ fn test_extract_pages_markdown_classification_matches_process_pdf() {
    let full = process_pdf_mem(&buf).unwrap();
    let page_count = full.page_count;
    let page_indices: Vec<u32> = (0..page_count).collect();
-    let result = extract_pages_markdown_mem(&buf, &page_indices).unwrap();
+    let result = extract_pages_markdown_mem(&buf, Some(&page_indices)).unwrap();

    assert_eq!(
        result.pages_with_tables, full.layout.pages_with_tables,
@@ -1605,7 +1605,7 @@ fn test_extract_pages_markdown_consistency_with_process_pdf() {
    // Get per-page output for all pages
    let page_count = full.page_count;
    let page_indices: Vec<u32> = (0..page_count).collect();
-    let result = extract_pages_markdown_mem(&buf, &page_indices).unwrap();
+    let result = extract_pages_markdown_mem(&buf, Some(&page_indices)).unwrap();

    // Concatenated per-page markdown should contain substantial overlap with
    // the full output (exact match not expected due to header/footer stripping
@@ -1630,3 +1630,41 @@ fn test_extract_pages_markdown_consistency_with_process_pdf() {
        full_md.len()
    );
 }
+
+#[test]
+fn test_extract_pages_markdown_none_returns_all_pages() {
+    let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
+    let page_count = process_pdf_mem(&buf).unwrap().page_count;
+
+    let result = extract_pages_markdown_mem(&buf, None).unwrap();
+
+    assert_eq!(result.pages.len() as u32, page_count);
+    for (i, page) in result.pages.iter().enumerate() {
+        assert_eq!(page.page, i as u32, "pages should be in document order");
+    }
+}
+
+#[test]
+fn test_extract_pages_markdown_path_api() {
+    let path = "tests/fixtures/nexo-price-en.pdf";
+    let buf = std::fs::read(path).unwrap();
+
+    let via_path = extract_pages_markdown(path, Some(&[0])).unwrap();
+    let via_mem = extract_pages_markdown_mem(&buf, Some(&[0])).unwrap();
+
+    assert_eq!(via_path.pages.len(), via_mem.pages.len());
+    assert_eq!(via_path.pages[0].markdown, via_mem.pages[0].markdown);
+    assert_eq!(via_path.pages[0].needs_ocr, via_mem.pages[0].needs_ocr);
+    assert_eq!(via_path.is_complex, via_mem.is_complex);
+}
+
+#[test]
+fn test_extract_pages_markdown_path_none_returns_all_pages() {
+    let path = "tests/fixtures/nexo-price-en.pdf";
+    let page_count = process_pdf_mem(&std::fs::read(path).unwrap())
+        .unwrap()
+        .page_count;
+
+    let result = extract_pages_markdown(path, None).unwrap();
+    assert_eq!(result.pages.len() as u32, page_count);
+}
@@ -270,6 +270,78 @@ class TestExtractTextInRegions:
            )


+# ---------------------------------------------------------------------------
+# extract_pages_markdown / extract_pages_markdown_bytes
+# ---------------------------------------------------------------------------
+
+
+class TestExtractPagesMarkdown:
+    def test_default_returns_all_pages(self):
+        result = pdf_inspector.extract_pages_markdown(
+            fixture_path("thermo-freon12.pdf")
+        )
+        assert len(result.pages) == 3
+        assert [p.page for p in result.pages] == [0, 1, 2]
+        assert all(isinstance(p.markdown, str) for p in result.pages)
+
+    def test_bytes_default_returns_all_pages(self):
+        data = fixture_bytes("thermo-freon12.pdf")
+        result = pdf_inspector.extract_pages_markdown_bytes(data)
+        assert len(result.pages) == 3
+
+    def test_selected_pages_preserve_order(self):
+        result = pdf_inspector.extract_pages_markdown(
+            fixture_path("thermo-freon12.pdf"), pages=[2, 0]
+        )
+        assert [p.page for p in result.pages] == [2, 0]
+
+    def test_bytes_selected_pages_preserve_order(self):
+        data = fixture_bytes("thermo-freon12.pdf")
+        result = pdf_inspector.extract_pages_markdown_bytes(data, pages=[1])
+        assert len(result.pages) == 1
+        assert result.pages[0].page == 1
+
+    def test_page_fields(self):
+        result = pdf_inspector.extract_pages_markdown(
+            fixture_path("thermo-freon12.pdf"), pages=[0]
+        )
+        page = result.pages[0]
+        assert isinstance(page.page, int)
+        assert isinstance(page.markdown, str)
+        assert isinstance(page.needs_ocr, bool)
+        assert not page.needs_ocr  # text-based fixture
+        assert len(page.markdown) > 0
+
+    def test_result_fields(self):
+        result = pdf_inspector.extract_pages_markdown(
+            fixture_path("thermo-freon12.pdf")
+        )
+        assert isinstance(result.pages, list)
+        assert isinstance(result.pages_with_tables, list)
+        assert isinstance(result.pages_with_columns, list)
+        assert isinstance(result.pages_needing_ocr, list)
+        assert isinstance(result.is_complex, bool)
+
+    def test_out_of_range_page_marks_needs_ocr(self):
+        result = pdf_inspector.extract_pages_markdown(
+            fixture_path("thermo-freon12.pdf"), pages=[9999]
+        )
+        assert len(result.pages) == 1
+        assert result.pages[0].needs_ocr
+        assert result.pages[0].markdown == ""
+
+    def test_repr(self):
+        result = pdf_inspector.extract_pages_markdown(
+            fixture_path("thermo-freon12.pdf"), pages=[0]
+        )
+        assert "PagesExtractionResult" in repr(result)
+        assert "PageMarkdown" in repr(result.pages[0])
+
+    def test_not_a_pdf(self):
+        with pytest.raises(ValueError):
+            pdf_inspector.extract_pages_markdown_bytes(b"not a pdf")
+
+
 # ---------------------------------------------------------------------------
 # Error handling
 # ---------------------------------------------------------------------------
Author	SHA1	Message	Date
Abimael MartellandClaude Opus 4.7	1061f5b5e0	fix: don't reclassify wrapped bold list leads as headings When a numbered/bulleted list item's bold lead phrase wraps onto a second visual line, that line is all_bold + standalone, which scored above the rarity heading threshold and was emitted as #### in the middle of the item. That reset in_list, so the body continuation below picked up a stray `- ` bullet via the struct-tree LI path, shattering a single item into heading + stray bullets. Guard the font heuristic: when already inside a list, skip heading classification for lines at the list continuation indent with a Y gap within para_threshold. Structure-tree headings still win. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-21 08:43:04 -07:00
Abimael Martell	df023474f1	update eval instructions	2026-04-20 22:50:57 -07:00
Abimael MartellandClaude Opus 4.7	f44509ac3a	fix: don't treat bullet-marker columns as real text columns (#55 ) On PDFs where every list item starts with ● at the left margin and content at a fixed offset, histogram column detection sees the gap between marker and content as a gutter and splits each line across two phantom "columns," scrambling the reading order (Anthropic's Mythos system card p.73–74). - layout: reject gutter candidates where the smaller side is ≥80% standalone bullet-marker glyphs (•, ●, ○, ◦, ▪, ▫, ◆, ◇, ■, □) - markdown/classify: add starts_with_bullet_marker helper (narrower than is_list_item — excludes numbered/lettered patterns like 1. and a) so numbered section headings stay as headings) - markdown/convert: skip heuristic heading detection on lines that start with a bullet marker - markdown/classify: strip a leading bullet wrapped in a bold/italic run (e.g. "● Label:" → "- Label:") — some PDFs put the marker inside the same bold run as the label Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-20 22:47:34 -07:00
Abimael MartellandClaude Opus 4.7	2e933ba8c1	fix: don't split tagged-LI continuation lines into separate bullets (#54 ) Some tagged PDFs use a flat style where each wrapped visual line of a list item gets its own MCID tagged directly under /LI. The LI branch was unconditionally prefixing every such line with "- ", turning continuation lines into their own bullets. Only emit a new bullet when we're not already inside a list; otherwise fall through to the existing continuation logic so the text is appended to the previous item. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-20 21:59:53 -07:00
Abimael MartellandClaude Opus 4.7	4b5ae91f54	feat: expose per-page markdown extraction to Python and Node (#53 ) * feat: expose per-page markdown extraction to Python and Node (#49) Implements the feature requested in issue #49: a list-of-pages markdown output from the Python API. Matching the existing project pattern, the feature lives in the Rust core and is surfaced through every binding. - Rust core: `extract_pages_markdown` (path) and `extract_pages_markdown_mem` (bytes) now take `Option<&[u32]>` — `None` returns every page in document order; a slice restricts and preserves caller order. - Python: new `extract_pages_markdown(path, pages=None)` and `extract_pages_markdown_bytes(data, pages=None)` functions plus `PageMarkdown` / `PagesExtractionResult` classes; stub file updated. - Node: `extractPagesMarkdown(buffer, pages?)` — `pages` is now optional. - Tests: 2 new Rust integration tests, 9 new Python tests, 2 new Node assertions. All 372 unit + 107 integration + 53 Python tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version from 1.3.0 to 1.4.0 Minor bump for the new per-page markdown extraction API exposed through the Python and Node bindings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-20 17:19:03 -07:00
Abimael MartellandClaude Opus 4.7	b4c34ba4b7	refactor: classify Table as Data vs Toc once at construction (#52 ) The TOC/data-table distinction was being recomputed at every consumer: - format.rs::table_to_markdown ran is_table_of_contents to decide between flat-list and markdown-table rendering. - compute_layout_complexity ran it again to filter TOCs out of pages_with_tables. - detect_heuristic validations used it to decide whether to relax val 1/9. Each caller had to remember tables can be either kind, which leaks the TOC concept across the codebase. Add `TableKind { Data, Toc }` and a `Table::new` constructor that classifies once from the cells. All five detectors (heuristic, rect, line, struct, columns) now go through `Table::new`. Consumers match on `kind` instead of re-running classification. Pure refactor — no behavior change. Verified: pdf-evals output is byte-for- byte identical (0 changed snapshots). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-20 14:04:50 -07:00
Abimael MartellandClaude Opus 4.7	462a7fdfde	fix: exclude TOC pages from pages_with_tables metadata (#51 ) * fix: detect hierarchical-indent TOCs as tables Validation 1 (≥25% of rows have first col) and validation 9 (paragraph content) were rejecting TOC pages where top-level chapters indent at the leftmost X but subsections cascade further right. The leftmost column ends up sparse (only chapter rows land there) and most cells are empty, but the structure is still an unambiguous TOC. Skip both validations when cells form a TOC pattern AND the table is narrow (≤5 cols). The width cap preserves existing handling of wide multi-column TOCs (e.g. back-of-book 2-up indices) where format_toc_as_list would mash adjacent visual entries together. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: exclude TOC pages from pages_with_tables metadata TOC detection routes through the table pipeline (detected as a table, then formatted as a flat list with tab-aligned page numbers via format_toc_as_list). That made TOC pages appear in LayoutComplexity's pages_with_tables, which: - Misleads downstream consumers that read this field as "this page has a data table". - Trips the table-page guard in column detection, which switches to a different valley-detection threshold for table pages. Add an is_table_of_contents check in compute_layout_complexity so TOC-shaped detections don't count toward the table flag. The TOC still renders correctly as a flat list — only the metadata classification changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-20 13:48:36 -07:00
Abimael MartellandClaude Opus 4.7	9abcbbb359	fix: detect hierarchical-indent TOCs as tables (#50 ) Validation 1 (≥25% of rows have first col) and validation 9 (paragraph content) were rejecting TOC pages where top-level chapters indent at the leftmost X but subsections cascade further right. The leftmost column ends up sparse (only chapter rows land there) and most cells are empty, but the structure is still an unambiguous TOC. Skip both validations when cells form a TOC pattern AND the table is narrow (≤5 cols). The width cap preserves existing handling of wide multi-column TOCs (e.g. back-of-book 2-up indices) where format_toc_as_list would mash adjacent visual entries together. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-20 13:41:59 -07:00
Abimael Martell	0b3ba8784c	Bump version from 1.2.0 to 1.3.0	2026-04-20 11:09:16 -07:00
Abimael MartellandClaude Opus 4.7	3cd9795fe1	fix: don't fire subset-GID remap when W array covers CMap CIDs (#47 ) The subset-GID-mismatch heuristic tripped whenever the CIDFont's W array started at CID 0 and the ToUnicode CMap's minimum source CID was > 2. That's the normal sparse layout for a correctly-subsetted Identity-H font with a .notdef at CID 0 and actual glyphs at high CIDs (Cyrillic, Arabic, etc.). The spurious remap to sequential CIDs, combined with score_text penalizing non-Latin letters as "other", caused the garbled CMap to win over the correct primary — spaces turned into %, digits shifted into punctuation, and Latin letters got rewritten to Cyrillic codepoints. Check whether the W array actually covers the CMap's maximum source CID. If it does, the font and CMap agree, no renumbering happened, and the remap stays off. True mismatches (CMap CIDs outside W coverage) still trigger the remap. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-19 09:02:23 -07:00
Abimael MartellandClaude Opus 4.7	c0607ab152	feat(npm): add Windows binary to publish matrix (#46 ) Adds x86_64-pc-windows-msvc target to the napi package and CI build matrix so the published @firecrawl/pdf-inspector npm package ships native Windows binaries alongside Linux and macOS. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>	2026-04-17 13:58:17 -07:00
Abimael Martell	cf2faa3221	Bump version to 1.1.0 in package.json	2026-04-17 11:59:14 -07:00
Abimael MartellandClaude Opus 4.7	e043c4414d	fix: reject dot-less TOC as heuristic table false-positive (#45 ) * fix: reject dot-less tables of contents as heuristic table false-positive The body-font heuristic table detector was firing on tagged PDF TOCs whose entries are laid out as multi-column rows (section number, title, page number) without dot leaders. Extend is_table_of_contents to flag this pattern by requiring: - 2+ columns and 4+ rows - >=60% of rows start with a dotted section number (e.g. "4.3.1") - >=70% of filled last-column cells are all-digit page numbers Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * format TOC tables as flat per-row list instead of dropping them Keeping the previous approach (rejecting TOCs at detection time) meant dot-less TOCs fell back to the column-aware text reader, which stacked every section title first and dumped all page numbers at the end of the region — so a reader saw a wall of titles followed by a wall of numbers. Keep the detected Table, and in the formatter interleave the cells into one line per row with the page number appended after a tab. The raw (pre-clean) cells are used for the TOC check because clean_table_cells can collapse genuine data tables into a TOC-looking shape. Tightened is_table_of_contents to require both dot leaders AND page numbers (previously dot-ratio alone was enough, which misfired on register tables that use "..." as a continuation marker). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * split TOC detection into dot-leader and tabular patterns Previous is_table_of_contents rejected every TOC-shaped table at detect time, dropping dot-leader TOCs that render well as flat-list output and misclassifying data tables with trailing "..." cells as TOCs. Now is_dot_leader_toc (structural + inline-leader) keeps per-row flat layouts for the formatter, while only wide inline-leader indices are rejected at detect time. row_cell_is_page_number accepts dashed section-page IDs ("A-1", "5-21") and rejects decimals/thousands separators; cell_has_trailing_leader requires alphabetic content so numeric data-row labels ("1973 ... ") no longer qualify. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>	2026-04-17 11:58:41 -07:00
Abimael Martell	6466e59271	update node pacakge paths in docs (add agents file too)	2026-04-17 08:33:37 -07:00
Abimael Martell	adcaaa1b3a	npm package under firecrawl organization	2026-04-17 08:31:18 -07:00
Abimael MartellandClaude Opus 4.6	b17e8e3443	feat: add pdf-inspector CLI to npm package (#44 ) * feat: add CLI bin to npm package Installing `firecrawl-pdf-inspector` now provides a `pdf-inspector` CLI command. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump npm package version to 0.8.0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump npm package version to 1.0.0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-17 07:23:08 -07:00
Abimael Martell	c409e3b8ca	fix: use first-glyph position for ActualText ligature items (#43 ) * fix: use first-glyph position for ActualText ligature items The content stream parser emitted ActualText items (from BDC/Span marked content) at the text matrix position captured at BDC entry. When the content stream contains Td operators between BDC and the first glyph Tj — common in PDFs generated by Google Docs, Figma, and web-to-PDF tools — the BDC-entry position is on the previous visual line while the actual glyph renders on the correct line. This caused ligature glyphs (fi, fl, ff, etc.) wrapped in ActualText spans to be positioned one text-leading offset above their surrounding text. Downstream merge logic couldn't reconnect them (different y-band), producing broken words like "fi" + "ndings" instead of "findings" throughout the document. Fix: capture the text matrix at the first Tj/TJ operation inside the BDC block (after any Td repositioning), and use that as the ActualText item's rendering position at EMC time. Falls back to the BDC-entry position when no glyph ops occur inside the block. One new variable, five insertion points, zero behavior change on PDFs that don't use ActualText marked content. * chore: allow clippy::collapsible_match for Rust 1.95 Rust 1.95 introduced the collapsible_match lint which flags `if` blocks inside match arms that could be converted to match guards. The content-stream parsers use this pattern extensively for readability (match on PDF operator name, then check preconditions like `in_text_block && !op.operands.is_empty()`). Allow crate-wide rather than refactoring 21 match arms across the parser files.	2026-04-16 17:14:00 -07:00
Abimael Martell	704ca588f5	Update package name in README for pdf inspector	2026-04-15 15:43:45 -07:00