Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Abimael Martell
|
0c89fbea53 | ||
|
Abimael MartellandClaude Opus 4.6
|
100cbe5453 | ||
|
Abimael MartellandClaude Opus 4.6
|
aa5dca7a34 | ||
|
Abimael MartellandClaude Opus 4.6
|
7f8773571d |
@@ -46,8 +46,6 @@ jobs:
|
||||
target: x86_64-unknown-linux-gnu
|
||||
- os: macos-14
|
||||
target: aarch64-apple-darwin
|
||||
- os: windows-latest
|
||||
target: x86_64-pc-windows-msvc
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
|
||||
@@ -35,8 +35,3 @@ scripts/
|
||||
# Test output
|
||||
test_output/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.pytest_cache/
|
||||
|
||||
|
||||
@@ -1,79 +0,0 @@
|
||||
# pdf-inspector
|
||||
|
||||
Fast PDF text extraction to structured Markdown. CLI binary: `pdf2md`. Detection binary: `detect-pdf`.
|
||||
|
||||
## Build & Test
|
||||
|
||||
```bash
|
||||
cargo fmt # format
|
||||
cargo clippy -- -D warnings # lint (enforced, zero warnings)
|
||||
cargo test # unit + integration tests (267+ unit, 73+ integration)
|
||||
cargo build --release # release binary for benchmarks
|
||||
```
|
||||
|
||||
All three must pass before committing.
|
||||
|
||||
## Binaries
|
||||
|
||||
- `pdf2md` — extract PDF → Markdown. Supports `--json` for structured output.
|
||||
- `detect-pdf` — classify PDF type (TextBased/Scanned/Mixed/ImageBased). Supports `--analyze --json`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/
|
||||
lib.rs – public API, process_pdf_with_options, encoding issue detection
|
||||
detector.rs – PDF type classification, tiled-scan detection, page sampling
|
||||
types.rs – TextItem, TextLine, PdfRect, PdfLine
|
||||
tounicode.rs – CMap/ToUnicode parsing, CID decoding
|
||||
text_utils.rs – CJK/RTL handling, Otsu threshold, ligature expansion, NFKC
|
||||
extractor/
|
||||
mod.rs – top-level extraction orchestrator
|
||||
content_stream.rs – PDF operator state machine (Tj/TJ/Td/Tm/q/Q)
|
||||
fonts.rs – font width/encoding, CMapDecisionCache, TrueType cmap fallback
|
||||
layout.rs – column detection (histogram), newspaper/tabular classification,
|
||||
spanning-line pre-masking, sidebar detection
|
||||
tables/
|
||||
detect_rects.rs – rect-based table detection (union-find clustering)
|
||||
detect_heuristic.rs – heuristic table detection (gap-histogram, body-font tables)
|
||||
detect_lines.rs – line-based table detection (H/V line grids)
|
||||
grid.rs – column/row boundaries, cell assignment
|
||||
format.rs – table→Markdown formatting, continuation row merging
|
||||
markdown/
|
||||
convert.rs – core line→Markdown loop, struct-tree role support
|
||||
analysis.rs – font stats, heading tiers, paragraph thresholds
|
||||
classify.rs – line classification (header, list, code, caption)
|
||||
preprocess.rs – drop cap merging, heading line merging
|
||||
postprocess.rs – dot leaders, hyphenation, page numbers, URL formatting
|
||||
```
|
||||
|
||||
## Key design decisions
|
||||
|
||||
- **Primary audience is AI agents.** Output optimized for token efficiency and semantic quality, not visual formatting. No cosmetic padding.
|
||||
- **Three table detection strategies** run in priority order: rect-based → line-based → heuristic. First valid result wins.
|
||||
- **Column detection** uses horizontal projection histograms with valley detection. Multi-item spanning lines (titles, headers) are pre-masked using column-aware thresholds before column assignment.
|
||||
- **Newspaper vs tabular** classification determines reading order: newspaper reads columns sequentially, tabular Y-interleaves them.
|
||||
- **Tiled-scan detection** catches scanned PDFs with JBIG2/strip images where no single tile exceeds the template threshold but aggregate area does (≥2M pixels).
|
||||
- **Garbage text upgrade** reclassifies Mixed PDFs as Scanned when extracted text is <50% alphanumeric.
|
||||
- **Tagged PDF support** uses structure tree roles (H1-H6, P, L, Code, BlockQuote) when available, falling back to font-size heuristics.
|
||||
|
||||
## Testing
|
||||
|
||||
- **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.
|
||||
- **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.
|
||||
- **Regression suite**: sibling repo `pdf-evals` with 179+ snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing.
|
||||
|
||||
## Debugging
|
||||
|
||||
```bash
|
||||
RUST_LOG=pdf_inspector::extractor::layout=debug cargo run --bin pdf2md -- file.pdf
|
||||
RUST_LOG=pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf
|
||||
RUST_LOG=pdf_inspector::detector=debug cargo run --release --bin detect-pdf -- file.pdf
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- Clippy: use `is_some_and(...)` not `map_or(false, ...)`
|
||||
- lopdf quirk: `ParseError` is private — match by string for `InvalidFileHeader`
|
||||
- Column limit for tables: 25 (wide statistical tables)
|
||||
- `propagate_merged_cells` skipped for >10 columns (spanning rects = background fills)
|
||||
@@ -55,12 +55,12 @@ print(result.markdown) # Markdown string or None
|
||||
### Node.js
|
||||
|
||||
```bash
|
||||
npm install @firecrawl/pdf-inspector
|
||||
npm install @firecrawl/pdf-inspector-js
|
||||
```
|
||||
|
||||
```javascript
|
||||
import { readFileSync } from 'fs';
|
||||
import { processPdf, classifyPdf } from '@firecrawl/pdf-inspector';
|
||||
import { processPdf, classifyPdf } from '@firecrawl/pdf-inspector-js';
|
||||
|
||||
const result = processPdf(readFileSync('document.pdf'));
|
||||
console.log(result.pdfType); // "TextBased", "Scanned", "ImageBased", "Mixed"
|
||||
|
||||
@@ -42,14 +42,6 @@ text = pdf_inspector.extract_text("document.pdf")
|
||||
items = pdf_inspector.extract_text_with_positions("document.pdf")
|
||||
for item in items[:5]:
|
||||
print(f"'{item.text}' at ({item.x:.0f}, {item.y:.0f}) size={item.font_size}")
|
||||
|
||||
# Per-page markdown (one Markdown string per page, plus layout metadata)
|
||||
result = pdf_inspector.extract_pages_markdown("document.pdf")
|
||||
for page in result.pages:
|
||||
print(f"Page {page.page}: {len(page.markdown)} chars, needs_ocr={page.needs_ocr}")
|
||||
|
||||
# Restrict to specific 0-indexed pages (preserves caller order)
|
||||
result = pdf_inspector.extract_pages_markdown("document.pdf", pages=[0, 2])
|
||||
```
|
||||
|
||||
## API reference
|
||||
@@ -68,8 +60,6 @@ result = pdf_inspector.extract_pages_markdown("document.pdf", pages=[0, 2])
|
||||
| `extract_text_with_positions_bytes(data, pages=None)` | Text with positions from bytes |
|
||||
| `extract_text_in_regions(path, page_regions)` | Extract text in bounding-box regions |
|
||||
| `extract_text_in_regions_bytes(data, page_regions)` | Region extraction from bytes |
|
||||
| `extract_pages_markdown(path, pages=None)` | Per-page Markdown + layout metadata (all pages by default) |
|
||||
| `extract_pages_markdown_bytes(data, pages=None)` | Per-page Markdown from bytes |
|
||||
|
||||
## Types
|
||||
|
||||
@@ -82,7 +72,3 @@ result = pdf_inspector.extract_pages_markdown("document.pdf", pages=[0, 2])
|
||||
**`RegionText` fields:** `text`, `needs_ocr`
|
||||
|
||||
**`PageRegionTexts` fields:** `page` (0-indexed), `regions` (list of RegionText)
|
||||
|
||||
**`PageMarkdown` fields:** `page` (0-indexed), `markdown`, `needs_ocr`
|
||||
|
||||
**`PagesExtractionResult` fields:** `pages` (list of PageMarkdown), `pages_with_tables` (1-indexed), `pages_with_columns` (1-indexed), `pages_needing_ocr` (1-indexed), `is_complex`
|
||||
|
||||
@@ -79,27 +79,6 @@ let bytes = std::fs::read("document.pdf")?;
|
||||
let result = process_pdf_mem(&bytes)?;
|
||||
```
|
||||
|
||||
Extract per-page Markdown (one string per page, plus document-wide layout
|
||||
metadata):
|
||||
|
||||
```rust
|
||||
use pdf_inspector::extract_pages_markdown;
|
||||
|
||||
// Pass `None` for every page in document order, or a slice of 0-indexed
|
||||
// pages to restrict the output (caller-supplied order is preserved).
|
||||
let result = extract_pages_markdown("document.pdf", None)?;
|
||||
|
||||
for page in &result.pages {
|
||||
if page.needs_ocr {
|
||||
// Route this page to OCR
|
||||
} else {
|
||||
println!("Page {}: {}", page.page, page.markdown);
|
||||
}
|
||||
}
|
||||
|
||||
println!("Complex layout? {}", result.is_complex);
|
||||
```
|
||||
|
||||
## Processing modes
|
||||
|
||||
| Mode | What it does | Returns |
|
||||
@@ -123,8 +102,6 @@ println!("Complex layout? {}", result.is_complex);
|
||||
| `to_markdown(text, options)` | Convert plain text to Markdown |
|
||||
| `to_markdown_from_items(items, options)` | Markdown from pre-extracted `TextItem`s |
|
||||
| `to_markdown_from_items_with_rects(items, options, rects)` | Markdown with rectangle-based table detection |
|
||||
| `extract_pages_markdown(path, pages)` | Per-page Markdown + layout metadata (file) |
|
||||
| `extract_pages_markdown_mem(bytes, pages)` | Per-page Markdown from bytes |
|
||||
|
||||
Low-level detection functions are also available via the `detector` module (`detect_pdf_type`, `detect_pdf_type_with_config`, etc.) for callers who need `PdfTypeResult` instead of `PdfProcessResult`.
|
||||
|
||||
@@ -142,6 +119,4 @@ Low-level detection functions are also available via the `detector` module (`det
|
||||
| `LayoutComplexity` | Layout analysis: is_complex, pages_with_tables, pages_with_columns |
|
||||
| `TextItem` | Text with position, font info, and page number |
|
||||
| `MarkdownOptions` | Configuration for Markdown formatting (page numbers, etc.) |
|
||||
| `PageMarkdown` | Per-page result: page (0-indexed), markdown, needs_ocr |
|
||||
| `PagesExtractionResult` | Per-page output + 1-indexed pages_with_tables / pages_with_columns / pages_needing_ocr, is_complex |
|
||||
| `PdfError` | `Io`, `Parse`, `Encrypted`, `InvalidStructure`, `NotAPdf` |
|
||||
|
||||
+5
-5
@@ -1,4 +1,4 @@
|
||||
# PDF Inspector
|
||||
# firecrawl-pdf-inspector
|
||||
|
||||
Fast PDF classification and region-based text extraction for Node.js/Bun. Native Rust performance via [napi-rs](https://napi.rs).
|
||||
|
||||
@@ -7,9 +7,9 @@ Built by [Firecrawl](https://firecrawl.dev) for hybrid OCR pipelines — extract
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm install @firecrawl/pdf-inspector
|
||||
npm install firecrawl-pdf-inspector
|
||||
# or
|
||||
bun add @firecrawl/pdf-inspector
|
||||
bun add firecrawl-pdf-inspector
|
||||
```
|
||||
|
||||
Prebuilt binaries included for **linux-x64** and **macOS ARM64**. No Rust toolchain needed.
|
||||
@@ -21,7 +21,7 @@ Prebuilt binaries included for **linux-x64** and **macOS ARM64**. No Rust toolch
|
||||
Classify a PDF as TextBased, Scanned, Mixed, or ImageBased (~10-50ms). Returns which pages need OCR.
|
||||
|
||||
```typescript
|
||||
import { classifyPdf } from '@firecrawl/pdf-inspector'
|
||||
import { classifyPdf } from 'firecrawl-pdf-inspector'
|
||||
import { readFileSync } from 'fs'
|
||||
|
||||
const pdf = readFileSync('document.pdf')
|
||||
@@ -40,7 +40,7 @@ Extract text within bounding-box regions from a PDF. Designed for hybrid OCR pip
|
||||
Each region result includes a `needsOcr` flag that signals unreliable extraction (empty text, GID-encoded fonts, garbage text, encoding issues).
|
||||
|
||||
```typescript
|
||||
import { extractTextInRegions } from '@firecrawl/pdf-inspector'
|
||||
import { extractTextInRegions } from 'firecrawl-pdf-inspector'
|
||||
|
||||
const result = extractTextInRegions(pdf, [
|
||||
{
|
||||
|
||||
@@ -1,131 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { readFileSync, writeFileSync } from "fs";
|
||||
import { createRequire } from "module";
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const { version } = require("../package.json");
|
||||
|
||||
const HELP = `pdf-inspector v${version} — Fast PDF text extraction to Markdown
|
||||
|
||||
Usage:
|
||||
pdf-inspector <file> Extract markdown (default)
|
||||
pdf-inspector detect <file> Classify PDF type
|
||||
|
||||
Options:
|
||||
--json Output as JSON
|
||||
--pages <pages> Comma-separated page numbers (e.g. 1,3,5)
|
||||
-o, --output <file> Write output to file instead of stdout
|
||||
-h, --help Show this help
|
||||
-v, --version Show version
|
||||
|
||||
Examples:
|
||||
pdf-inspector document.pdf
|
||||
pdf-inspector document.pdf --json
|
||||
pdf-inspector document.pdf --pages 1,2,3
|
||||
pdf-inspector detect document.pdf --json
|
||||
cat document.pdf | pdf-inspector -`;
|
||||
|
||||
function die(msg) {
|
||||
process.stderr.write(`error: ${msg}\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const opts = { json: false, pages: null, output: null, file: null, command: "extract" };
|
||||
let i = 0;
|
||||
|
||||
// Check for subcommand
|
||||
if (argv[0] === "detect") {
|
||||
opts.command = "detect";
|
||||
i = 1;
|
||||
}
|
||||
|
||||
while (i < argv.length) {
|
||||
const arg = argv[i];
|
||||
if (arg === "-h" || arg === "--help") {
|
||||
process.stdout.write(HELP + "\n");
|
||||
process.exit(0);
|
||||
} else if (arg === "-v" || arg === "--version") {
|
||||
process.stdout.write(`${version}\n`);
|
||||
process.exit(0);
|
||||
} else if (arg === "--json") {
|
||||
opts.json = true;
|
||||
} else if (arg === "--pages") {
|
||||
i++;
|
||||
if (!argv[i]) die("--pages requires a value (e.g. 1,3,5)");
|
||||
opts.pages = argv[i].split(",").map((p) => {
|
||||
const n = parseInt(p.trim(), 10);
|
||||
if (Number.isNaN(n) || n < 1) die(`invalid page number: ${p}`);
|
||||
return n;
|
||||
});
|
||||
} else if (arg === "-o" || arg === "--output") {
|
||||
i++;
|
||||
if (!argv[i]) die("-o requires a filename");
|
||||
opts.output = argv[i];
|
||||
} else if (arg === "-" || !arg.startsWith("-")) {
|
||||
if (opts.file) die(`unexpected argument: ${arg}`);
|
||||
opts.file = arg;
|
||||
} else {
|
||||
die(`unknown option: ${arg}`);
|
||||
}
|
||||
i++;
|
||||
}
|
||||
|
||||
return opts;
|
||||
}
|
||||
|
||||
function readInput(file) {
|
||||
if (file === "-") {
|
||||
return readFileSync(0); // stdin fd
|
||||
}
|
||||
try {
|
||||
return readFileSync(file);
|
||||
} catch (err) {
|
||||
if (err.code === "ENOENT") die(`file not found: ${file}`);
|
||||
die(err.message);
|
||||
}
|
||||
}
|
||||
|
||||
function output(text, outputPath) {
|
||||
if (outputPath) {
|
||||
writeFileSync(outputPath, text);
|
||||
} else {
|
||||
process.stdout.write(text);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- main ----
|
||||
|
||||
const opts = parseArgs(process.argv.slice(2));
|
||||
|
||||
if (!opts.file) {
|
||||
// Check if stdin is piped
|
||||
if (process.stdin.isTTY !== false) {
|
||||
process.stderr.write(HELP + "\n");
|
||||
process.exit(1);
|
||||
}
|
||||
opts.file = "-";
|
||||
}
|
||||
|
||||
const { processPdf, classifyPdf } = await import("../index.js");
|
||||
const buffer = readInput(opts.file);
|
||||
|
||||
if (opts.command === "detect") {
|
||||
const result = classifyPdf(buffer);
|
||||
if (opts.json) {
|
||||
output(JSON.stringify(result, null, 2) + "\n", opts.output);
|
||||
} else {
|
||||
const ocr = result.pagesNeedingOcr.length > 0
|
||||
? `, ${result.pagesNeedingOcr.length} pages need OCR`
|
||||
: "";
|
||||
output(`${result.pdfType} (${result.pageCount} pages, confidence: ${result.confidence.toFixed(2)}${ocr})\n`, opts.output);
|
||||
}
|
||||
} else {
|
||||
const result = processPdf(buffer, opts.pages ?? undefined);
|
||||
if (opts.json) {
|
||||
output(JSON.stringify(result, null, 2) + "\n", opts.output);
|
||||
} else {
|
||||
output((result.markdown ?? "") + "\n", opts.output);
|
||||
}
|
||||
}
|
||||
+3
-8
@@ -1,12 +1,9 @@
|
||||
{
|
||||
"name": "@firecrawl/pdf-inspector",
|
||||
"version": "1.4.0",
|
||||
"name": "firecrawl-pdf-inspector",
|
||||
"version": "0.7.2",
|
||||
"description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
|
||||
"main": "index.js",
|
||||
"types": "index.d.ts",
|
||||
"bin": {
|
||||
"pdf-inspector": "bin/pdf-inspector.mjs"
|
||||
},
|
||||
"license": "MIT",
|
||||
"keywords": [
|
||||
"pdf",
|
||||
@@ -23,7 +20,6 @@
|
||||
"index.js",
|
||||
"index.d.ts",
|
||||
"*.node",
|
||||
"bin/",
|
||||
"README.md"
|
||||
],
|
||||
"repository": {
|
||||
@@ -38,8 +34,7 @@
|
||||
"binaryName": "pdf-inspector",
|
||||
"targets": [
|
||||
"x86_64-unknown-linux-gnu",
|
||||
"aarch64-apple-darwin",
|
||||
"x86_64-pc-windows-msvc"
|
||||
"aarch64-apple-darwin"
|
||||
],
|
||||
"package": {
|
||||
"name": "@firecrawl/pdf-inspector-js"
|
||||
|
||||
+5
-10
@@ -343,26 +343,21 @@ pub struct PagesExtractionResult {
|
||||
pub is_complex: bool,
|
||||
}
|
||||
|
||||
/// Extract formatted markdown for pages of a PDF, with layout classification
|
||||
/// metadata.
|
||||
/// Extract formatted markdown for specific pages of a PDF, with layout
|
||||
/// classification metadata.
|
||||
///
|
||||
/// Returns per-page markdown and classification data (tables, columns,
|
||||
/// OCR needs) from a single parse. Font statistics are computed from the
|
||||
/// full document so header detection is consistent across pages.
|
||||
///
|
||||
/// Omit `pages` (or pass `undefined`) to return every page in document
|
||||
/// order. Pass an array of 0-indexed page numbers to restrict output to
|
||||
/// those pages, in caller-supplied order.
|
||||
#[napi]
|
||||
pub fn extract_pages_markdown(
|
||||
buffer: Buffer,
|
||||
pages: Option<Vec<u32>>,
|
||||
pages: Vec<u32>,
|
||||
) -> Result<PagesExtractionResult> {
|
||||
let bytes: Vec<u8> = buffer.to_vec();
|
||||
catch_panic("extract_pages_markdown", move || {
|
||||
let result =
|
||||
pdf_inspector::extract_pages_markdown_mem(&bytes, pages.as_deref())
|
||||
.map_err(|e| to_napi_err(e, "extract_pages_markdown"))?;
|
||||
let result = pdf_inspector::extract_pages_markdown_mem(&bytes, &pages)
|
||||
.map_err(|e| to_napi_err(e, "extract_pages_markdown"))?;
|
||||
Ok(PagesExtractionResult {
|
||||
pages: result
|
||||
.pages
|
||||
|
||||
@@ -7,7 +7,6 @@ import {
|
||||
extractText,
|
||||
extractTextWithPositions,
|
||||
extractTextInRegions,
|
||||
extractPagesMarkdown,
|
||||
} from './index.js';
|
||||
|
||||
const fixture = readFileSync('../tests/fixtures/thermo-freon12.pdf');
|
||||
@@ -90,28 +89,6 @@ assert.equal(typeof regionResults[0].regions[0].text, 'string');
|
||||
assert.equal(typeof regionResults[0].regions[0].needsOcr, 'boolean');
|
||||
console.log(' extractTextInRegions: OK');
|
||||
|
||||
// --- extractPagesMarkdown ---
|
||||
console.log('Testing extractPagesMarkdown...');
|
||||
|
||||
// omit pages → every page in document order
|
||||
const allPages = extractPagesMarkdown(fixture);
|
||||
assert.equal(allPages.pages.length, 3);
|
||||
assert.deepEqual(allPages.pages.map(p => p.page), [0, 1, 2]);
|
||||
assert.ok(typeof allPages.pages[0].markdown === 'string');
|
||||
assert.equal(typeof allPages.pages[0].needsOcr, 'boolean');
|
||||
assert.ok(Array.isArray(allPages.pagesWithTables));
|
||||
assert.ok(Array.isArray(allPages.pagesWithColumns));
|
||||
assert.ok(Array.isArray(allPages.pagesNeedingOcr));
|
||||
assert.equal(typeof allPages.isComplex, 'boolean');
|
||||
console.log(' extractPagesMarkdown (no pages arg): OK');
|
||||
|
||||
// selected pages preserve caller order
|
||||
const picked = extractPagesMarkdown(fixture, [2, 0]);
|
||||
assert.equal(picked.pages.length, 2);
|
||||
assert.equal(picked.pages[0].page, 2);
|
||||
assert.equal(picked.pages[1].page, 0);
|
||||
console.log(' extractPagesMarkdown with pages: OK');
|
||||
|
||||
// --- Error handling ---
|
||||
console.log('Testing error handling...');
|
||||
assert.throws(() => processPdf(Buffer.from('not a pdf')), /process_pdf/);
|
||||
|
||||
@@ -52,28 +52,6 @@ class PageRegionTexts:
|
||||
"""0-indexed page number."""
|
||||
regions: list[RegionText]
|
||||
|
||||
class PageMarkdown:
|
||||
"""Per-page markdown extraction result."""
|
||||
page: int
|
||||
"""0-indexed page number."""
|
||||
markdown: str
|
||||
"""Formatted markdown for this page (empty string when needs_ocr is True)."""
|
||||
needs_ocr: bool
|
||||
"""True when text on this page is unreliable and OCR should be used instead."""
|
||||
|
||||
class PagesExtractionResult:
|
||||
"""Per-page markdown output with document-wide layout classification."""
|
||||
pages: list[PageMarkdown]
|
||||
"""Per-page markdown results, in the order requested."""
|
||||
pages_with_tables: list[int]
|
||||
"""1-indexed pages where tables were detected."""
|
||||
pages_with_columns: list[int]
|
||||
"""1-indexed pages where multi-column layout was detected."""
|
||||
pages_needing_ocr: list[int]
|
||||
"""1-indexed pages that need OCR."""
|
||||
is_complex: bool
|
||||
"""True if any page has tables or multi-column layout."""
|
||||
|
||||
def process_pdf(path: str, pages: Optional[list[int]] = None) -> PdfResult:
|
||||
"""Process a PDF: detect type, extract text, convert to Markdown."""
|
||||
...
|
||||
@@ -137,31 +115,3 @@ def extract_text_in_regions_bytes(
|
||||
page_regions: List of (page_0indexed, [[x1, y1, x2, y2], ...]) tuples.
|
||||
"""
|
||||
...
|
||||
|
||||
def extract_pages_markdown(
|
||||
path: str,
|
||||
pages: Optional[list[int]] = None,
|
||||
) -> PagesExtractionResult:
|
||||
"""Extract formatted markdown for pages of a PDF, with layout classification.
|
||||
|
||||
Args:
|
||||
path: Path to the PDF file.
|
||||
pages: Optional list of 0-indexed pages. When ``None`` (default), every
|
||||
page is returned in document order. Otherwise, output matches the
|
||||
caller-supplied order.
|
||||
|
||||
Returns:
|
||||
PagesExtractionResult with per-page markdown and document-wide layout
|
||||
classification (tables, columns, OCR needs).
|
||||
"""
|
||||
...
|
||||
|
||||
def extract_pages_markdown_bytes(
|
||||
data: bytes,
|
||||
pages: Optional[list[int]] = None,
|
||||
) -> PagesExtractionResult:
|
||||
"""Extract formatted markdown for pages of a PDF from bytes.
|
||||
|
||||
See :func:`extract_pages_markdown` for details.
|
||||
"""
|
||||
...
|
||||
|
||||
+87
-1909
File diff suppressed because it is too large
Load Diff
@@ -216,7 +216,6 @@ pub(crate) fn extract_page_text_items(
|
||||
let mut marked_content_stack: Vec<MarkedContentEntry> = Vec::new();
|
||||
let mut suppress_glyph_extraction = false;
|
||||
let mut actual_text_start_tm: Option<[f32; 6]> = None; // text matrix at BDC entry
|
||||
let mut actual_text_glyph_tm: Option<[f32; 6]> = None; // text matrix at first glyph inside BDC
|
||||
/// Get the innermost MCID from the marked content stack.
|
||||
fn current_mcid(stack: &[MarkedContentEntry]) -> Option<i64> {
|
||||
stack.iter().rev().find_map(|e| e.mcid)
|
||||
@@ -350,15 +349,8 @@ pub(crate) fn extract_page_text_items(
|
||||
)
|
||||
})
|
||||
});
|
||||
// ActualText: suppress glyph extraction, just advance text matrix.
|
||||
// Capture the FIRST glyph's text matrix as the rendering position
|
||||
// for the ActualText item. Td ops between BDC and the first Tj
|
||||
// may have moved the position to the correct line — the BDC-entry
|
||||
// position (actual_text_start_tm) can be on the previous line.
|
||||
// ActualText: suppress glyph extraction, just advance text matrix
|
||||
if suppress_glyph_extraction {
|
||||
if actual_text_glyph_tm.is_none() {
|
||||
actual_text_glyph_tm = Some(text_matrix);
|
||||
}
|
||||
if let Some(w_ts) = w_ts_opt {
|
||||
text_matrix[4] += w_ts * text_matrix[0];
|
||||
text_matrix[5] += w_ts * text_matrix[1];
|
||||
@@ -433,10 +425,6 @@ pub(crate) fn extract_page_text_items(
|
||||
let font_info = font_widths.get(¤t_font);
|
||||
let is_invisible = (text_rendering_mode == 3 && !include_invisible)
|
||||
|| suppress_glyph_extraction;
|
||||
// Capture first-glyph position for ActualText
|
||||
if suppress_glyph_extraction && actual_text_glyph_tm.is_none() {
|
||||
actual_text_glyph_tm = Some(text_matrix);
|
||||
}
|
||||
|
||||
// Compute space threshold based on font metrics when available
|
||||
let space_threshold = if let Some(font_info) = font_info {
|
||||
@@ -712,7 +700,6 @@ pub(crate) fn extract_page_text_items(
|
||||
if actual_text.is_some() {
|
||||
suppress_glyph_extraction = true;
|
||||
actual_text_start_tm = Some(text_matrix);
|
||||
actual_text_glyph_tm = None; // reset — will be captured at first Tj/TJ
|
||||
}
|
||||
marked_content_stack.push(MarkedContentEntry { actual_text, mcid });
|
||||
}
|
||||
@@ -720,13 +707,8 @@ pub(crate) fn extract_page_text_items(
|
||||
// End Marked Content — emit ActualText item with correct width
|
||||
if let Some(entry) = marked_content_stack.pop() {
|
||||
if let Some(at) = entry.actual_text {
|
||||
// Use the first-glyph position (if available) instead of the
|
||||
// BDC-entry position. Td operators between BDC and the first
|
||||
// Tj may have moved the text position to the correct line —
|
||||
// the BDC-entry position can be on the previous line.
|
||||
let glyph_tm = actual_text_glyph_tm.take();
|
||||
let entry_tm = actual_text_start_tm.take();
|
||||
if let Some(start_tm) = glyph_tm.or(entry_tm) {
|
||||
// Compute width from text matrix advancement during BDC..EMC
|
||||
if let Some(start_tm) = actual_text_start_tm.take() {
|
||||
let combined = multiply_matrices(&start_tm, &ctm);
|
||||
if combined[0].abs() >= combined[1].abs() {
|
||||
rotation_votes.horizontal += 1;
|
||||
|
||||
+7
-48
@@ -1,9 +1,3 @@
|
||||
// Rust 1.95 introduced collapsible_match for `if` inside match arms.
|
||||
// The content-stream parsers use this pattern extensively (match on operator
|
||||
// name, then check `in_text_block && !op.operands.is_empty()`). Collapsing
|
||||
// these into match guards would hurt readability. Allow crate-wide.
|
||||
#![allow(clippy::collapsible_match)]
|
||||
|
||||
//! Smart PDF detection and text extraction using lopdf
|
||||
//!
|
||||
//! # Quick start
|
||||
@@ -336,17 +330,13 @@ pub struct PagesExtractionResult {
|
||||
pub is_complex: bool,
|
||||
}
|
||||
|
||||
/// Extract formatted markdown for pages of a PDF, with layout
|
||||
/// Extract formatted markdown for specific pages of a PDF, with layout
|
||||
/// classification metadata.
|
||||
///
|
||||
/// Unlike [`process_pdf_mem`] which returns one concatenated markdown string,
|
||||
/// this returns per-page markdown so callers can mix direct extraction
|
||||
/// (for simple text pages) with GPU OCR (for complex/scanned pages).
|
||||
///
|
||||
/// When `pages` is `None`, every page (0-indexed, in document order) is
|
||||
/// returned. When `Some(&[...])`, only the listed 0-indexed pages are
|
||||
/// returned, in the caller's order.
|
||||
///
|
||||
/// Font statistics are computed from the full document so header
|
||||
/// detection thresholds are consistent regardless of which pages are
|
||||
/// requested. Per-page `needs_ocr` is set when the page has GID-encoded
|
||||
@@ -356,7 +346,7 @@ pub struct PagesExtractionResult {
|
||||
/// at near-zero cost since the items/rects/lines are already in memory.
|
||||
pub fn extract_pages_markdown_mem(
|
||||
buffer: &[u8],
|
||||
pages: Option<&[u32]>,
|
||||
pages: &[u32],
|
||||
) -> Result<PagesExtractionResult, PdfError> {
|
||||
validate_pdf_bytes(buffer)?;
|
||||
let (doc, page_count) = load_document_from_mem(buffer)?;
|
||||
@@ -372,20 +362,10 @@ pub fn extract_pages_markdown_mem(
|
||||
// Compute font stats from full document (cross-page consistency).
|
||||
let font_stats = markdown::analysis::calculate_font_stats_from_items(&all_items);
|
||||
|
||||
// When caller doesn't specify pages, return every page in document order.
|
||||
let all_pages: Vec<u32>;
|
||||
let pages_slice: &[u32] = match pages {
|
||||
Some(p) => p,
|
||||
None => {
|
||||
all_pages = (0..page_count).collect();
|
||||
&all_pages
|
||||
}
|
||||
};
|
||||
|
||||
let mut results = Vec::with_capacity(pages_slice.len());
|
||||
let mut results = Vec::with_capacity(pages.len());
|
||||
let mut pages_needing_ocr = Vec::new();
|
||||
|
||||
for &page_0idx in pages_slice {
|
||||
for &page_0idx in pages {
|
||||
// Out-of-range pages → empty + needs_ocr
|
||||
if page_0idx >= page_count {
|
||||
pages_needing_ocr.push(page_0idx + 1);
|
||||
@@ -458,20 +438,6 @@ pub fn extract_pages_markdown_mem(
|
||||
})
|
||||
}
|
||||
|
||||
/// Path-based wrapper for [`extract_pages_markdown_mem`].
|
||||
///
|
||||
/// Reads the PDF from disk and extracts per-page markdown. Pass `None` for
|
||||
/// `pages` to return every page in document order, or `Some(&[...])` to
|
||||
/// restrict to specific 0-indexed pages (in caller-supplied order).
|
||||
pub fn extract_pages_markdown<P: AsRef<Path>>(
|
||||
path: P,
|
||||
pages: Option<&[u32]>,
|
||||
) -> Result<PagesExtractionResult, PdfError> {
|
||||
validate_pdf_file(&path)?;
|
||||
let buffer = std::fs::read(path.as_ref())?;
|
||||
extract_pages_markdown_mem(&buffer, pages)
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Region-based text extraction (for hybrid OCR pipelines)
|
||||
// =========================================================================
|
||||
@@ -1810,26 +1776,19 @@ fn compute_layout_complexity(
|
||||
markdown::filter_lines_to_band(lines, page, x_lo, x_hi)
|
||||
};
|
||||
|
||||
// TOC pages route through the table detector but render as flat
|
||||
// lists. They aren't tables in any user-facing sense, so don't
|
||||
// count them toward LayoutComplexity (would also trip the
|
||||
// table-page guard in column detection below).
|
||||
let has_data_table =
|
||||
|tables: &[tables::Table]| tables.iter().any(|t| t.kind == tables::TableKind::Data);
|
||||
|
||||
let (rect_tables, _) = tables::detect_tables_from_rects(&band_items, &band_rects, page);
|
||||
if has_data_table(&rect_tables) {
|
||||
if !rect_tables.is_empty() {
|
||||
found_table = true;
|
||||
break;
|
||||
}
|
||||
let line_tables = tables::detect_tables_from_lines(&band_items, &band_lines, page);
|
||||
if has_data_table(&line_tables) {
|
||||
if !line_tables.is_empty() {
|
||||
found_table = true;
|
||||
break;
|
||||
}
|
||||
// Heuristic fallback for borderless tables
|
||||
let heuristic_tables = tables::detect_tables(&band_items, base_size, false);
|
||||
if has_data_table(&heuristic_tables) {
|
||||
if !heuristic_tables.is_empty() {
|
||||
found_table = true;
|
||||
break;
|
||||
}
|
||||
|
||||
-121
@@ -146,67 +146,6 @@ impl PyPageRegionTexts {
|
||||
// Text item wrapper
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Per-page markdown extraction result.
|
||||
#[pyclass(name = "PageMarkdown")]
|
||||
#[derive(Clone)]
|
||||
pub struct PyPageMarkdown {
|
||||
/// 0-indexed page number.
|
||||
#[pyo3(get)]
|
||||
pub page: u32,
|
||||
/// Formatted markdown for this page.
|
||||
#[pyo3(get)]
|
||||
pub markdown: String,
|
||||
/// True when text on this page is unreliable (GID-encoded fonts,
|
||||
/// encoding issues, garbage text, or empty extraction).
|
||||
#[pyo3(get)]
|
||||
pub needs_ocr: bool,
|
||||
}
|
||||
|
||||
#[pymethods]
|
||||
impl PyPageMarkdown {
|
||||
fn __repr__(&self) -> String {
|
||||
format!(
|
||||
"PageMarkdown(page={}, markdown='{}', needs_ocr={})",
|
||||
self.page,
|
||||
self.markdown.chars().take(40).collect::<String>(),
|
||||
self.needs_ocr
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// Combined per-page markdown extraction and layout classification result.
|
||||
#[pyclass(name = "PagesExtractionResult")]
|
||||
#[derive(Clone)]
|
||||
pub struct PyPagesExtractionResult {
|
||||
/// Per-page markdown results, in the order requested.
|
||||
#[pyo3(get)]
|
||||
pub pages: Vec<PyPageMarkdown>,
|
||||
/// 1-indexed pages where tables were detected.
|
||||
#[pyo3(get)]
|
||||
pub pages_with_tables: Vec<u32>,
|
||||
/// 1-indexed pages where multi-column layout was detected.
|
||||
#[pyo3(get)]
|
||||
pub pages_with_columns: Vec<u32>,
|
||||
/// 1-indexed pages that need OCR (scanned/image-based or unreliable text).
|
||||
#[pyo3(get)]
|
||||
pub pages_needing_ocr: Vec<u32>,
|
||||
/// True if any page has tables or columns.
|
||||
#[pyo3(get)]
|
||||
pub is_complex: bool,
|
||||
}
|
||||
|
||||
#[pymethods]
|
||||
impl PyPagesExtractionResult {
|
||||
fn __repr__(&self) -> String {
|
||||
format!(
|
||||
"PagesExtractionResult(pages={}, pages_with_tables={:?}, is_complex={})",
|
||||
self.pages.len(),
|
||||
self.pages_with_tables,
|
||||
self.is_complex
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// A positioned text item extracted from a PDF.
|
||||
#[pyclass(name = "TextItem")]
|
||||
#[derive(Clone)]
|
||||
@@ -341,24 +280,6 @@ fn parse_page_regions(
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn to_py_pages_result(r: crate::PagesExtractionResult) -> PyPagesExtractionResult {
|
||||
PyPagesExtractionResult {
|
||||
pages: r
|
||||
.pages
|
||||
.into_iter()
|
||||
.map(|p| PyPageMarkdown {
|
||||
page: p.page,
|
||||
markdown: p.markdown,
|
||||
needs_ocr: p.needs_ocr,
|
||||
})
|
||||
.collect(),
|
||||
pages_with_tables: r.pages_with_tables,
|
||||
pages_with_columns: r.pages_with_columns,
|
||||
pages_needing_ocr: r.pages_needing_ocr,
|
||||
is_complex: r.is_complex,
|
||||
}
|
||||
}
|
||||
|
||||
fn convert_region_results(results: Vec<crate::PageRegionResult>) -> Vec<PyPageRegionTexts> {
|
||||
results
|
||||
.into_iter()
|
||||
@@ -521,44 +442,6 @@ fn extract_text_in_regions_bytes(
|
||||
Ok(convert_region_results(results))
|
||||
}
|
||||
|
||||
/// Extract formatted markdown for pages of a PDF file, with layout
|
||||
/// classification metadata.
|
||||
///
|
||||
/// Returns per-page markdown and classification data (tables, columns,
|
||||
/// OCR needs) from a single parse. Font statistics are computed from the
|
||||
/// full document so header detection is consistent across pages.
|
||||
///
|
||||
/// Args:
|
||||
/// path: Path to the PDF file.
|
||||
/// pages: Optional list of 0-indexed pages. When None (default), every
|
||||
/// page is returned in document order. When provided, output
|
||||
/// matches the caller-supplied order.
|
||||
///
|
||||
/// Returns:
|
||||
/// PagesExtractionResult with per-page markdown and classification data.
|
||||
#[pyfunction]
|
||||
#[pyo3(signature = (path, pages=None))]
|
||||
fn extract_pages_markdown(
|
||||
path: &str,
|
||||
pages: Option<Vec<u32>>,
|
||||
) -> PyResult<PyPagesExtractionResult> {
|
||||
let result = crate::extract_pages_markdown(path, pages.as_deref()).map_err(to_py_err)?;
|
||||
Ok(to_py_pages_result(result))
|
||||
}
|
||||
|
||||
/// Extract formatted markdown for pages of a PDF from bytes.
|
||||
///
|
||||
/// See [`extract_pages_markdown`] for details.
|
||||
#[pyfunction]
|
||||
#[pyo3(signature = (data, pages=None))]
|
||||
fn extract_pages_markdown_bytes(
|
||||
data: &[u8],
|
||||
pages: Option<Vec<u32>>,
|
||||
) -> PyResult<PyPagesExtractionResult> {
|
||||
let result = crate::extract_pages_markdown_mem(data, pages.as_deref()).map_err(to_py_err)?;
|
||||
Ok(to_py_pages_result(result))
|
||||
}
|
||||
|
||||
/// Python module definition.
|
||||
#[pymodule]
|
||||
fn pdf_inspector(m: &Bound<'_, PyModule>) -> PyResult<()> {
|
||||
@@ -567,8 +450,6 @@ fn pdf_inspector(m: &Bound<'_, PyModule>) -> PyResult<()> {
|
||||
m.add_class::<PyTextItem>()?;
|
||||
m.add_class::<PyRegionText>()?;
|
||||
m.add_class::<PyPageRegionTexts>()?;
|
||||
m.add_class::<PyPageMarkdown>()?;
|
||||
m.add_class::<PyPagesExtractionResult>()?;
|
||||
m.add_function(wrap_pyfunction!(process_pdf, m)?)?;
|
||||
m.add_function(wrap_pyfunction!(process_pdf_bytes, m)?)?;
|
||||
m.add_function(wrap_pyfunction!(detect_pdf, m)?)?;
|
||||
@@ -581,7 +462,5 @@ fn pdf_inspector(m: &Bound<'_, PyModule>) -> PyResult<()> {
|
||||
m.add_function(wrap_pyfunction!(extract_text_with_positions_bytes, m)?)?;
|
||||
m.add_function(wrap_pyfunction!(extract_text_in_regions, m)?)?;
|
||||
m.add_function(wrap_pyfunction!(extract_text_in_regions_bytes, m)?)?;
|
||||
m.add_function(wrap_pyfunction!(extract_pages_markdown, m)?)?;
|
||||
m.add_function(wrap_pyfunction!(extract_pages_markdown_bytes, m)?)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
+64
-523
@@ -581,14 +581,8 @@ fn detect_table_in_region(items: &[(usize, &TextItem)], mode: TableDetectionMode
|
||||
// Validation 1: some rows should have content in first column.
|
||||
// Use a lower threshold (25%) for tables with wrapped cells where
|
||||
// continuation lines leave the first column empty.
|
||||
// Skip when cells form a narrow TOC pattern: hierarchical entries indented
|
||||
// across multiple X levels leave the leftmost column sparse (only top-level
|
||||
// chapters land there) but the structure is still a valid TOC. Narrow only
|
||||
// (<=5 cols) — wide multi-column TOCs (e.g. 2-up indices) would render
|
||||
// poorly through format_toc_as_list, which assumes one entry per row.
|
||||
let rows_with_first_col = cells.iter().filter(|row| !row[0].is_empty()).count();
|
||||
let is_narrow_toc = columns.len() <= 5 && is_table_of_contents(&cells);
|
||||
if rows_with_first_col < rows.len() / 4 && !is_narrow_toc {
|
||||
if rows_with_first_col < rows.len() / 4 {
|
||||
log::debug!(
|
||||
" validation 1 fail: {}/{} rows have first col",
|
||||
rows_with_first_col,
|
||||
@@ -659,24 +653,15 @@ fn detect_table_in_region(items: &[(usize, &TextItem)], mode: TableDetectionMode
|
||||
return None;
|
||||
}
|
||||
|
||||
// Validation 8: Reject paragraph-like content falsely detected as tables.
|
||||
// TOC pages with deep indentation (top-level chapters in col 0, subsections
|
||||
// in cols 1-3, page numbers in last col) leave most cells empty and trip
|
||||
// the paragraph heuristic; TOC shape is a safer signal here. Narrow only
|
||||
// — see narrow-TOC rationale at validation 1.
|
||||
if is_paragraph_content(&cells) && !is_narrow_toc {
|
||||
log::debug!(" validation 9 fail: paragraph content");
|
||||
// Validation 8: Check for Table of Contents pattern
|
||||
if is_table_of_contents(&cells) {
|
||||
log::debug!(" validation 8 fail: table of contents");
|
||||
return None;
|
||||
}
|
||||
|
||||
// Validation 9: Reject wide "index" layouts where every cell carries a
|
||||
// full "label ... page" fragment (back-of-book IRS-style indices).
|
||||
// These render poorly in any structured form; text flow is the best
|
||||
// fallback. Narrow dot-leader TOCs (2-3 cols) are kept so format.rs
|
||||
// can emit them as a per-row flat list with titles tab-joined to page
|
||||
// numbers.
|
||||
if is_inline_leader_index(&cells) {
|
||||
log::debug!(" validation 9 fail: inline-leader index");
|
||||
// Validation 9: Reject paragraph-like content falsely detected as tables
|
||||
if is_paragraph_content(&cells) {
|
||||
log::debug!(" validation 9 fail: paragraph content");
|
||||
return None;
|
||||
}
|
||||
|
||||
@@ -687,7 +672,12 @@ fn detect_table_in_region(items: &[(usize, &TextItem)], mode: TableDetectionMode
|
||||
item_indices.len()
|
||||
);
|
||||
|
||||
Some(Table::new(columns, rows, cells, item_indices))
|
||||
Some(Table {
|
||||
columns,
|
||||
rows,
|
||||
cells,
|
||||
item_indices,
|
||||
})
|
||||
}
|
||||
|
||||
/// Check if this looks like a key-value pair layout rather than a table
|
||||
@@ -908,247 +898,77 @@ fn looks_like_number(s: &str) -> bool {
|
||||
&& s.chars().any(|c| c.is_ascii_digit())
|
||||
}
|
||||
|
||||
/// Check if this looks like a Table of Contents (either style).
|
||||
///
|
||||
/// Used by format.rs to render TOCs as flat lists instead of markdown tables.
|
||||
pub fn is_table_of_contents(cells: &[Vec<String>]) -> bool {
|
||||
is_dot_leader_toc(cells) || is_tabular_toc(cells)
|
||||
}
|
||||
|
||||
/// Dot-leader TOC: any "Chapter 1 ........ 42" style with explicit leader
|
||||
/// dots. Covers both narrow 2-3 col TOCs (where the leader is a dedicated
|
||||
/// cell) and wide indices (where each cell encodes a full "label ... page"
|
||||
/// fragment). Used by format.rs to render as a flat list.
|
||||
pub(super) fn is_dot_leader_toc(cells: &[Vec<String>]) -> bool {
|
||||
has_structural_dot_leader(cells) || is_inline_leader_index(cells)
|
||||
}
|
||||
|
||||
/// Rows with a dedicated dots-only cell flanked by label + number (2-3 col
|
||||
/// TOC layout). Format.rs handles these well via per-row flat-list
|
||||
/// rendering; they should NOT be rejected at detect time.
|
||||
fn has_structural_dot_leader(cells: &[Vec<String>]) -> bool {
|
||||
/// Check if this looks like a Table of Contents
|
||||
/// TOCs have characteristic patterns: leader dots, page numbers, section names
|
||||
fn is_table_of_contents(cells: &[Vec<String>]) -> bool {
|
||||
if cells.is_empty() {
|
||||
return false;
|
||||
}
|
||||
let structural_rows = cells.iter().filter(|row| row_has_dot_leader(row)).count();
|
||||
structural_rows as f32 / cells.len() as f32 >= 0.3
|
||||
}
|
||||
|
||||
/// Wide index layout: each cell holds a full "label ... page" fragment
|
||||
/// because the column detector kept multi-column indices as single cells.
|
||||
/// These render poorly both as markdown tables (column boundaries are
|
||||
/// arbitrary) and as flat lists (each row holds 3+ separate index
|
||||
/// entries). Reject these at detect time so they fall back to the page's
|
||||
/// normal text flow.
|
||||
pub(super) fn is_inline_leader_index(cells: &[Vec<String>]) -> bool {
|
||||
let mut inline_cells = 0;
|
||||
let mut total_nonempty = 0;
|
||||
let num_cols = cells[0].len();
|
||||
let mut dot_cells = 0;
|
||||
let mut page_number_cells = 0;
|
||||
let mut total_cells = 0;
|
||||
// Track which columns contain dots vs numbers to distinguish
|
||||
// TOC (dots span middle, page number at end) from data tables
|
||||
// (dots only in label column, many number columns).
|
||||
let mut dot_cols = vec![0u32; num_cols];
|
||||
let mut numeric_cols = vec![0u32; num_cols];
|
||||
|
||||
for row in cells {
|
||||
for cell in row {
|
||||
for (ci, cell) in row.iter().enumerate() {
|
||||
let trimmed = cell.trim();
|
||||
if trimmed.is_empty() {
|
||||
continue;
|
||||
}
|
||||
total_nonempty += 1;
|
||||
if cell_is_inline_leader(trimmed) {
|
||||
inline_cells += 1;
|
||||
total_cells += 1;
|
||||
|
||||
// Check for leader dots (sequences of periods)
|
||||
// TOCs often have "........" or ". . . ." patterns
|
||||
let dot_count = trimmed.chars().filter(|&c| c == '.').count();
|
||||
let is_mostly_dots = dot_count > trimmed.len() / 2 && dot_count >= 3;
|
||||
if is_mostly_dots {
|
||||
dot_cells += 1;
|
||||
if ci < num_cols {
|
||||
dot_cols[ci] += 1;
|
||||
}
|
||||
}
|
||||
|
||||
// Check for standalone page numbers (1-4 digits, possibly with spaces)
|
||||
let digits_only: String = trimmed.chars().filter(|c| !c.is_whitespace()).collect();
|
||||
if digits_only.len() <= 4
|
||||
&& !digits_only.is_empty()
|
||||
&& digits_only.chars().all(|c| c.is_ascii_digit())
|
||||
{
|
||||
page_number_cells += 1;
|
||||
if ci < num_cols {
|
||||
numeric_cols[ci] += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
total_nonempty >= 4 && inline_cells as f32 / total_nonempty as f32 >= 0.25
|
||||
}
|
||||
|
||||
/// A row with a dot-leader. Accepts two layouts:
|
||||
/// 1. A dedicated dots-only cell ("....") with a text label somewhere
|
||||
/// to its left and a page number somewhere to its right.
|
||||
/// 2. A "title ... " cell (trailing leader dots glued to the title)
|
||||
/// with a page number elsewhere in the same row.
|
||||
fn row_has_dot_leader(row: &[String]) -> bool {
|
||||
let has_page_number = row.iter().any(|c| row_cell_is_page_number(c));
|
||||
|
||||
for (ci, cell) in row.iter().enumerate() {
|
||||
let trimmed = cell.trim();
|
||||
|
||||
// Pattern 1: dedicated dots-only cell.
|
||||
let dot_count = trimmed.chars().filter(|&c| c == '.').count();
|
||||
let is_mostly_dots = dot_count >= 3
|
||||
&& dot_count > trimmed.len() / 2
|
||||
&& trimmed.chars().all(|c| c == '.' || c.is_whitespace());
|
||||
if is_mostly_dots {
|
||||
let has_label_left = row[..ci].iter().any(|c| {
|
||||
let t = c.trim();
|
||||
!t.is_empty() && t.chars().any(|ch| ch.is_alphabetic())
|
||||
});
|
||||
if has_label_left && has_page_number {
|
||||
return true;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// Pattern 2: cell ends with a trailing " ... " run after a label.
|
||||
if has_page_number && cell_has_trailing_leader(trimmed) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
/// Cell ends with a run of ≥3 dots preceded by alphabetic text and a
|
||||
/// space — the "Title ... " layout where the leader is glued to the name.
|
||||
/// Alphabetic (not alphanumeric) so that data-table row labels like
|
||||
/// "1973 ... " do not register as titles.
|
||||
fn cell_has_trailing_leader(cell: &str) -> bool {
|
||||
let trimmed = cell.trim_end();
|
||||
if !trimmed.ends_with('.') {
|
||||
return false;
|
||||
}
|
||||
let without_dots = trimmed.trim_end_matches('.');
|
||||
let dot_run = trimmed.len() - without_dots.len();
|
||||
if dot_run < 3 {
|
||||
return false;
|
||||
}
|
||||
// Require a space before the dot run (rules out "etc..." / "Mr...") and
|
||||
// at least one alphabetic char (rules out "1973 ... " data-row labels).
|
||||
without_dots.ends_with(' ') && without_dots.trim().chars().any(|c| c.is_alphabetic())
|
||||
}
|
||||
|
||||
/// Page-number shape: single ≤4-digit integer, a ", "-separated list of
|
||||
/// ≤4-digit integers ("18, 36, 107"), or a dashed section-page ID
|
||||
/// ("A-1", "5-21"). Rejects decimal cells ("4. 0"), thousands-separated
|
||||
/// values ("189,164"), and other long numeric data that appears in
|
||||
/// statistical tables.
|
||||
fn row_cell_is_page_number(cell: &str) -> bool {
|
||||
let t = cell.trim();
|
||||
if t.is_empty() {
|
||||
return false;
|
||||
}
|
||||
if looks_like_section_page_id(t) {
|
||||
return true;
|
||||
}
|
||||
// Page list: ", " separator (with space) distinguishes real page lists
|
||||
// from thousands-separated numbers like "189,164".
|
||||
let parts: Vec<&str> = t.split(", ").collect();
|
||||
parts
|
||||
.iter()
|
||||
.all(|p| !p.is_empty() && p.len() <= 4 && p.chars().all(|c| c.is_ascii_digit()))
|
||||
}
|
||||
|
||||
/// A cell shaped like an index leader fragment. Accepts two forms:
|
||||
/// - "text ... number" — label + dots + page number in one cell
|
||||
/// - "... number" — bare leader + number (row where the label
|
||||
/// landed in a separate column)
|
||||
///
|
||||
/// Both only count if followed by pure numeric content (optionally
|
||||
/// comma-separated page lists like "127, 213").
|
||||
fn cell_is_inline_leader(cell: &str) -> bool {
|
||||
let cell = cell.trim();
|
||||
|
||||
// Find the first "..." run. Surrounding-whitespace checks below
|
||||
// reject intra-word ellipses ("etc...").
|
||||
let idx = match cell.match_indices("...").next() {
|
||||
Some((i, _)) => i,
|
||||
None => return false,
|
||||
};
|
||||
|
||||
let before = &cell[..idx];
|
||||
let after_dots = &cell[idx + 3..];
|
||||
// Allow extra dots (e.g. "....") by skipping any additional '.'
|
||||
let after = after_dots.trim_start_matches('.');
|
||||
|
||||
// Require space (or start-of-cell) before the dots and space/digit
|
||||
// after — blocks intra-word ellipses.
|
||||
let before_ok = before.is_empty() || before.ends_with(' ');
|
||||
let after_ok = after.starts_with(' ') || after.is_empty();
|
||||
if !before_ok || !after_ok {
|
||||
if total_cells == 0 {
|
||||
return false;
|
||||
}
|
||||
|
||||
let after_trim = after.trim();
|
||||
if after_trim.is_empty() {
|
||||
return false;
|
||||
}
|
||||
// Tail must be purely numeric/page-list content.
|
||||
let tail_numeric = after_trim
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_digit() || matches!(c, ',' | ' ' | '.' | '-' | '$'))
|
||||
&& after_trim.chars().any(|c| c.is_ascii_digit());
|
||||
if !tail_numeric {
|
||||
// Data tables with dot leaders (e.g. "1973....") have dots concentrated
|
||||
// in one column (the label column) while many other columns contain numbers.
|
||||
// True TOCs have dots spanning the middle and one page-number column at the end.
|
||||
// If dots are confined to ≤1 column AND there are ≥3 columns with numbers,
|
||||
// this is a data table, not a TOC.
|
||||
let cols_with_dots = dot_cols.iter().filter(|&&c| c >= 2).count();
|
||||
let cols_with_numbers = numeric_cols.iter().filter(|&&c| c >= 2).count();
|
||||
if cols_with_dots <= 1 && cols_with_numbers >= 3 {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Either we have a label before, or the leader is bare (starts the cell)
|
||||
// — both are legitimate index fragments.
|
||||
before.chars().any(|c| c.is_alphabetic()) || before.trim().is_empty()
|
||||
}
|
||||
// If a significant portion of cells are dots or page numbers, it's likely a TOC
|
||||
let dot_ratio = dot_cells as f32 / total_cells as f32;
|
||||
let page_num_ratio = page_number_cells as f32 / total_cells as f32;
|
||||
|
||||
/// Dot-less tabular TOC: tagged PDFs emit entries as rows where the first
|
||||
/// column starts with a dotted section number (e.g. "4.3.1 Something") and
|
||||
/// the last column is one or more page numbers. These have no leader dots
|
||||
/// and benefit from flat-list formatting (page numbers aligned to titles).
|
||||
pub(super) fn is_tabular_toc(cells: &[Vec<String>]) -> bool {
|
||||
if cells.is_empty() {
|
||||
return false;
|
||||
}
|
||||
let num_cols = cells[0].len();
|
||||
if num_cols < 2 || cells.len() < 4 {
|
||||
return false;
|
||||
}
|
||||
|
||||
let section_rows = cells
|
||||
.iter()
|
||||
.filter(|row| {
|
||||
row.iter()
|
||||
.find(|c| !c.trim().is_empty())
|
||||
.is_some_and(|c| starts_with_section_number(c.trim()))
|
||||
})
|
||||
.count();
|
||||
|
||||
let last_col = num_cols - 1;
|
||||
let (last_filled, last_page_num) = cells.iter().fold((0u32, 0u32), |(f, n), row| {
|
||||
let cell = row.get(last_col).map(|s| s.trim()).unwrap_or("");
|
||||
if cell.is_empty() {
|
||||
return (f, n);
|
||||
}
|
||||
let is_page_nums = cell
|
||||
.split_whitespace()
|
||||
.all(|tok| !tok.is_empty() && tok.chars().all(|c| c.is_ascii_digit()));
|
||||
(f + 1, n + if is_page_nums { 1 } else { 0 })
|
||||
});
|
||||
|
||||
let section_ratio = section_rows as f32 / cells.len() as f32;
|
||||
let page_num_last_ratio = if last_filled > 0 {
|
||||
last_page_num as f32 / last_filled as f32
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
|
||||
section_ratio >= 0.6 && last_filled >= 3 && page_num_last_ratio >= 0.7
|
||||
}
|
||||
|
||||
/// Matches dashed section-page identifiers used in technical manuals:
|
||||
/// "5-21", "A-1", "B--3", "TC-2". At least one ASCII digit is required.
|
||||
fn looks_like_section_page_id(s: &str) -> bool {
|
||||
let ok = s
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_digit() || c.is_ascii_uppercase() || c == '-');
|
||||
ok && s.chars().any(|c| c.is_ascii_digit())
|
||||
}
|
||||
|
||||
/// Returns true when the leading token looks like a dotted section number:
|
||||
/// "1", "1.2", "1.2.3", "4.3.1.2" — integer components joined by dots,
|
||||
/// with at least one dot (single-number prefixes are too ambiguous).
|
||||
fn starts_with_section_number(s: &str) -> bool {
|
||||
let Some(first) = s.split_whitespace().next() else {
|
||||
return false;
|
||||
};
|
||||
let first = first.trim_end_matches('.');
|
||||
let parts: Vec<&str> = first.split('.').collect();
|
||||
if parts.len() < 2 || parts.len() > 6 {
|
||||
return false;
|
||||
}
|
||||
parts
|
||||
.iter()
|
||||
.all(|p| !p.is_empty() && p.len() <= 3 && p.chars().all(|c| c.is_ascii_digit()))
|
||||
// TOC typically has >15% dot cells and >10% page number cells
|
||||
dot_ratio > 0.15 || (dot_ratio > 0.05 && page_num_ratio > 0.15)
|
||||
}
|
||||
|
||||
/// Check if detected "table" cells are actually paragraph text fragments.
|
||||
@@ -1605,283 +1425,4 @@ mod tests {
|
||||
"data table with dot-leader labels should not be rejected as TOC"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_table_of_contents_accepts_hierarchical_indented_toc() {
|
||||
// Mythos system card pages 4-5: top-level chapters indent at col 0,
|
||||
// subsections at cols 1-2, leaving col 0 mostly empty (only ~10% of
|
||||
// rows). Validation 1 was rejecting these even though the structure
|
||||
// is unambiguously a TOC.
|
||||
let cells = vec![
|
||||
vec!["Abstract".to_string(), String::new(), "3".to_string()],
|
||||
vec![
|
||||
"1 Introduction".to_string(),
|
||||
String::new(),
|
||||
"10".to_string(),
|
||||
],
|
||||
vec![
|
||||
String::new(),
|
||||
"1.1 Model training".to_string(),
|
||||
"11".to_string(),
|
||||
],
|
||||
vec![
|
||||
String::new(),
|
||||
"1.1.1 Training data".to_string(),
|
||||
"11".to_string(),
|
||||
],
|
||||
vec![
|
||||
String::new(),
|
||||
"1.1.2 Crowd workers".to_string(),
|
||||
"12".to_string(),
|
||||
],
|
||||
vec![
|
||||
String::new(),
|
||||
"1.2 Release decision".to_string(),
|
||||
"13".to_string(),
|
||||
],
|
||||
vec![
|
||||
"2 RSP evaluations".to_string(),
|
||||
String::new(),
|
||||
"16".to_string(),
|
||||
],
|
||||
vec![
|
||||
String::new(),
|
||||
"2.1 RSP risk assessment".to_string(),
|
||||
"16".to_string(),
|
||||
],
|
||||
vec![String::new(), "2.1.1 Context".to_string(), "16".to_string()],
|
||||
vec![
|
||||
String::new(),
|
||||
"2.2 CB evaluations".to_string(),
|
||||
"20".to_string(),
|
||||
],
|
||||
];
|
||||
assert!(
|
||||
is_table_of_contents(&cells),
|
||||
"hierarchical TOC with sparse col 0 should still be detected"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_table_of_contents_rejects_dotless_toc() {
|
||||
// Tabular TOC without leader dots: first column starts with dotted
|
||||
// section numbers, last column is page numbers. Pattern from
|
||||
// Mythos system card pages 6-8.
|
||||
let cells = vec![
|
||||
vec![
|
||||
"4.3 Case studies and targeted evaluations".to_string(),
|
||||
String::new(),
|
||||
"86".to_string(),
|
||||
],
|
||||
vec![
|
||||
"4.3.1 Destructive or reckless actions".to_string(),
|
||||
"4.3.1.1 Synthetic-backend evaluation".to_string(),
|
||||
"86 86".to_string(),
|
||||
],
|
||||
vec![
|
||||
"4.3.2 Adherence to constitution".to_string(),
|
||||
"4.3.2.1 Overview".to_string(),
|
||||
"89 89".to_string(),
|
||||
],
|
||||
vec![
|
||||
"4.3.3 Honesty and hallucinations".to_string(),
|
||||
"4.3.3.1 Factual hallucinations".to_string(),
|
||||
"93 94".to_string(),
|
||||
],
|
||||
vec![
|
||||
"4.4 Capability evaluations".to_string(),
|
||||
String::new(),
|
||||
"101".to_string(),
|
||||
],
|
||||
];
|
||||
assert!(
|
||||
is_table_of_contents(&cells),
|
||||
"dot-less TOC with section numbers + page numbers should be rejected"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dot_leader_toc_accepts_short_inline_leaders() {
|
||||
// Index-style cells where the full "label ... number" pattern is
|
||||
// preserved in a single cell (IRS Publication 17 back-of-book index).
|
||||
let cells = vec![
|
||||
vec!["Child tax credit ... 235".to_string(), String::new()],
|
||||
vec!["Church employee ... 252".to_string(), String::new()],
|
||||
vec!["Citizens outside the U.S ... 6".to_string(), String::new()],
|
||||
vec![
|
||||
"Claim for refund ... 18, 36, 107".to_string(),
|
||||
String::new(),
|
||||
],
|
||||
vec!["Clergy ... 7, 52".to_string(), String::new()],
|
||||
];
|
||||
assert!(is_dot_leader_toc(&cells));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dot_leader_toc_allows_ellipsis_data_table() {
|
||||
// Data tables using "..." as a row-omission marker must not be
|
||||
// mistaken for dot-leader TOCs. Based on MCF5235RM QSPI RAM layout.
|
||||
let cells = vec![
|
||||
vec![
|
||||
"0x00".to_string(),
|
||||
"QTR0".to_string(),
|
||||
"Transmit RAM".to_string(),
|
||||
],
|
||||
vec!["0x01".to_string(), "QTR1".to_string(), String::new()],
|
||||
vec![
|
||||
"...".to_string(),
|
||||
"...".to_string(),
|
||||
"16 bits wide".to_string(),
|
||||
],
|
||||
vec!["0x0F".to_string(), "QTR15".to_string(), String::new()],
|
||||
vec![
|
||||
"0x10".to_string(),
|
||||
"QRR0".to_string(),
|
||||
"Receive RAM".to_string(),
|
||||
],
|
||||
vec!["0x11".to_string(), "QRR1".to_string(), String::new()],
|
||||
vec![
|
||||
"...".to_string(),
|
||||
"...".to_string(),
|
||||
"16 bits wide".to_string(),
|
||||
],
|
||||
vec!["0x1F".to_string(), "QRR15".to_string(), String::new()],
|
||||
];
|
||||
assert!(
|
||||
!is_dot_leader_toc(&cells),
|
||||
"ellipsis markers in a data table should not match TOC detection"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dot_leader_toc_rejects_year_row_data_table() {
|
||||
// ERP-2025 economic data tables: year labels with trailing " ... ",
|
||||
// a final " ... " column, and decimal-looking numeric cells. The
|
||||
// detection previously classified these as dot-leader TOCs and
|
||||
// routed them through flat-list formatting, destroying the grid.
|
||||
let cells = vec![
|
||||
vec![
|
||||
"1973 ... ".to_string(),
|
||||
"4. 0".to_string(),
|
||||
"1. 8".to_string(),
|
||||
"0. 4".to_string(),
|
||||
"3. 2".to_string(),
|
||||
" ... ".to_string(),
|
||||
],
|
||||
vec![
|
||||
"1974 ... ".to_string(),
|
||||
"–1. 9".to_string(),
|
||||
"–1. 6".to_string(),
|
||||
"–5. 6".to_string(),
|
||||
"2. 4".to_string(),
|
||||
" ... ".to_string(),
|
||||
],
|
||||
vec![
|
||||
"1975 ... ".to_string(),
|
||||
"2. 6".to_string(),
|
||||
"5. 1".to_string(),
|
||||
"6. 1".to_string(),
|
||||
"4. 1".to_string(),
|
||||
" ... ".to_string(),
|
||||
],
|
||||
vec![
|
||||
"1976 ... ".to_string(),
|
||||
"4. 3".to_string(),
|
||||
"5. 4".to_string(),
|
||||
"6. 4".to_string(),
|
||||
"4. 5".to_string(),
|
||||
" ... ".to_string(),
|
||||
],
|
||||
];
|
||||
assert!(
|
||||
!is_dot_leader_toc(&cells),
|
||||
"year-indexed data tables with decimal cells must not match TOC detection"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dot_leader_toc_rejects_monthly_data_table() {
|
||||
// ERP-2025 Table B-22: monthly labor-force rows with "Jan ... ",
|
||||
// "Feb ... " labels and thousands-separated cells ("189,164").
|
||||
// Previously matched TOC detection because "Jan ..." has alphabetic
|
||||
// text and "189,164" passed the page-number shape check.
|
||||
let cells = vec![
|
||||
vec![
|
||||
"2023: Jan ... ".to_string(),
|
||||
"265,962".to_string(),
|
||||
"165,871".to_string(),
|
||||
"160,152".to_string(),
|
||||
"62. 4".to_string(),
|
||||
],
|
||||
vec![
|
||||
"Feb ... ".to_string(),
|
||||
"266,112".to_string(),
|
||||
"166,263".to_string(),
|
||||
"160,301".to_string(),
|
||||
"62. 5".to_string(),
|
||||
],
|
||||
vec![
|
||||
"Mar ... ".to_string(),
|
||||
"266,272".to_string(),
|
||||
"166,690".to_string(),
|
||||
"160,824".to_string(),
|
||||
"62. 6".to_string(),
|
||||
],
|
||||
vec![
|
||||
"Apr ... ".to_string(),
|
||||
"266,443".to_string(),
|
||||
"166,678".to_string(),
|
||||
"160,962".to_string(),
|
||||
"62. 6".to_string(),
|
||||
],
|
||||
];
|
||||
assert!(
|
||||
!is_dot_leader_toc(&cells),
|
||||
"monthly labor-force rows with thousands-separated data must not match TOC detection"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tabular_toc_requires_section_numbers_and_pages() {
|
||||
// Dot-less tabular TOC matches is_tabular_toc but not dot-leader.
|
||||
let cells = vec![
|
||||
vec![
|
||||
"4.3 Case studies".to_string(),
|
||||
String::new(),
|
||||
"86".to_string(),
|
||||
],
|
||||
vec![
|
||||
"4.3.1 Destructive actions".to_string(),
|
||||
String::new(),
|
||||
"86".to_string(),
|
||||
],
|
||||
vec![
|
||||
"4.3.2 Adherence".to_string(),
|
||||
String::new(),
|
||||
"89".to_string(),
|
||||
],
|
||||
vec!["4.3.3 Honesty".to_string(), String::new(), "93".to_string()],
|
||||
];
|
||||
assert!(is_tabular_toc(&cells));
|
||||
assert!(!is_dot_leader_toc(&cells));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn starts_with_section_number_matches_dotted() {
|
||||
assert!(starts_with_section_number("1.2"));
|
||||
assert!(starts_with_section_number("4.3.1"));
|
||||
assert!(starts_with_section_number("4.3.1.2"));
|
||||
assert!(starts_with_section_number("4.3 Case studies"));
|
||||
assert!(starts_with_section_number("2.2.5.1 Expert red teaming"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn starts_with_section_number_rejects_non_sections() {
|
||||
assert!(!starts_with_section_number("Chapter 1"));
|
||||
assert!(!starts_with_section_number("1973"));
|
||||
assert!(!starts_with_section_number("1.5M"));
|
||||
assert!(!starts_with_section_number("10.0%"));
|
||||
assert!(!starts_with_section_number(""));
|
||||
assert!(!starts_with_section_number("Hello world"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -265,12 +265,12 @@ pub fn detect_tables_from_lines(items: &[TextItem], lines: &[PdfLine], page: u32
|
||||
page, num_rows, num_cols, item_indices.len(), page_item_count, non_empty_rows, cols_with_content
|
||||
);
|
||||
|
||||
vec![Table::new(
|
||||
col_edges,
|
||||
row_edges_desc[..num_rows].to_vec(),
|
||||
vec![Table {
|
||||
columns: col_edges,
|
||||
rows: row_edges_desc[..num_rows].to_vec(),
|
||||
cells,
|
||||
item_indices,
|
||||
)]
|
||||
}]
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -1037,7 +1037,12 @@ fn try_build_grid(
|
||||
(columns, cells)
|
||||
};
|
||||
|
||||
GridResult::Ok(Table::new(columns, rows, cells, item_indices))
|
||||
GridResult::Ok(Table {
|
||||
columns,
|
||||
rows,
|
||||
cells,
|
||||
item_indices,
|
||||
})
|
||||
}
|
||||
|
||||
/// Deduplicate nearby edge values within a tolerance, returning sorted unique edges.
|
||||
@@ -1437,7 +1442,12 @@ fn detect_row_stripe_table(
|
||||
content_ratio * 100.0
|
||||
);
|
||||
|
||||
Some(Table::new(column_centers, row_centers, cells, item_indices))
|
||||
Some(Table {
|
||||
columns: column_centers,
|
||||
rows: row_centers,
|
||||
cells,
|
||||
item_indices,
|
||||
})
|
||||
}
|
||||
|
||||
/// Detect a table from cell-background rects that failed grid detection.
|
||||
@@ -1681,7 +1691,12 @@ fn detect_row_stripe_table_from_cell_rects(
|
||||
non_empty_cells as f32 / total_cells * 100.0
|
||||
);
|
||||
|
||||
Some(Table::new(column_centers, row_centers, cells, item_indices))
|
||||
Some(Table {
|
||||
columns: column_centers,
|
||||
rows: row_centers,
|
||||
cells,
|
||||
item_indices,
|
||||
})
|
||||
}
|
||||
|
||||
/// Detect a table by merging all cluster rects into one group.
|
||||
@@ -1860,7 +1875,12 @@ fn detect_merged_cluster_table(
|
||||
content_ratio * 100.0
|
||||
);
|
||||
|
||||
Some(Table::new(column_centers, row_centers, cells, item_indices))
|
||||
Some(Table {
|
||||
columns: column_centers,
|
||||
rows: row_centers,
|
||||
cells,
|
||||
item_indices,
|
||||
})
|
||||
}
|
||||
|
||||
/// Cluster text item X positions into column centers with a given minimum threshold.
|
||||
|
||||
@@ -188,12 +188,12 @@ pub fn detect_tables_from_struct_tree(
|
||||
all_item_indices.sort_unstable();
|
||||
all_item_indices.dedup();
|
||||
|
||||
tables.push(Table::new(
|
||||
col_positions,
|
||||
row_positions,
|
||||
tables.push(Table {
|
||||
columns: col_positions,
|
||||
rows: row_positions,
|
||||
cells,
|
||||
all_item_indices,
|
||||
));
|
||||
item_indices: all_item_indices,
|
||||
});
|
||||
}
|
||||
|
||||
tables
|
||||
|
||||
+1
-151
@@ -1,22 +1,12 @@
|
||||
//! Table-to-markdown formatting and cell cleanup.
|
||||
|
||||
use super::{Table, TableKind};
|
||||
use super::Table;
|
||||
|
||||
pub fn table_to_markdown(table: &Table) -> String {
|
||||
if table.cells.is_empty() || table.cells[0].is_empty() {
|
||||
return String::new();
|
||||
}
|
||||
|
||||
// TOCs render poorly as markdown tables — emit a flat per-row text list
|
||||
// instead so the page numbers stay aligned with their section titles
|
||||
// rather than drifting to a separate column. Format from raw cells
|
||||
// because continuation-row merging in clean_table_cells collapses
|
||||
// separate TOC entries (e.g. "6.2 Contamination" + "6.2.1 SWE-bench")
|
||||
// into one line where sub-entries leave column 0 empty.
|
||||
if table.kind == TableKind::Toc {
|
||||
return format_toc_as_list(&table.cells, &[]);
|
||||
}
|
||||
|
||||
// Clean up the table: merge continuation rows, extract footnotes, remove empty rows
|
||||
let (cleaned_cells, footnotes) = clean_table_cells(&table.cells);
|
||||
|
||||
@@ -59,101 +49,6 @@ pub fn table_to_markdown(table: &Table) -> String {
|
||||
output
|
||||
}
|
||||
|
||||
/// Render a table-of-contents as a flat per-row text block.
|
||||
///
|
||||
/// Each row becomes one line: non-empty cells joined with spaces, and the
|
||||
/// last cell (typically a page number) is separated by a tab so the page
|
||||
/// numbers stay aligned with their titles instead of being pulled into a
|
||||
/// separate column by the column-aware reader.
|
||||
fn format_toc_as_list(cells: &[Vec<String>], footnotes: &[String]) -> String {
|
||||
let mut output = String::new();
|
||||
|
||||
for row in cells {
|
||||
let trimmed: Vec<&str> = row.iter().map(|c| c.trim()).collect();
|
||||
let last_idx = trimmed.iter().rposition(|c| !c.is_empty());
|
||||
let Some(last_idx) = last_idx else {
|
||||
continue;
|
||||
};
|
||||
|
||||
let last_cell = trimmed[last_idx];
|
||||
let last_is_page = is_page_number_cell(last_cell);
|
||||
|
||||
let (title_cells, trailing) = if last_is_page && last_idx > 0 {
|
||||
(&trimmed[..last_idx], Some(last_cell))
|
||||
} else {
|
||||
(&trimmed[..=last_idx], None)
|
||||
};
|
||||
|
||||
// Skip dots-only cells when joining the title — in a detected TOC
|
||||
// layout, a "...." cell is a leader separator, not part of the
|
||||
// entry name.
|
||||
let title = title_cells
|
||||
.iter()
|
||||
.filter(|c| !c.is_empty() && !is_dots_only(c))
|
||||
.copied()
|
||||
.collect::<Vec<_>>()
|
||||
.join(" ");
|
||||
|
||||
if title.is_empty() && trailing.is_none() {
|
||||
continue;
|
||||
}
|
||||
|
||||
if !title.is_empty() {
|
||||
output.push_str(&title);
|
||||
}
|
||||
if let Some(page) = trailing {
|
||||
if !title.is_empty() {
|
||||
output.push('\t');
|
||||
}
|
||||
output.push_str(page);
|
||||
}
|
||||
output.push('\n');
|
||||
}
|
||||
|
||||
if !footnotes.is_empty() {
|
||||
output.push('\n');
|
||||
for footnote in footnotes {
|
||||
output.push_str(footnote);
|
||||
output.push('\n');
|
||||
}
|
||||
}
|
||||
|
||||
output
|
||||
}
|
||||
|
||||
/// True when the cell looks like a page number. Accepts:
|
||||
/// - plain digit tokens: "42", "86 86"
|
||||
/// - dashed section-page IDs: "5-21", "A-1", "B--3", "TC-2" (common in
|
||||
/// technical manuals)
|
||||
fn is_page_number_cell(cell: &str) -> bool {
|
||||
let tokens: Vec<&str> = cell.split_whitespace().collect();
|
||||
if tokens.is_empty() {
|
||||
return false;
|
||||
}
|
||||
tokens.iter().all(|t| {
|
||||
if t.is_empty() || t.len() > 8 {
|
||||
return false;
|
||||
}
|
||||
let all_digits = t.chars().all(|c| c.is_ascii_digit());
|
||||
if all_digits {
|
||||
return t.len() <= 4;
|
||||
}
|
||||
// Section-page form: uppercase letters, digits, dashes; at least
|
||||
// one digit present.
|
||||
t.chars()
|
||||
.all(|c| c.is_ascii_digit() || c.is_ascii_uppercase() || c == '-')
|
||||
&& t.chars().any(|c| c.is_ascii_digit())
|
||||
})
|
||||
}
|
||||
|
||||
/// True when the cell is purely leader dots (any length ≥ 3) with optional
|
||||
/// whitespace.
|
||||
fn is_dots_only(cell: &str) -> bool {
|
||||
let t = cell.trim();
|
||||
let dots = t.chars().filter(|&c| c == '.').count();
|
||||
dots >= 3 && t.chars().all(|c| c == '.' || c.is_whitespace())
|
||||
}
|
||||
|
||||
/// Clean up table cells: merge continuation rows, extract footnotes, remove empty rows
|
||||
fn clean_table_cells(cells: &[Vec<String>]) -> (Vec<Vec<String>>, Vec<String>) {
|
||||
let mut cleaned: Vec<Vec<String>> = Vec::new();
|
||||
@@ -463,7 +358,6 @@ mod tests {
|
||||
vec!["Bob".into(), "25".into()],
|
||||
],
|
||||
item_indices: vec![],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
let md = table_to_markdown(&table);
|
||||
assert!(md.contains("|Name|"));
|
||||
@@ -479,7 +373,6 @@ mod tests {
|
||||
rows: vec![500.0],
|
||||
cells: vec![vec!["Only".into(), "Row".into()]],
|
||||
item_indices: vec![],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
let md = table_to_markdown(&table);
|
||||
assert!(md.contains("|Only|"));
|
||||
@@ -493,7 +386,6 @@ mod tests {
|
||||
rows: vec![],
|
||||
cells: vec![],
|
||||
item_indices: vec![],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
assert_eq!(table_to_markdown(&table), "");
|
||||
}
|
||||
@@ -509,7 +401,6 @@ mod tests {
|
||||
vec!["(1)".into(), "Footnote text".into()],
|
||||
],
|
||||
item_indices: vec![],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
let md = table_to_markdown(&table);
|
||||
assert!(md.contains("(1) Footnote text"));
|
||||
@@ -525,7 +416,6 @@ mod tests {
|
||||
vec!["太郎".into(), "25".into()],
|
||||
],
|
||||
item_indices: vec![],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
let md = table_to_markdown(&table);
|
||||
assert!(md.contains("名前"));
|
||||
@@ -539,47 +429,7 @@ mod tests {
|
||||
rows: vec![500.0],
|
||||
cells: vec![vec![]],
|
||||
item_indices: vec![],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
assert_eq!(table_to_markdown(&table), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_table_to_markdown_toc_renders_as_flat_list() {
|
||||
// A TOC-shaped table with section numbers in col 0 and page numbers
|
||||
// in the last column should render as a flat list, not a markdown
|
||||
// table, so the page numbers stay on the same line as their titles.
|
||||
let table = Table::new(
|
||||
vec![50.0, 80.0, 300.0],
|
||||
vec![500.0; 5],
|
||||
vec![
|
||||
vec![
|
||||
"4.3".into(),
|
||||
"Case studies and targeted evaluations".into(),
|
||||
"86".into(),
|
||||
],
|
||||
vec![
|
||||
"4.3.1".into(),
|
||||
"Destructive or reckless actions".into(),
|
||||
"86".into(),
|
||||
],
|
||||
vec![
|
||||
"4.3.2".into(),
|
||||
"Adherence to its constitution".into(),
|
||||
"89".into(),
|
||||
],
|
||||
vec!["4.4".into(), "Capability evaluations".into(), "101".into()],
|
||||
vec!["4.5".into(), "White-box analyses".into(), "113".into()],
|
||||
],
|
||||
vec![],
|
||||
);
|
||||
assert_eq!(table.kind, TableKind::Toc);
|
||||
let md = table_to_markdown(&table);
|
||||
assert!(
|
||||
!md.contains("|---|"),
|
||||
"TOC should not render as a markdown table: {md}"
|
||||
);
|
||||
assert!(md.contains("4.3 Case studies and targeted evaluations\t86"));
|
||||
assert!(md.contains("4.5 White-box analyses\t113"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -499,7 +499,6 @@ pub(crate) fn recover_header_row(
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::tables::TableKind;
|
||||
use crate::types::ItemType;
|
||||
|
||||
fn make_item(text: &str, x: f32, y: f32, font_size: f32) -> TextItem {
|
||||
@@ -757,7 +756,6 @@ mod tests {
|
||||
rows: vec![500.0, 480.0],
|
||||
cells: vec![vec!["A".into(), "B".into()], vec!["C".into(), "D".into()]],
|
||||
item_indices: vec![2, 3],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
|
||||
recover_header_row(&mut table, &all_items, 9.0);
|
||||
@@ -776,7 +774,6 @@ mod tests {
|
||||
rows: vec![500.0],
|
||||
cells: vec![vec!["A".into(), "B".into()]],
|
||||
item_indices: vec![0, 1],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
|
||||
let rows_before = table.rows.len();
|
||||
@@ -797,7 +794,6 @@ mod tests {
|
||||
rows: vec![500.0, 480.0],
|
||||
cells: vec![vec!["A".into(), "B".into()], vec!["C".into(), "D".into()]],
|
||||
item_indices: vec![2, 3],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
|
||||
let rows_before = table.rows.len();
|
||||
@@ -818,7 +814,6 @@ mod tests {
|
||||
rows: vec![500.0],
|
||||
cells: vec![vec!["A".into(), "B".into()]],
|
||||
item_indices: vec![1, 2],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
|
||||
let rows_before = table.rows.len();
|
||||
@@ -834,7 +829,6 @@ mod tests {
|
||||
rows: vec![],
|
||||
cells: vec![],
|
||||
item_indices: vec![],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
|
||||
recover_header_row(&mut table, &all_items, 9.0);
|
||||
|
||||
+11
-49
@@ -11,7 +11,6 @@ mod format;
|
||||
mod grid;
|
||||
|
||||
pub use detect_heuristic::detect_tables;
|
||||
pub(crate) use detect_heuristic::is_table_of_contents;
|
||||
pub use detect_lines::detect_tables_from_lines;
|
||||
pub(crate) use detect_rects::cluster_rects;
|
||||
pub use detect_rects::{detect_tables_from_rects, RectHintRegion};
|
||||
@@ -167,12 +166,12 @@ pub(crate) fn try_build_rect_guided_table(
|
||||
used_indices.sort_unstable();
|
||||
used_indices.dedup();
|
||||
|
||||
Some(Table::new(
|
||||
col_boundaries,
|
||||
row_boundaries,
|
||||
Some(Table {
|
||||
columns: col_boundaries,
|
||||
rows: row_boundaries,
|
||||
cells,
|
||||
used_indices,
|
||||
))
|
||||
item_indices: used_indices,
|
||||
})
|
||||
}
|
||||
|
||||
/// Split a TextItem whose text contains multiple whitespace-separated tokens
|
||||
@@ -542,22 +541,12 @@ pub(crate) fn try_build_table_from_columns(items: &[TextItem], page: u32) -> Opt
|
||||
multi_col_rows
|
||||
);
|
||||
|
||||
Some(Table::new(col_xs, row_ys, cells, item_indices))
|
||||
}
|
||||
|
||||
/// What kind of structure a detected `Table` represents. Classification is
|
||||
/// computed once at construction so consumers don't have to re-analyze the
|
||||
/// cells (and stay consistent across detection backends).
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub enum TableKind {
|
||||
/// A real data table — renders as markdown table syntax.
|
||||
#[default]
|
||||
Data,
|
||||
/// A table of contents — renders as a flat list with tab-aligned page
|
||||
/// numbers via `format_toc_as_list`. Detected through the table pipeline
|
||||
/// because TOCs share row/column structure with tables, but they are not
|
||||
/// data tables and shouldn't appear in `pages_with_tables` etc.
|
||||
Toc,
|
||||
Some(Table {
|
||||
columns: col_xs,
|
||||
rows: row_ys,
|
||||
cells,
|
||||
item_indices,
|
||||
})
|
||||
}
|
||||
|
||||
/// A detected table.
|
||||
@@ -571,31 +560,6 @@ pub struct Table {
|
||||
pub cells: Vec<Vec<String>>,
|
||||
/// Items that belong to this table
|
||||
pub item_indices: Vec<usize>,
|
||||
/// Data table vs TOC. Set by `Table::new` from `cells`.
|
||||
pub kind: TableKind,
|
||||
}
|
||||
|
||||
impl Table {
|
||||
/// Build a table and classify it (data vs TOC) from its cells.
|
||||
pub fn new(
|
||||
columns: Vec<f32>,
|
||||
rows: Vec<f32>,
|
||||
cells: Vec<Vec<String>>,
|
||||
item_indices: Vec<usize>,
|
||||
) -> Self {
|
||||
let kind = if is_table_of_contents(&cells) {
|
||||
TableKind::Toc
|
||||
} else {
|
||||
TableKind::Data
|
||||
};
|
||||
Self {
|
||||
columns,
|
||||
rows,
|
||||
cells,
|
||||
item_indices,
|
||||
kind,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -678,7 +642,6 @@ mod tests {
|
||||
vec!["Cell 1".into(), "Cell 2".into()],
|
||||
],
|
||||
item_indices: vec![],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
|
||||
let md = table_to_markdown(&table);
|
||||
@@ -1039,7 +1002,6 @@ mod tests {
|
||||
vec!["3".into(), "5/2".into(), "Item C".into(), "300".into()],
|
||||
],
|
||||
item_indices: vec![],
|
||||
kind: TableKind::Data,
|
||||
};
|
||||
|
||||
let md = table_to_markdown(&table);
|
||||
|
||||
@@ -520,18 +520,6 @@ impl ToUnicodeCMap {
|
||||
}
|
||||
}
|
||||
|
||||
/// Get the maximum source CID across all mappings (char_map + ranges).
|
||||
fn max_source_cid(&self) -> Option<u16> {
|
||||
let char_max = self.char_map.keys().copied().max();
|
||||
let range_max = self.ranges.iter().map(|&(_, end, _)| end).max();
|
||||
match (char_max, range_max) {
|
||||
(Some(a), Some(b)) => Some(a.max(b)),
|
||||
(a @ Some(_), None) => a,
|
||||
(None, b @ Some(_)) => b,
|
||||
(None, None) => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Remap a CMap that references pre-subsetting GIDs to sequential post-subsetting GIDs.
|
||||
/// Collects all source CIDs, sorts them, and reassigns to 1, 2, 3, ...
|
||||
pub fn remap_to_sequential(&self) -> ToUnicodeCMap {
|
||||
@@ -669,81 +657,6 @@ fn get_w_array_start_cid(cid_font_dict: &lopdf::Dictionary, doc: &Document) -> O
|
||||
}
|
||||
}
|
||||
|
||||
/// Return true if the CIDFont's W (widths) array explicitly covers the given CID.
|
||||
///
|
||||
/// The W array uses two formats (PDF 32000-1:2008, §9.7.4.3):
|
||||
/// 1. `c [w1 w2 ... wn]` — widths for CIDs c, c+1, ..., c+n-1
|
||||
/// 2. `c_first c_last w` — CIDs c_first..c_last all have width w
|
||||
fn w_array_covers_cid(cid_font_dict: &lopdf::Dictionary, doc: &Document, target: u16) -> bool {
|
||||
let Ok(w_obj) = cid_font_dict.get(b"W") else {
|
||||
return false;
|
||||
};
|
||||
let arr = match w_obj {
|
||||
Object::Array(arr) => arr,
|
||||
Object::Reference(r) => match doc.get_object(*r) {
|
||||
Ok(Object::Array(arr)) => arr,
|
||||
_ => return false,
|
||||
},
|
||||
_ => return false,
|
||||
};
|
||||
|
||||
let resolve_int = |o: &Object| -> Option<i64> {
|
||||
match o {
|
||||
Object::Integer(n) => Some(*n),
|
||||
Object::Reference(r) => match doc.get_object(*r) {
|
||||
Ok(Object::Integer(n)) => Some(*n),
|
||||
_ => None,
|
||||
},
|
||||
_ => None,
|
||||
}
|
||||
};
|
||||
|
||||
let resolve_arr = |o: &Object| -> Option<Vec<Object>> {
|
||||
match o {
|
||||
Object::Array(a) => Some(a.clone()),
|
||||
Object::Reference(r) => match doc.get_object(*r) {
|
||||
Ok(Object::Array(a)) => Some(a.clone()),
|
||||
_ => None,
|
||||
},
|
||||
_ => None,
|
||||
}
|
||||
};
|
||||
|
||||
let target = target as i64;
|
||||
let mut i = 0usize;
|
||||
while i < arr.len() {
|
||||
let Some(first) = resolve_int(&arr[i]) else {
|
||||
break;
|
||||
};
|
||||
i += 1;
|
||||
if i >= arr.len() {
|
||||
break;
|
||||
}
|
||||
// Peek at arr[i] to decide format.
|
||||
if let Some(widths) = resolve_arr(&arr[i]) {
|
||||
// Format 1: c [w1 ... wn]
|
||||
let last = first + widths.len() as i64 - 1;
|
||||
if target >= first && target <= last {
|
||||
return true;
|
||||
}
|
||||
i += 1;
|
||||
} else if let Some(last) = resolve_int(&arr[i]) {
|
||||
// Format 2: c_first c_last w
|
||||
i += 1;
|
||||
if i < arr.len() {
|
||||
i += 1; // skip the width value
|
||||
}
|
||||
if target >= first && target <= last {
|
||||
return true;
|
||||
}
|
||||
} else {
|
||||
// Unknown token — abort parsing safely
|
||||
break;
|
||||
}
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
/// Extract CIDToGIDMap as a vector of GIDs (u16) indexed by CID.
|
||||
fn get_cid_to_gid_map(cid_font_dict: &lopdf::Dictionary, doc: &Document) -> Option<Vec<u16>> {
|
||||
let obj = cid_font_dict.get(b"CIDToGIDMap").ok()?;
|
||||
@@ -839,20 +752,6 @@ fn try_remap_subset_cmap(
|
||||
_ => return (cmap, None),
|
||||
};
|
||||
|
||||
// If the W array actually covers the CMap's max source CID, the CMap is
|
||||
// aligned with the font — no sequential renumbering happened. A sparse W
|
||||
// array starting at CID 0 (for .notdef) with additional high-CID entries
|
||||
// matching the CMap is the normal subset layout, not a mismatch.
|
||||
if let Some(max_cid) = cmap.max_source_cid() {
|
||||
if w_array_covers_cid(cid_font_dict, doc, max_cid) {
|
||||
debug!(
|
||||
"Subset remap skipped for obj={}: W array covers CMap max CID {}",
|
||||
obj_num, max_cid
|
||||
);
|
||||
return (cmap, None);
|
||||
}
|
||||
}
|
||||
|
||||
debug!(
|
||||
"Subset GID mismatch detected for obj={}: W starts at CID {}, CMap min CID {}. Remapping to sequential.",
|
||||
obj_num, w_start, min_cid
|
||||
@@ -2818,187 +2717,4 @@ endbfchar
|
||||
assert_eq!(remapped.unwrap().char_map.len(), 50);
|
||||
assert_eq!(fallback.unwrap().char_map.len(), 10);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_max_source_cid() {
|
||||
let cmap_content = r#"
|
||||
1 begincodespacerange
|
||||
<0000><FFFF>
|
||||
endcodespacerange
|
||||
2 beginbfchar
|
||||
<0003> <0020>
|
||||
<0031> <004E>
|
||||
endbfchar
|
||||
1 beginbfrange
|
||||
<0208> <0227> <0430>
|
||||
endbfrange
|
||||
"#;
|
||||
let cmap = ToUnicodeCMap::parse(cmap_content.as_bytes()).unwrap();
|
||||
assert_eq!(cmap.min_source_cid(), Some(0x0003));
|
||||
assert_eq!(cmap.max_source_cid(), Some(0x0227));
|
||||
}
|
||||
|
||||
/// Helper: build a minimal CIDFont dict with a W array and check coverage.
|
||||
fn cid_font_dict_with_w(w_items: Vec<lopdf::Object>) -> lopdf::Dictionary {
|
||||
let mut d = lopdf::Dictionary::new();
|
||||
d.set("W", lopdf::Object::Array(w_items));
|
||||
d
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_w_array_covers_cid_format1() {
|
||||
// Format 1: `c [w1 w2 ... wn]` — widths for CIDs c..c+n-1.
|
||||
// Mimics the 16.pdf Tahoma W array: 0[1000] 3[313] 5[401] 11[383 383] 16[363 303 382]
|
||||
let doc = Document::new();
|
||||
let d = cid_font_dict_with_w(vec![
|
||||
lopdf::Object::Integer(0),
|
||||
lopdf::Object::Array(vec![lopdf::Object::Integer(1000)]),
|
||||
lopdf::Object::Integer(3),
|
||||
lopdf::Object::Array(vec![lopdf::Object::Integer(313)]),
|
||||
lopdf::Object::Integer(5),
|
||||
lopdf::Object::Array(vec![lopdf::Object::Integer(401)]),
|
||||
lopdf::Object::Integer(11),
|
||||
lopdf::Object::Array(vec![
|
||||
lopdf::Object::Integer(383),
|
||||
lopdf::Object::Integer(383),
|
||||
]),
|
||||
lopdf::Object::Integer(16),
|
||||
lopdf::Object::Array(vec![
|
||||
lopdf::Object::Integer(363),
|
||||
lopdf::Object::Integer(303),
|
||||
lopdf::Object::Integer(382),
|
||||
]),
|
||||
lopdf::Object::Integer(570),
|
||||
lopdf::Object::Array(vec![lopdf::Object::Integer(667); 26]),
|
||||
]);
|
||||
|
||||
assert!(w_array_covers_cid(&d, &doc, 0));
|
||||
assert!(w_array_covers_cid(&d, &doc, 3));
|
||||
assert!(w_array_covers_cid(&d, &doc, 5));
|
||||
assert!(w_array_covers_cid(&d, &doc, 11));
|
||||
assert!(w_array_covers_cid(&d, &doc, 12));
|
||||
assert!(w_array_covers_cid(&d, &doc, 16));
|
||||
assert!(w_array_covers_cid(&d, &doc, 18));
|
||||
assert!(w_array_covers_cid(&d, &doc, 570));
|
||||
assert!(w_array_covers_cid(&d, &doc, 595));
|
||||
// Gaps are NOT covered
|
||||
assert!(!w_array_covers_cid(&d, &doc, 1));
|
||||
assert!(!w_array_covers_cid(&d, &doc, 4));
|
||||
assert!(!w_array_covers_cid(&d, &doc, 19));
|
||||
assert!(!w_array_covers_cid(&d, &doc, 596));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_w_array_covers_cid_format2() {
|
||||
// Format 2: `c_first c_last w` — CIDs c_first..c_last all have width w.
|
||||
let doc = Document::new();
|
||||
let d = cid_font_dict_with_w(vec![
|
||||
lopdf::Object::Integer(100),
|
||||
lopdf::Object::Integer(120),
|
||||
lopdf::Object::Integer(500),
|
||||
]);
|
||||
|
||||
assert!(w_array_covers_cid(&d, &doc, 100));
|
||||
assert!(w_array_covers_cid(&d, &doc, 110));
|
||||
assert!(w_array_covers_cid(&d, &doc, 120));
|
||||
assert!(!w_array_covers_cid(&d, &doc, 99));
|
||||
assert!(!w_array_covers_cid(&d, &doc, 121));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_w_array_covers_cid_missing_w() {
|
||||
let doc = Document::new();
|
||||
let d = lopdf::Dictionary::new();
|
||||
assert!(!w_array_covers_cid(&d, &doc, 3));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_try_remap_skipped_when_w_covers_cmap() {
|
||||
// Simulates 16.pdf: CMap's max source CID (0x0279 = 633) is explicitly
|
||||
// in the W array, so no subset-renumbering happened — remap must NOT fire.
|
||||
let cmap_content = r#"
|
||||
1 begincodespacerange
|
||||
<0000><FFFF>
|
||||
endcodespacerange
|
||||
2 beginbfchar
|
||||
<0003> <0020>
|
||||
<0031> <004E>
|
||||
endbfchar
|
||||
2 beginbfrange
|
||||
<023A> <0253> <0410>
|
||||
<0255> <0279> <042B>
|
||||
endbfrange
|
||||
"#;
|
||||
let cmap = ToUnicodeCMap::parse(cmap_content.as_bytes()).unwrap();
|
||||
|
||||
let mut doc = Document::new();
|
||||
// Build a CIDFont dict with Identity CIDToGIDMap and a W array that
|
||||
// covers CID 633 via `597 [widths...]`.
|
||||
let mut cid_font = lopdf::Dictionary::new();
|
||||
cid_font.set("CIDToGIDMap", lopdf::Object::Name(b"Identity".to_vec()));
|
||||
cid_font.set(
|
||||
"W",
|
||||
lopdf::Object::Array(vec![
|
||||
lopdf::Object::Integer(0),
|
||||
lopdf::Object::Array(vec![lopdf::Object::Integer(750)]),
|
||||
lopdf::Object::Integer(597),
|
||||
lopdf::Object::Array(vec![lopdf::Object::Integer(500); 37]), // 597..633
|
||||
]),
|
||||
);
|
||||
let cid_font_id = doc.add_object(cid_font);
|
||||
|
||||
// Build the Type0 font dict with Identity-H + DescendantFonts ref.
|
||||
let mut font_dict = lopdf::Dictionary::new();
|
||||
font_dict.set("Encoding", lopdf::Object::Name(b"Identity-H".to_vec()));
|
||||
font_dict.set(
|
||||
"DescendantFonts",
|
||||
lopdf::Object::Array(vec![lopdf::Object::Reference(cid_font_id)]),
|
||||
);
|
||||
|
||||
let (primary, remapped) = try_remap_subset_cmap(cmap, &font_dict, &doc, 123);
|
||||
assert!(
|
||||
remapped.is_none(),
|
||||
"Remap must be skipped when W covers CMap max CID (this is 16.pdf)"
|
||||
);
|
||||
assert_eq!(primary.lookup(0x0003), Some(" ".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_try_remap_fires_for_true_subset_mismatch() {
|
||||
// True mismatch: CMap has high CIDs (512-544) but W only lists low sequential CIDs.
|
||||
let cmap_content = r#"
|
||||
1 begincodespacerange
|
||||
<0000><FFFF>
|
||||
endcodespacerange
|
||||
1 beginbfrange
|
||||
<0200> <0220> <0410>
|
||||
endbfrange
|
||||
"#;
|
||||
let cmap = ToUnicodeCMap::parse(cmap_content.as_bytes()).unwrap();
|
||||
|
||||
let mut doc = Document::new();
|
||||
let mut cid_font = lopdf::Dictionary::new();
|
||||
cid_font.set("CIDToGIDMap", lopdf::Object::Name(b"Identity".to_vec()));
|
||||
cid_font.set(
|
||||
"W",
|
||||
lopdf::Object::Array(vec![
|
||||
lopdf::Object::Integer(0),
|
||||
lopdf::Object::Array(vec![lopdf::Object::Integer(500); 34]), // 0..33
|
||||
]),
|
||||
);
|
||||
let cid_font_id = doc.add_object(cid_font);
|
||||
|
||||
let mut font_dict = lopdf::Dictionary::new();
|
||||
font_dict.set("Encoding", lopdf::Object::Name(b"Identity-H".to_vec()));
|
||||
font_dict.set(
|
||||
"DescendantFonts",
|
||||
lopdf::Object::Array(vec![lopdf::Object::Reference(cid_font_id)]),
|
||||
);
|
||||
|
||||
let (_primary, remapped) = try_remap_subset_cmap(cmap, &font_dict, &doc, 456);
|
||||
assert!(
|
||||
remapped.is_some(),
|
||||
"Remap must fire when CMap's CIDs are outside W array coverage"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
+15
-53
@@ -4,10 +4,10 @@ use pdf_inspector::detector::{DetectionConfig, ScanStrategy};
|
||||
use pdf_inspector::extractor::group_into_lines;
|
||||
use pdf_inspector::types::TextLine;
|
||||
use pdf_inspector::{
|
||||
detect_pdf_type, extract_pages_markdown, extract_pages_markdown_mem,
|
||||
extract_tables_in_regions_mem, extract_text, extract_text_in_regions_mem,
|
||||
extract_text_with_positions, process_pdf_mem, process_pdf_with_options, to_markdown,
|
||||
MarkdownOptions, PdfError, PdfOptions, PdfType, TextItem,
|
||||
detect_pdf_type, extract_pages_markdown_mem, extract_tables_in_regions_mem, extract_text,
|
||||
extract_text_in_regions_mem, extract_text_with_positions, process_pdf_mem,
|
||||
process_pdf_with_options, to_markdown, MarkdownOptions, PdfError, PdfOptions, PdfType,
|
||||
TextItem,
|
||||
};
|
||||
use std::collections::HashSet;
|
||||
|
||||
@@ -1481,7 +1481,7 @@ fn test_bits_pilani_page8_table_detection() {
|
||||
#[test]
|
||||
fn test_extract_pages_markdown_basic() {
|
||||
let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, Some(&[0, 1])).unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, &[0, 1]).unwrap();
|
||||
|
||||
assert_eq!(result.pages.len(), 2);
|
||||
assert_eq!(result.pages[0].page, 0);
|
||||
@@ -1495,7 +1495,7 @@ fn test_extract_pages_markdown_basic() {
|
||||
fn test_extract_pages_markdown_page_ordering() {
|
||||
let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
|
||||
// Request pages in non-sequential order
|
||||
let result = extract_pages_markdown_mem(&buf, Some(&[1, 0])).unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, &[1, 0]).unwrap();
|
||||
|
||||
assert_eq!(result.pages.len(), 2);
|
||||
// Results should match input order, not document order
|
||||
@@ -1506,7 +1506,7 @@ fn test_extract_pages_markdown_page_ordering() {
|
||||
#[test]
|
||||
fn test_extract_pages_markdown_out_of_range() {
|
||||
let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, Some(&[9999])).unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, &[9999]).unwrap();
|
||||
|
||||
assert_eq!(result.pages.len(), 1);
|
||||
assert_eq!(result.pages[0].page, 9999);
|
||||
@@ -1518,14 +1518,14 @@ fn test_extract_pages_markdown_out_of_range() {
|
||||
#[test]
|
||||
fn test_extract_pages_markdown_empty_pages_list() {
|
||||
let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, Some(&[])).unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, &[]).unwrap();
|
||||
assert!(result.pages.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_extract_pages_markdown_single_page() {
|
||||
let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, Some(&[0])).unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, &[0]).unwrap();
|
||||
|
||||
assert_eq!(result.pages.len(), 1);
|
||||
assert_eq!(result.pages[0].page, 0);
|
||||
@@ -1535,7 +1535,7 @@ fn test_extract_pages_markdown_single_page() {
|
||||
|
||||
#[test]
|
||||
fn test_extract_pages_markdown_invalid_buffer() {
|
||||
let result = extract_pages_markdown_mem(b"not a pdf", Some(&[0]));
|
||||
let result = extract_pages_markdown_mem(b"not a pdf", &[0]);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
@@ -1543,7 +1543,7 @@ fn test_extract_pages_markdown_invalid_buffer() {
|
||||
fn test_extract_pages_markdown_gid_pages_need_ocr() {
|
||||
// shinagawa_identity_h.pdf has GID-encoded fonts
|
||||
let buf = std::fs::read("tests/fixtures/shinagawa_identity_h.pdf").unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, Some(&[0])).unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, &[0]).unwrap();
|
||||
|
||||
assert_eq!(result.pages.len(), 1);
|
||||
assert!(result.pages[0].needs_ocr);
|
||||
@@ -1556,7 +1556,7 @@ fn test_extract_pages_markdown_classification_with_tables() {
|
||||
let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
|
||||
let page_count = process_pdf_mem(&buf).unwrap().page_count;
|
||||
let page_indices: Vec<u32> = (0..page_count).collect();
|
||||
let result = extract_pages_markdown_mem(&buf, Some(&page_indices)).unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, &page_indices).unwrap();
|
||||
|
||||
assert!(
|
||||
!result.pages_with_tables.is_empty(),
|
||||
@@ -1569,7 +1569,7 @@ fn test_extract_pages_markdown_classification_with_tables() {
|
||||
fn test_extract_pages_markdown_simple_pdf_no_complexity() {
|
||||
// bare_name_struct.pdf is a simple document with a heading and code block
|
||||
let buf = std::fs::read("tests/fixtures/bare_name_struct.pdf").unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, Some(&[0])).unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, &[0]).unwrap();
|
||||
|
||||
assert!(result.pages_with_tables.is_empty());
|
||||
assert!(result.pages_with_columns.is_empty());
|
||||
@@ -1582,7 +1582,7 @@ fn test_extract_pages_markdown_classification_matches_process_pdf() {
|
||||
let full = process_pdf_mem(&buf).unwrap();
|
||||
let page_count = full.page_count;
|
||||
let page_indices: Vec<u32> = (0..page_count).collect();
|
||||
let result = extract_pages_markdown_mem(&buf, Some(&page_indices)).unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, &page_indices).unwrap();
|
||||
|
||||
assert_eq!(
|
||||
result.pages_with_tables, full.layout.pages_with_tables,
|
||||
@@ -1605,7 +1605,7 @@ fn test_extract_pages_markdown_consistency_with_process_pdf() {
|
||||
// Get per-page output for all pages
|
||||
let page_count = full.page_count;
|
||||
let page_indices: Vec<u32> = (0..page_count).collect();
|
||||
let result = extract_pages_markdown_mem(&buf, Some(&page_indices)).unwrap();
|
||||
let result = extract_pages_markdown_mem(&buf, &page_indices).unwrap();
|
||||
|
||||
// Concatenated per-page markdown should contain substantial overlap with
|
||||
// the full output (exact match not expected due to header/footer stripping
|
||||
@@ -1630,41 +1630,3 @@ fn test_extract_pages_markdown_consistency_with_process_pdf() {
|
||||
full_md.len()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_extract_pages_markdown_none_returns_all_pages() {
|
||||
let buf = std::fs::read("tests/fixtures/nexo-price-en.pdf").unwrap();
|
||||
let page_count = process_pdf_mem(&buf).unwrap().page_count;
|
||||
|
||||
let result = extract_pages_markdown_mem(&buf, None).unwrap();
|
||||
|
||||
assert_eq!(result.pages.len() as u32, page_count);
|
||||
for (i, page) in result.pages.iter().enumerate() {
|
||||
assert_eq!(page.page, i as u32, "pages should be in document order");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_extract_pages_markdown_path_api() {
|
||||
let path = "tests/fixtures/nexo-price-en.pdf";
|
||||
let buf = std::fs::read(path).unwrap();
|
||||
|
||||
let via_path = extract_pages_markdown(path, Some(&[0])).unwrap();
|
||||
let via_mem = extract_pages_markdown_mem(&buf, Some(&[0])).unwrap();
|
||||
|
||||
assert_eq!(via_path.pages.len(), via_mem.pages.len());
|
||||
assert_eq!(via_path.pages[0].markdown, via_mem.pages[0].markdown);
|
||||
assert_eq!(via_path.pages[0].needs_ocr, via_mem.pages[0].needs_ocr);
|
||||
assert_eq!(via_path.is_complex, via_mem.is_complex);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_extract_pages_markdown_path_none_returns_all_pages() {
|
||||
let path = "tests/fixtures/nexo-price-en.pdf";
|
||||
let page_count = process_pdf_mem(&std::fs::read(path).unwrap())
|
||||
.unwrap()
|
||||
.page_count;
|
||||
|
||||
let result = extract_pages_markdown(path, None).unwrap();
|
||||
assert_eq!(result.pages.len() as u32, page_count);
|
||||
}
|
||||
|
||||
@@ -270,78 +270,6 @@ class TestExtractTextInRegions:
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# extract_pages_markdown / extract_pages_markdown_bytes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExtractPagesMarkdown:
|
||||
def test_default_returns_all_pages(self):
|
||||
result = pdf_inspector.extract_pages_markdown(
|
||||
fixture_path("thermo-freon12.pdf")
|
||||
)
|
||||
assert len(result.pages) == 3
|
||||
assert [p.page for p in result.pages] == [0, 1, 2]
|
||||
assert all(isinstance(p.markdown, str) for p in result.pages)
|
||||
|
||||
def test_bytes_default_returns_all_pages(self):
|
||||
data = fixture_bytes("thermo-freon12.pdf")
|
||||
result = pdf_inspector.extract_pages_markdown_bytes(data)
|
||||
assert len(result.pages) == 3
|
||||
|
||||
def test_selected_pages_preserve_order(self):
|
||||
result = pdf_inspector.extract_pages_markdown(
|
||||
fixture_path("thermo-freon12.pdf"), pages=[2, 0]
|
||||
)
|
||||
assert [p.page for p in result.pages] == [2, 0]
|
||||
|
||||
def test_bytes_selected_pages_preserve_order(self):
|
||||
data = fixture_bytes("thermo-freon12.pdf")
|
||||
result = pdf_inspector.extract_pages_markdown_bytes(data, pages=[1])
|
||||
assert len(result.pages) == 1
|
||||
assert result.pages[0].page == 1
|
||||
|
||||
def test_page_fields(self):
|
||||
result = pdf_inspector.extract_pages_markdown(
|
||||
fixture_path("thermo-freon12.pdf"), pages=[0]
|
||||
)
|
||||
page = result.pages[0]
|
||||
assert isinstance(page.page, int)
|
||||
assert isinstance(page.markdown, str)
|
||||
assert isinstance(page.needs_ocr, bool)
|
||||
assert not page.needs_ocr # text-based fixture
|
||||
assert len(page.markdown) > 0
|
||||
|
||||
def test_result_fields(self):
|
||||
result = pdf_inspector.extract_pages_markdown(
|
||||
fixture_path("thermo-freon12.pdf")
|
||||
)
|
||||
assert isinstance(result.pages, list)
|
||||
assert isinstance(result.pages_with_tables, list)
|
||||
assert isinstance(result.pages_with_columns, list)
|
||||
assert isinstance(result.pages_needing_ocr, list)
|
||||
assert isinstance(result.is_complex, bool)
|
||||
|
||||
def test_out_of_range_page_marks_needs_ocr(self):
|
||||
result = pdf_inspector.extract_pages_markdown(
|
||||
fixture_path("thermo-freon12.pdf"), pages=[9999]
|
||||
)
|
||||
assert len(result.pages) == 1
|
||||
assert result.pages[0].needs_ocr
|
||||
assert result.pages[0].markdown == ""
|
||||
|
||||
def test_repr(self):
|
||||
result = pdf_inspector.extract_pages_markdown(
|
||||
fixture_path("thermo-freon12.pdf"), pages=[0]
|
||||
)
|
||||
assert "PagesExtractionResult" in repr(result)
|
||||
assert "PageMarkdown" in repr(result.pages[0])
|
||||
|
||||
def test_not_a_pdf(self):
|
||||
with pytest.raises(ValueError):
|
||||
pdf_inspector.extract_pages_markdown_bytes(b"not a pdf")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Error handling
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user