Abimael MartellandClaude Opus 4.6
423e969843
The previous approach used a 30% page-width gap threshold on sorted X positions, which never detected real column gutters (typically 2-5% of page width). This caused text from adjacent columns to be interleaved on the same line (e.g. "Brookfieldchief", "areclinging"). The new approach builds an occupancy histogram across the page width, finds empty valleys (gutters), and validates them with vertical consistency checks. Also adds spanning-item detection so full-width headers/titles are handled correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
pdf-inspector
Fast Rust library for PDF inspection, classification, and text extraction. Intelligently detects scanned vs text-based PDFs to enable smart routing decisions.
Supported Features
| Category | Feature | Description |
|---|---|---|
| Detection | Fast Classification | ~10-50ms by sampling content streams |
| PDF Types | TextBased, Scanned, ImageBased, Mixed | |
| Confidence Scoring | 0.0-1.0 scale for classification certainty | |
| Configurable Thresholds | Tune sampling depth and detection sensitivity | |
| Metadata Extraction | Document title from PDF Info dictionary | |
| Text Extraction | Plain Text | Direct extraction from text-based PDFs |
| Position-Aware | Text with X/Y coordinates, font info, page numbers | |
| Multi-Column Support | Automatic detection and proper reading order | |
| Text Encoding | UTF-16BE, UTF-8, and Latin-1 | |
| ToUnicode CMap | Proper decoding of CID-keyed fonts (Type0/Identity-H) | |
| Linearized PDFs | Raw stream extraction for optimized PDFs | |
| Headers | Auto Detection | H1-H4 based on font size ratios |
| Lists | Bullet Points | •, -, *, ○, ●, ◦ |
| Numbered Lists | 1., 1), (1) |
|
| Letter Lists | a., a), (a) |
|
| Code Blocks | Monospace Fonts | Courier, Consolas, Monaco, Menlo, Fira Code, JetBrains Mono |
| Keyword Detection | Language keywords and syntax patterns | |
| Tables | Region Detection | Automatic table boundary identification |
| Column/Row Detection | Position clustering for structure | |
| Markdown Output | Proper alignment and formatting | |
| Footnotes | Extraction and formatting | |
| Text Processing | Subscript/Superscript | Font size and Y-offset detection |
| Hyphenation Fixing | Rejoins words broken across lines | |
| Page Number Filtering | Removes isolated page numbers | |
| URL Formatting | Converts URLs to markdown links | |
| Drop Cap Merging | Handles large initial letters |
Output Formats
| Format | Description |
|---|---|
| Markdown | Headers, lists, code blocks, tables, page breaks |
| Plain Text | Basic text extraction |
| JSON | Metadata with type, confidence, page count, timing |
| Positioned Items | Low-level text with coordinates and font info |
CLI Tools
| Tool | Description |
|---|---|
pdf2md |
Convert PDF to Markdown (supports --json output) |
detect-pdf |
Detect PDF type without conversion (supports --json output) |
API Overview
Functions
| Function | Description |
|---|---|
process_pdf / process_pdf_mem |
Detect, extract, and convert to markdown |
detect_pdf_type / detect_pdf_type_mem |
Fast type detection only |
extract_text / extract_text_mem |
Plain text extraction |
extract_text_with_positions |
Text with coordinates |
to_markdown |
Convert text to markdown |
Types
| Type | Description |
|---|---|
PdfType |
TextBased, Scanned, ImageBased, Mixed |
PdfProcessResult |
Full result with text, markdown, and metadata |
PdfTypeResult |
Detection result with type, confidence, page count |
TextItem |
Text with position, font info, and page number |
TextLine |
Grouped items on the same line |
MarkdownOptions |
Configuration for markdown conversion |
DetectionConfig |
Configuration for PDF type detection |
PdfError |
Io, Parse, Encrypted, InvalidStructure |
How Detection Works
- Load only metadata (xref table, trailer, page count)
- Sample first ~5 pages' content streams
- Scan raw bytes for
Tj/TJ(text) andDo(image) operators - Classify based on text operator presence
This allows detecting 300+ page PDFs in milliseconds.
License
MIT