89 lines
3.7 KiB
Markdown
89 lines
3.7 KiB
Markdown
# pdf-inspector
|
|
|
|
Fast Rust library for PDF inspection, classification, and text extraction. Intelligently detects scanned vs text-based PDFs to enable smart routing decisions.
|
|
|
|
## Supported Features
|
|
|
|
| Category | Feature | Description |
|
|
|----------|---------|-------------|
|
|
| **Detection** | Fast Classification | ~10-50ms by sampling content streams |
|
|
| | PDF Types | TextBased, Scanned, ImageBased, Mixed |
|
|
| | Confidence Scoring | 0.0-1.0 scale for classification certainty |
|
|
| | Configurable Thresholds | Tune sampling depth and detection sensitivity |
|
|
| | Metadata Extraction | Document title from PDF Info dictionary |
|
|
| **Text Extraction** | Plain Text | Direct extraction from text-based PDFs |
|
|
| | Position-Aware | Text with X/Y coordinates, font info, page numbers |
|
|
| | Multi-Column Support | Automatic detection and proper reading order |
|
|
| | Text Encoding | UTF-16BE, UTF-8, and Latin-1 |
|
|
| | ToUnicode CMap | Proper decoding of CID-keyed fonts (Type0/Identity-H) |
|
|
| | Linearized PDFs | Raw stream extraction for optimized PDFs |
|
|
| **Headers** | Auto Detection | H1-H4 based on font size ratios |
|
|
| **Lists** | Bullet Points | `•`, `-`, `*`, `○`, `●`, `◦` |
|
|
| | Numbered Lists | `1.`, `1)`, `(1)` |
|
|
| | Letter Lists | `a.`, `a)`, `(a)` |
|
|
| **Code Blocks** | Monospace Fonts | Courier, Consolas, Monaco, Menlo, Fira Code, JetBrains Mono |
|
|
| | Keyword Detection | Language keywords and syntax patterns |
|
|
| **Tables** | Region Detection | Automatic table boundary identification |
|
|
| | Column/Row Detection | Position clustering for structure |
|
|
| | Markdown Output | Proper alignment and formatting |
|
|
| | Footnotes | Extraction and formatting |
|
|
| **Text Processing** | Subscript/Superscript | Font size and Y-offset detection |
|
|
| | Hyphenation Fixing | Rejoins words broken across lines |
|
|
| | Page Number Filtering | Removes isolated page numbers |
|
|
| | URL Formatting | Converts URLs to markdown links |
|
|
| | Drop Cap Merging | Handles large initial letters |
|
|
|
|
## Output Formats
|
|
|
|
| Format | Description |
|
|
|--------|-------------|
|
|
| Markdown | Headers, lists, code blocks, tables, page breaks |
|
|
| Plain Text | Basic text extraction |
|
|
| JSON | Metadata with type, confidence, page count, timing |
|
|
| Positioned Items | Low-level text with coordinates and font info |
|
|
|
|
## CLI Tools
|
|
|
|
| Tool | Description |
|
|
|------|-------------|
|
|
| `pdf2md` | Convert PDF to Markdown (supports `--json` output) |
|
|
| `detect-pdf` | Detect PDF type without conversion (supports `--json` output) |
|
|
|
|
## API Overview
|
|
|
|
### Functions
|
|
|
|
| Function | Description |
|
|
|----------|-------------|
|
|
| `process_pdf` / `process_pdf_mem` | Detect, extract, and convert to markdown |
|
|
| `detect_pdf_type` / `detect_pdf_type_mem` | Fast type detection only |
|
|
| `extract_text` / `extract_text_mem` | Plain text extraction |
|
|
| `extract_text_with_positions` | Text with coordinates |
|
|
| `to_markdown` | Convert text to markdown |
|
|
|
|
### Types
|
|
|
|
| Type | Description |
|
|
|------|-------------|
|
|
| `PdfType` | `TextBased`, `Scanned`, `ImageBased`, `Mixed` |
|
|
| `PdfProcessResult` | Full result with text, markdown, and metadata |
|
|
| `PdfTypeResult` | Detection result with type, confidence, page count |
|
|
| `TextItem` | Text with position, font info, and page number |
|
|
| `TextLine` | Grouped items on the same line |
|
|
| `MarkdownOptions` | Configuration for markdown conversion |
|
|
| `DetectionConfig` | Configuration for PDF type detection |
|
|
| `PdfError` | `Io`, `Parse`, `Encrypted`, `InvalidStructure` |
|
|
|
|
## How Detection Works
|
|
|
|
1. Load only metadata (xref table, trailer, page count)
|
|
2. Sample first ~5 pages' content streams
|
|
3. Scan raw bytes for `Tj`/`TJ` (text) and `Do` (image) operators
|
|
4. Classify based on text operator presence
|
|
|
|
This allows detecting 300+ page PDFs in milliseconds.
|
|
|
|
## License
|
|
|
|
MIT
|