remove agents
This commit is contained in:
Abimael Martell
@@ -1,146 +0,0 @@
|
||||
# AGENTS.md — pdf-inspector Codebase Guide
|
||||
|
||||
## Project Overview
|
||||
|
||||
Rust crate (`pdf-inspector`) that extracts text from PDFs and converts it to structured Markdown. Ships a CLI binary `pdf2md`.
|
||||
|
||||
- **Crate name:** `pdf-inspector`
|
||||
- **Binary:** `pdf2md` (`src/bin/pdf2md.rs`)
|
||||
- **PDF parsing:** `lopdf` crate (v0.39.0, git dependency)
|
||||
- **Test PDFs:** `/Users/abimaelmartell/Code/pdf-evals/pdfs/`
|
||||
|
||||
## Module Map
|
||||
|
||||
```
|
||||
src/
|
||||
lib.rs — Public API, re-exports
|
||||
types.rs — Shared types: TextItem, TextLine, PdfRect, ItemType
|
||||
text_utils.rs — Character/text helpers: CJK, RTL, ligatures, bold/italic
|
||||
detector.rs — Fast PDF type detection (text vs scanned) without full load
|
||||
glyph_names.rs — Adobe Glyph List → Unicode mapping
|
||||
tounicode.rs — ToUnicode CMap parsing for CID-encoded text
|
||||
|
||||
extractor/
|
||||
mod.rs — Public API: extract_text, extract_text_with_positions
|
||||
fonts.rs — Font width parsing, encoding, text decoding
|
||||
content_stream.rs — PDF operator state machine (Tm, Td, Tj, TJ, etc.)
|
||||
xobjects.rs — Form XObject and image XObject extraction
|
||||
links.rs — Hyperlink and AcroForm field extraction
|
||||
layout.rs — Column detection, line grouping, reading order
|
||||
|
||||
tables/
|
||||
mod.rs — Table struct, TableDetectionMode, re-exports
|
||||
detect_rects.rs — Rectangle-based table detection (union-find clustering)
|
||||
detect_heuristic.rs — Heuristic table detection + validation
|
||||
financial.rs — Financial token splitting for consolidated values
|
||||
grid.rs — Column/row boundaries, cell assignment
|
||||
format.rs — Table → Markdown formatting, footnotes
|
||||
|
||||
markdown/
|
||||
mod.rs — MarkdownOptions, public API (to_markdown, to_markdown_from_items)
|
||||
convert.rs — Core line-to-markdown loop, table/image interleaving
|
||||
analysis.rs — Font statistics, heading tiers, paragraph thresholds
|
||||
classify.rs — Caption, list, code detection
|
||||
preprocess.rs — Heading merging, drop cap handling
|
||||
postprocess.rs — Cleanup: dot leaders, hyphenation, page numbers, URLs
|
||||
|
||||
bin/
|
||||
pdf2md.rs — CLI: PDF → Markdown
|
||||
detect_pdf.rs — CLI: detect PDF type
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
PDF bytes
|
||||
│
|
||||
├─► detector.rs → PdfType (TextBased / Scanned / ImageBased)
|
||||
│
|
||||
└─► extractor/
|
||||
├─ fonts.rs → font widths, encodings
|
||||
├─ content_stream.rs → walk operators → Vec<TextItem> + Vec<PdfRect>
|
||||
├─ xobjects.rs → Form XObject text, image placeholders
|
||||
├─ links.rs → hyperlinks, AcroForm fields
|
||||
└─ layout.rs → column detection → group_into_lines → Vec<TextLine>
|
||||
│
|
||||
├─► tables/
|
||||
│ ├─ detect_rects.rs → rect-based tables (PdfRect clusters)
|
||||
│ ├─ detect_heuristic.rs → heuristic tables (font-size + alignment)
|
||||
│ ├─ grid.rs → column/row assignment → cells
|
||||
│ └─ format.rs → Table → Markdown string
|
||||
│
|
||||
└─► markdown/
|
||||
├─ analysis.rs → font stats, heading tiers
|
||||
├─ preprocess.rs → merge headings, drop caps
|
||||
├─ convert.rs → line loop + table/image insertion
|
||||
├─ classify.rs → captions, lists, code
|
||||
└─ postprocess.rs → cleanup → final Markdown string
|
||||
```
|
||||
|
||||
## Critical Implementation Details
|
||||
|
||||
### Text Matrix Math
|
||||
PDF text positioning uses two matrices: `text_matrix` (Tm) and `line_matrix`. The `Td`/`TD` operators provide offsets in **text space**, which must be scaled by `line_matrix`:
|
||||
|
||||
```
|
||||
e += tx * a + ty * c
|
||||
f += tx * b + ty * d
|
||||
```
|
||||
|
||||
When `Tm` has scaling (e.g., `[12,0,0,12,x,y]`), failing to apply this scaling produces incorrect positions. The `T*` and `'` operators are equivalent to `0 -TL Td` and need the same treatment.
|
||||
|
||||
### Font Size
|
||||
Font size can come from the `Tf` operand **or** the `Tm` matrix scaling. Use `effective_font_size()` from `text_utils.rs` to get the correct value.
|
||||
|
||||
### White-Fill Text
|
||||
Text drawn with white fill (`1 g` before text ops) should be skipped during extraction but the text matrix must still advance to keep positions correct.
|
||||
|
||||
### CID Fonts
|
||||
Fonts named `C2_*` or `C0_*` are CID fonts that emit one word per `Tj` operator. Spaces must be inserted between consecutive `Tj` items.
|
||||
|
||||
### lopdf Quirks
|
||||
- `lopdf::error::ParseError` is private — match by string for `InvalidFileHeader`
|
||||
- Clippy enforces `-D warnings` — use `is_some_and(...)` instead of `map_or(false, ...)`
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
cargo test # Run all 66 unit tests
|
||||
cargo clippy -- -D warnings # Lint (enforced in CI)
|
||||
cargo fmt --check # Format check
|
||||
cargo run --release --bin pdf2md -- <file.pdf> # Smoke test
|
||||
```
|
||||
|
||||
## Debugging with RUST_LOG
|
||||
|
||||
All debug output uses structured logging via the `log` crate. Set `RUST_LOG` to control output:
|
||||
|
||||
```bash
|
||||
RUST_LOG=pdf_inspector::extractor::content_stream=trace # raw PDF operators
|
||||
RUST_LOG=pdf_inspector::extractor::fonts=debug # font metadata + encodings
|
||||
RUST_LOG=pdf_inspector::tounicode=debug # CMap parsing
|
||||
RUST_LOG=pdf_inspector::extractor=debug # text items per page
|
||||
RUST_LOG=pdf_inspector::extractor::layout=debug # columns, reading order
|
||||
RUST_LOG=pdf_inspector::markdown::analysis=debug # Y-gaps, paragraph threshold
|
||||
RUST_LOG=pdf_inspector::tables=debug # table detection
|
||||
RUST_LOG=pdf_inspector=debug # everything
|
||||
```
|
||||
|
||||
Example: `RUST_LOG=pdf_inspector::extractor::fonts=debug cargo run --bin pdf2md -- file.pdf > /dev/null`
|
||||
|
||||
## Common Tasks
|
||||
|
||||
| Task | Where to Edit |
|
||||
|------|--------------|
|
||||
| Fix text positioning bugs | `extractor/content_stream.rs` |
|
||||
| Add font encoding support | `extractor/fonts.rs`, `tounicode.rs` |
|
||||
| Fix column/reading order | `extractor/layout.rs` |
|
||||
| Improve table detection | `tables/detect_heuristic.rs` |
|
||||
| Fix table formatting | `tables/format.rs`, `tables/grid.rs` |
|
||||
| Add rectangle-based tables | `tables/detect_rects.rs` |
|
||||
| Change heading detection | `markdown/analysis.rs` |
|
||||
| Fix list/code detection | `markdown/classify.rs` |
|
||||
| Fix paragraph breaks | `markdown/convert.rs`, `markdown/analysis.rs` |
|
||||
| Fix URL/hyphenation cleanup | `markdown/postprocess.rs` |
|
||||
| Add new PDF type detection | `detector.rs` |
|
||||
| Add new text item type | `types.rs`, then update consumers |
|
||||
Reference in New Issue
Block a user