* fix(tables): flatten page-number tables of contents instead of gridding
A title-based contents page ("About the Publisher vii", "Experiment #1
… 3") with no dot leaders and no section numbers was detected as a
2-column data table and rendered as a markdown grid, scrambling the
linear reading order (a top cause of NID loss on affected docs) and
scoring 0 on table structure.
Add is_page_number_toc: a narrow (2-3 col) list whose last column is
mostly page numbers (short integers or roman numerals) that are mostly
non-decreasing, with a text-title first column and NO header row (a
TOC's first row is already an entry). Such tables now route through the
existing flat-list TOC renderer.
The no-header + narrow-width + monotonic guards keep real data tables
intact — e.g. a 4-column regional table, or a 2-column "Mineral | CEC"
table with a header row and ascending values.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: bump reading-order (NID) benchmark to 0.89
Reflects the phantom-TOC fix in this PR: NID 0.88 -> 0.89 on the
200-doc benchmark. Other cells are unchanged at 2-decimal precision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: canonical roman validation, real-first-row header check, roman page cells
- page_number_value now requires a *canonical* roman numeral (re-encode
and compare), so words like "civil"/"mix"/"ill" are no longer parsed
as page numbers.
- The no-header guard checks the actual first row's last cell instead of
the first non-empty one, so a blank header cell ("Category | ") still
rejects the TOC heuristic.
- format::is_page_number_cell recognizes canonical roman numerals, so
roman front-matter pages (vii, ix) get proper title/page separation in
the flat TOC list.
- Fix the non-monotonic test to use 5 rows so it exercises the
monotonicity guard rather than the row-count early return; add
roman-lookalike and blank-header rejection tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: share roman helper, widen length to 8, require page-span for TOC
- Extract canonical_roman_value + to_roman_lower into tables/mod.rs and
use them from both the TOC detector and the formatter, removing the
duplicated mapping/loop and keeping them in sync. The shared helper
accepts ≤8 chars, so longer front-matter numerals (xxxviii) flatten
consistently on both sides.
- Add a page-span guard to is_page_number_toc: real page numbers skip
through the document (range >> entry count), so a dense consecutive
ordinal/rank/ID column (1,2,3,…) is rejected — monotonicity alone did
not separate those data tables from contents.
Costs ~0.001 aggregate on the benchmark (NID 0.888->0.887) for the added
precision; still a clear win over baseline (NID 0.883, TEDS unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: recover consecutive-page TOCs via a title signal
The strict page-span rule rejected legitimate one-page-per-entry TOCs
(range ~= entry count). Relax it: accept any page sequence with a gap
(nearly all real contents). Only a *perfectly dense* consecutive run —
which rank/ID/ordinal columns produce, but a chapter-per-page TOC can
too — falls back to a title signal: flatten when the first-column
entries average multi-word headings, keep as a table when they are the
short single-word labels typical of leaderboards/ID lists.
Recovers the ~0.001 the range-only rule cost (NID back to 0.888) while
still rejecting dense ordinal data tables.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Re-ran opendataloader-bench on current main. pdf-inspector improved
across the board since the last table: overall 0.78→0.83, tables
0.59→0.66, headings 0.57→0.74 (competitor rows unchanged). Updated the
prose — heading detection no longer lags opendataloader, and overall is
now within 0.01 of it at ~2.5× the speed.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Cargo.toml and pyproject.toml already declare MIT but the repo had no
LICENSE file, so GitHub and package registries couldn't display it.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(extractor): geometric underline detection on TextItem (ENG-5015)
PDFs carry no underline font flag — underlines are stroked horizontal
lines or thin filled rects drawn under the baseline. Correlate those
graphics (already parsed from the content stream) with text items in a
post-pass: a rule within ~0.35em below the baseline covering >=60% of
an item's width marks is_underline.
Exposed through the napi and python bindings. Verified on real docs:
4/4 underlined sentences flagged on a Japanese report, links/headings
flagged on 8 of 10 underline-bearing eval docs, zero flags on docs
without underlines. Known FP source (table cell borders) documented —
downstream applies inline styling only to plain-text regions.
napi 1.9.8 -> 1.9.9.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): underline rules only from painted rects, normalized extents (review)
Two review fixes: (1) normalize rect extents before the thickness/width
checks — `re` operands pass through the CTM so width/height can be
negative, which missed negative-width rules and let negative-height
bands pass as thin; (2) only feed painted rects to underline detection —
`re` rects now wait in a pending list until a paint operator (S/s, f/F/
f*, B/B*/b/b*) confirms them, and `re W n` clip-only paths are discarded
at `n`, so invisible clip boundaries no longer underline nearby text.
Marking moved into content_stream where paint state lives (pre-rotation,
consistent device space).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): harden underline detection
* feat(cli): export positioned text item json
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Compare pdf-inspector against other direct text extraction engines
(no OCR/ML) on the opendataloader-bench corpus.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move detailed Python, Rust, and debugging docs into docs/ to keep
the main README focused on overview and quick start examples.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both bindings now expose the same 6 function families: process, detect,
classify, extractText, extractTextWithPositions, and extractTextInRegions.
Bumps PyO3 from 0.22 to 0.25 for Python 3.14 support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Expose the pdf-inspector Rust library as a Python package using PyO3 + maturin.
Python users can now `pip install` and use `import pdf_inspector` for PDF
classification, text extraction, and markdown conversion with native Rust speed.
Adds:
- src/python.rs: PyO3 bindings (process_pdf, detect_pdf, extract_text, etc.)
- pyproject.toml: maturin build configuration
- pdf_inspector.pyi: type stubs for IDE support
- tests/test_python.py: 21 pytest tests covering all Python API functions
- examples/basic_usage.py: example script demonstrating all features
- Updated README with Python quick start and API reference
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace hardcoded max_pages_to_sample with a ScanStrategy enum that
supports EarlyExit (default), Full, Sample(n), and Pages(vec) modes.
Add process_pdf_with_config and process_pdf_mem_with_config to the
public API. Update README with usage examples and strategy docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
