mu-ref/pdf-inspector
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(bench): compare OpenDataLoader builds (#175)

Browse Source 
* test(bench): compare OpenDataLoader builds

* docs(bench): keep reference comparisons generic

* fix(bench): keep regression gates complete

* fix(bench): clarify missing reference gates

* fix(bench): validate nonnegative limits

* fix(bench): isolate prediction runs

* chore(bench): refresh review
This commit is contained in:
Abimael Martell
2026-07-16 14:25:25 -07:00
committed by GitHub
parent 64a0930f9f
commit 0c06dac976
7 changed files with 598 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+8 -6
View File
@@ -27,16 +27,18 @@ Evaluated on the [opendataloader-bench](https://github.com/opendataloader-projec
| Engine | Overall | Reading Order (NID) | Tables (TEDS) | Headings (MHS) | Speed (200 docs) |
|---|---|---|---|---|---|
| pdf-inspector | 0.83 | 0.89 | 0.66 | 0.74 | 4s |
| opendataloader | 0.84 | 0.91 | 0.49 | 0.74 | 11s |
| pdf-inspector | **0.875** | **0.915** | **0.814** | 0.788 | 3.3s |
| opendataloader | 0.831 | 0.902 | 0.489 | 0.739 | 3.0s |
| pymupdf4llm | 0.73 | 0.89 | 0.40 | 0.41 | 18s |
| markitdown | 0.58 | 0.88 | 0.00 | 0.00 | 8s |
| markitdown | 0.59 | 0.84 | 0.27 | 0.00 | 23s |
For context, engines that use OCR/ML (docling, marker, mineru) score 0.83-0.88 overall but take 2-180 minutes on the same corpus — pdf-inspector reaches the low end of that range without any OCR, in 4 seconds.
For context, engines that use OCR/ML (docling, marker, mineru) score 0.83-0.88 overall but take 2-180 minutes on the same corpus — pdf-inspector reaches the top of that range without any OCR, in 3.3 seconds.
**Where we do well:** Speed (fastest of all engines), the best table detection of any engine shown, and heading detection now on par with opendataloader. Overall lands within 0.01 of opendataloader at roughly 2.5× the speed.
**Where we do well:** The best overall, reading-order, and table scores among the direct extraction engines shown.
**Where we lag:** Reading order still trails opendataloader slightly, and table structure trails OCR-based engines that can see visual layout.
**Where we lag:** Some direct engines remain slightly faster, and OCR-based engines can recover text that has no usable PDF text layer.
Use the [paired benchmark harness](docs/benchmarking.md) to compare two local builds against the exact same corpus and evaluator revision.
## Quick start