mu-ref/pdf-inspector
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(bench): compare OpenDataLoader builds (#175)

Browse Source 
* test(bench): compare OpenDataLoader builds

* docs(bench): keep reference comparisons generic

* fix(bench): keep regression gates complete

* fix(bench): clarify missing reference gates

* fix(bench): validate nonnegative limits

* fix(bench): isolate prediction runs

* chore(bench): refresh review
This commit is contained in:
Abimael Martell
2026-07-16 14:25:25 -07:00
committed by GitHub
parent 64a0930f9f
commit 0c06dac976
7 changed files with 598 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/benchmarking.md
+29
View File
@@ -0,0 +1,29 @@
# Benchmarking against OpenDataLoader
The paired harness runs two `pdf2md` binaries through the same local
OpenDataLoader corpus, evaluates both outputs, and reports aggregate and
per-document deltas. This avoids comparing results produced from different
corpus revisions or evaluator versions.
Build a candidate and provide a released or worktree build as the baseline:
```bash
cargo build --release
python3 scripts/bench_opendataloader.py \
--bench-dir ../opendataloader-bench \
--baseline ../pdf-inspector-main/target/release/pdf2md \
--candidate target/release/pdf2md \
--max-document-regression 0.02 \
--json-output /tmp/pdf-inspector-benchmark.json
```
Pass `--reference-evaluation path/to/evaluation.json` to report the candidate
delta against another evaluation, and add `--require-reference-lead` to make a
negative reference delta fail the run. By default, the candidate must not
regress the baseline overall score or introduce missing predictions. Use
`--min-overall-delta` to require a specific aggregate gain.
The OpenDataLoader repository is external and keeps its normal
`prediction/pdf-inspector` output. Paired evaluation copies each run into a
temporary directory before evaluating it, so the baseline and candidate cannot
overwrite one another.
docs/python.md
+2 -2
View File
@@ -18,8 +18,8 @@ Built by [Firecrawl](https://firecrawl.dev) to handle text-based PDFs locally in
| Engine | Overall | Reading order | Tables (TEDS) | Headings | Speed |
|---|---|---|---|---|---|
| **pdf-inspector** | 0.83 | 0.88 | **0.66** | 0.74 | **4s** |
| opendataloader | 0.84 | 0.91 | 0.49 | 0.74 | 11s |
| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | 3.3s |
| opendataloader | 0.831 | 0.902 | 0.489 | 0.739 | 3.0s |
| pymupdf4llm | 0.73 | 0.89 | 0.40 | 0.41 | 18s |
OCR/ML engines (docling, marker, mineru) score 0.83–0.88 overall but take 2–180 minutes on the same corpus. Full numbers in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark).
docs/rust-api.md
+2 -2
View File
@@ -18,8 +18,8 @@ Built by [Firecrawl](https://firecrawl.dev) to handle text-based PDFs locally in
| Engine | Overall | Reading order | Tables (TEDS) | Headings | Speed |
|---|---|---|---|---|---|
| **pdf-inspector** | 0.83 | 0.88 | **0.66** | 0.74 | **4s** |
| opendataloader | 0.84 | 0.91 | 0.49 | 0.74 | 11s |
| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | 3.3s |
| opendataloader | 0.831 | 0.902 | 0.489 | 0.739 | 3.0s |
| pymupdf4llm | 0.73 | 0.89 | 0.40 | 0.41 | 18s |
OCR/ML engines (docling, marker, mineru) score 0.83–0.88 overall but take 2–180 minutes on the same corpus. Full numbers in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark).