From a910b7df1d2fbd03817c90e9c6fd914cd016f221 Mon Sep 17 00:00:00 2001
From: Abimael Martell <1450169+abimaelmartell@users.noreply.github.com>
Date: Thu, 16 Jul 2026 17:43:53 -0700
Subject: [PATCH] docs(benchmark): refresh parser comparison (#178)

* docs: refresh benchmark comparison

* docs(site): refresh benchmark section

* docs: reframe benchmark positioning

* docs: focus benchmark positioning on best fit
---
 README.md            | 17 +++++++++--------
 docs/benchmarking.md |  8 ++++++++
 docs/python.md       | 12 +++++++-----
 docs/rust-api.md     | 12 +++++++-----
 napi/README.md       | 12 +++++++-----
 site/index.html      | 19 +++++++++----------
 6 files changed, 47 insertions(+), 33 deletions(-)

diff --git a/README.md b/README.md
index 4617985..e90203e 100644
--- a/README.md
+++ b/README.md
@@ -23,20 +23,21 @@ Built by [Firecrawl](https://firecrawl.dev) to handle text-based PDFs locally in
 
 ## Benchmark
 
-Evaluated on the [opendataloader-bench](https://github.com/opendataloader-project/opendataloader-bench) corpus (200 PDFs). Only direct text extraction engines are shown — no OCR, no ML models. Scores are 0-1, higher is better.
+Evaluated on the [opendataloader-bench](https://github.com/opendataloader-project/opendataloader-bench) corpus (200 PDFs). Only local engines without model-based PDF parsing are shown; OCR was disabled. Scores are 0-1, higher is better.
 
 | Engine | Overall | Reading Order (NID) | Tables (TEDS) | Headings (MHS) | Speed (200 docs) |
 |---|---|---|---|---|---|
-| pdf-inspector | **0.875** | **0.915** | **0.814** | 0.788 | 3.3s |
-| opendataloader | 0.831 | 0.902 | 0.489 | 0.739 | 3.0s |
-| pymupdf4llm | 0.73 | 0.89 | 0.40 | 0.41 | 18s |
-| markitdown | 0.59 | 0.84 | 0.27 | 0.00 | 23s |
+| pdf-inspector | **0.875** | **0.915** | **0.814** | 0.788 | **2.8s** |
+| liteparse | 0.870 | 0.908 | 0.693 | **0.811** | 13.9s |
+| opendataloader | 0.843 | 0.912 | 0.489 | 0.760 | 9.8s |
+| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 15.5s |
+| markitdown | 0.583 | 0.879 | 0.000 | 0.000 | 6.7s |
 
-For context, engines that use OCR/ML (docling, marker, mineru) score 0.83-0.88 overall but take 2-180 minutes on the same corpus — pdf-inspector reaches the top of that range without any OCR, in 3.3 seconds.
+Results were refreshed on July 16, 2026, on an Apple M4 Pro. Engine versions were pdf-inspector 0.1.6, LiteParse 2.6.0, OpenDataLoader 2.1.1, PyMuPDF4LLM 0.2.0, and MarkItDown 0.1.4. Speed is the median of three complete corpus runs.
 
-**Where we do well:** The best overall, reading-order, and table scores among the direct extraction engines shown.
+For context, engines that use OCR or model-based document parsing (docling, marker, mineru) score 0.83-0.88 overall but take 2-180 minutes on the same corpus — pdf-inspector reaches the top of that range without either, in 2.8 seconds.
 
-**Where we lag:** Some direct engines remain slightly faster, and OCR-based engines can recover text that has no usable PDF text layer.
+**Best fit:** Native-text PDFs where speed, reading order, and table structure matter. pdf-inspector delivered the highest overall, reading-order, and table scores, along with the fastest complete run in this benchmark. That makes it a strong local default for reports, research papers, financial documents, invoices, and legal PDFs that need clean, structured Markdown without adding OCR latency or infrastructure.
 
 Use the [paired benchmark harness](docs/benchmarking.md) to compare two local builds against the exact same corpus and evaluator revision.
 
diff --git a/docs/benchmarking.md b/docs/benchmarking.md
index fc0a8b7..b00e046 100644
--- a/docs/benchmarking.md
+++ b/docs/benchmarking.md
@@ -28,6 +28,14 @@ The OpenDataLoader repository is external and keeps its normal
 temporary directory before evaluating it, so the baseline and candidate cannot
 overwrite one another.
 
+## Published comparison protocol
+
+The public benchmark table was refreshed on July 16, 2026, on an Apple M4 Pro
+using pdf-inspector 0.1.6, LiteParse 2.6.0, OpenDataLoader 2.1.1,
+PyMuPDF4LLM 0.2.0, and MarkItDown 0.1.4. Every engine processed the same 200
+PDFs with OCR disabled. Reported speed is the median of three complete corpus
+runs; quality scores come from the benchmark evaluator over all 200 outputs.
+
 ## Optional backend evidence probe
 
 The evidence probe compares positioned `pdf2md` items with MuPDF structured
diff --git a/docs/python.md b/docs/python.md
index 4a6cf77..7417ae0 100644
--- a/docs/python.md
+++ b/docs/python.md
@@ -14,15 +14,17 @@ Built by [Firecrawl](https://firecrawl.dev) to handle text-based PDFs locally in
 
 ## Benchmark
 
-[opendataloader-bench](https://github.com/opendataloader-project/opendataloader-bench) corpus (200 PDFs), direct-extraction engines only — no OCR, no ML. Scores 0–1, higher is better:
+[opendataloader-bench](https://github.com/opendataloader-project/opendataloader-bench) corpus (200 PDFs), local engines without model-based PDF parsing; OCR disabled. Scores 0–1, higher is better:
 
 | Engine | Overall | Reading order | Tables (TEDS) | Headings | Speed |
 |---|---|---|---|---|---|
-| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | 3.3s |
-| opendataloader | 0.831 | 0.902 | 0.489 | 0.739 | 3.0s |
-| pymupdf4llm | 0.73 | 0.89 | 0.40 | 0.41 | 18s |
+| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | **2.8s** |
+| liteparse | 0.870 | 0.908 | 0.693 | **0.811** | 13.9s |
+| opendataloader | 0.843 | 0.912 | 0.489 | 0.760 | 9.8s |
+| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 15.5s |
+| markitdown | 0.583 | 0.879 | 0.000 | 0.000 | 6.7s |
 
-OCR/ML engines (docling, marker, mineru) score 0.83–0.88 overall but take 2–180 minutes on the same corpus. Full numbers in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark).
+Refreshed July 16, 2026, on Apple M4 Pro; speed is the median of three complete corpus runs. Full methodology and versions are in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark).
 
 ## Install
 
diff --git a/docs/rust-api.md b/docs/rust-api.md
index a80133a..d99fe30 100644
--- a/docs/rust-api.md
+++ b/docs/rust-api.md
@@ -14,15 +14,17 @@ Built by [Firecrawl](https://firecrawl.dev) to handle text-based PDFs locally in
 
 ## Benchmark
 
-[opendataloader-bench](https://github.com/opendataloader-project/opendataloader-bench) corpus (200 PDFs), direct-extraction engines only — no OCR, no ML. Scores 0–1, higher is better:
+[opendataloader-bench](https://github.com/opendataloader-project/opendataloader-bench) corpus (200 PDFs), local engines without model-based PDF parsing; OCR disabled. Scores 0–1, higher is better:
 
 | Engine | Overall | Reading order | Tables (TEDS) | Headings | Speed |
 |---|---|---|---|---|---|
-| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | 3.3s |
-| opendataloader | 0.831 | 0.902 | 0.489 | 0.739 | 3.0s |
-| pymupdf4llm | 0.73 | 0.89 | 0.40 | 0.41 | 18s |
+| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | **2.8s** |
+| liteparse | 0.870 | 0.908 | 0.693 | **0.811** | 13.9s |
+| opendataloader | 0.843 | 0.912 | 0.489 | 0.760 | 9.8s |
+| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 15.5s |
+| markitdown | 0.583 | 0.879 | 0.000 | 0.000 | 6.7s |
 
-OCR/ML engines (docling, marker, mineru) score 0.83–0.88 overall but take 2–180 minutes on the same corpus. Full numbers in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark).
+Refreshed July 16, 2026, on Apple M4 Pro; speed is the median of three complete corpus runs. Full methodology and versions are in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark).
 
 ## Install
 
diff --git a/napi/README.md b/napi/README.md
index b60ff8c..a9a9d9b 100644
--- a/napi/README.md
+++ b/napi/README.md
@@ -14,15 +14,17 @@ Built by [Firecrawl](https://firecrawl.dev) for hybrid OCR pipelines — extract
 
 ## Benchmark
 
-[opendataloader-bench](https://github.com/opendataloader-project/opendataloader-bench) corpus (200 PDFs), direct-extraction engines only — no OCR, no ML. Scores 0–1, higher is better:
+[opendataloader-bench](https://github.com/opendataloader-project/opendataloader-bench) corpus (200 PDFs), local engines without model-based PDF parsing; OCR disabled. Scores 0–1, higher is better:
 
 | Engine | Overall | Reading order | Tables (TEDS) | Headings | Speed |
 |---|---|---|---|---|---|
-| **pdf-inspector** | 0.83 | 0.88 | **0.66** | 0.74 | **4s** |
-| opendataloader | 0.84 | 0.91 | 0.49 | 0.74 | 11s |
-| pymupdf4llm | 0.73 | 0.89 | 0.40 | 0.41 | 18s |
+| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | **2.8s** |
+| liteparse | 0.870 | 0.908 | 0.693 | **0.811** | 13.9s |
+| opendataloader | 0.843 | 0.912 | 0.489 | 0.760 | 9.8s |
+| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 15.5s |
+| markitdown | 0.583 | 0.879 | 0.000 | 0.000 | 6.7s |
 
-OCR/ML engines (docling, marker, mineru) score 0.83–0.88 overall but take 2–180 minutes on the same corpus. Full numbers in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark).
+Refreshed July 16, 2026, on Apple M4 Pro; speed is the median of three complete corpus runs. Full methodology and versions are in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark).
 
 ## Install
 
diff --git a/site/index.html b/site/index.html
index 122e713..e2c631f 100644
--- a/site/index.html
+++ b/site/index.html
@@ -147,8 +147,7 @@
   tbody tr.us td:first-child::before { content: "▸ "; color: var(--accent); }
   .bench-foot { padding: 15px 18px; font-size: 13.5px; color: var(--ink-soft); background: var(--paper-2); border-top: 1px solid var(--line); }
   .bench-wrap { overflow-x: auto; }
-  .callouts { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; margin-top: 22px; }
-  @media (max-width: 640px) { .callouts { grid-template-columns: 1fr; } }
+  .callouts { display: grid; grid-template-columns: 1fr; gap: 16px; margin-top: 22px; }
   .callout { border-left: 3px solid var(--accent); padding: 4px 0 4px 16px; }
   .callout .ct { font-family: var(--mono); font-size: 11px; letter-spacing: 0.1em; text-transform: uppercase; color: var(--accent-deep); margin-bottom: 5px; }
   .callout p { font-size: 14.5px; color: var(--ink-soft); }
@@ -301,7 +300,7 @@
     <div class="sec-head">
       <span class="sec-num">02</span>
       <h2 class="sec-title">Fastest of the direct-text engines</h2>
-      <p class="sec-sub">Evaluated on the <a href="https://github.com/opendataloader-project/opendataloader-bench" style="color:var(--accent-deep);border-bottom:1px solid var(--line)">opendataloader-bench</a> corpus (200 PDFs). Direct text-extraction engines only — no OCR, no ML. Higher is better.</p>
+      <p class="sec-sub">Evaluated on the <a href="https://github.com/opendataloader-project/opendataloader-bench" style="color:var(--accent-deep);border-bottom:1px solid var(--line)">opendataloader-bench</a> corpus (200 PDFs). Local engines without model-based PDF parsing; OCR disabled. Higher is better.</p>
     </div>
     <div class="bench">
       <div class="bench-wrap">
@@ -310,18 +309,18 @@
           <tr><th>Engine</th><th>Overall</th><th>Reading order</th><th>Tables</th><th>Headings</th><th>200 docs</th></tr>
         </thead>
         <tbody>
-          <tr class="us"><td>pdf-inspector</td><td>0.83</td><td>0.89</td><td>0.66</td><td>0.74</td><td>4s</td></tr>
-          <tr><td>opendataloader</td><td>0.84</td><td>0.91</td><td>0.49</td><td>0.74</td><td>11s</td></tr>
-          <tr><td>pymupdf4llm</td><td>0.73</td><td>0.89</td><td>0.40</td><td>0.41</td><td>18s</td></tr>
-          <tr><td>markitdown</td><td>0.58</td><td>0.88</td><td>0.00</td><td>0.00</td><td>8s</td></tr>
+          <tr class="us"><td>pdf-inspector</td><td>0.875</td><td>0.915</td><td>0.814</td><td>0.788</td><td>2.8s</td></tr>
+          <tr><td>liteparse</td><td>0.870</td><td>0.908</td><td>0.693</td><td>0.811</td><td>13.9s</td></tr>
+          <tr><td>opendataloader</td><td>0.843</td><td>0.912</td><td>0.489</td><td>0.760</td><td>9.8s</td></tr>
+          <tr><td>pymupdf4llm</td><td>0.735</td><td>0.886</td><td>0.401</td><td>0.424</td><td>15.5s</td></tr>
+          <tr><td>markitdown</td><td>0.583</td><td>0.879</td><td>0.000</td><td>0.000</td><td>6.7s</td></tr>
         </tbody>
       </table>
       </div>
-      <div class="bench-foot">OCR/ML engines (docling, marker, mineru) score 0.83–0.88 overall — but take 2–180 minutes on the same corpus.</div>
+      <div class="bench-foot">Refreshed July 16, 2026, on Apple M4 Pro. Speed is the median of three complete corpus runs.</div>
     </div>
     <div class="callouts">
-      <div class="callout"><div class="ct">Where we win</div><p>Fastest engine measured, the best table detection of any engine here, and heading quality now on par with opendataloader — at ~2.5× its speed.</p></div>
-      <div class="callout"><div class="ct">Where we're working</div><p>Reading order still trails opendataloader slightly, and tables that need the visual structure only an OCR engine can see.</p></div>
+      <div class="callout"><div class="ct">Best fit</div><p>Native-text PDFs where speed, reading order, and table structure matter. pdf-inspector delivered the highest overall, reading-order, and table scores, along with the fastest complete run in this benchmark. That makes it a strong local default for reports, research papers, financial documents, invoices, and legal PDFs that need clean, structured Markdown without adding OCR latency or infrastructure.</p></div>
     </div>
   </div>
 </section>