fix(extractor): preserve mixed-case glyph boundaries

fix(review): convention gate on both tiers; Han/Kana floor always infinite (PR #133 review)
- Long lowercase spaced-single runs ("a b c d e") had the tracked gap shape in the >=4-gap tier with no convention guard — word boundaries lost. The caps/CJK/title-case gate now applies to BOTH tiers; a title-case single word ("B u f f a l o") also qualifies. - Han/Kana runs skipped straight to the bimodal split, so a nonuniform gap distribution (justification, punctuation spacing) could manufacture a word boundary. Han/Kana now always floors at infinity; Hangul deliberately keeps word-boundary handling — Korean spaces between words (is_spaceless_cjk excludes it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
2026-07-11 00:24:59 -07:00 · 2026-07-10 23:16:01 -07:00 · 2026-07-10 23:04:45 -07:00
2 changed files with 5 additions and 5 deletions
@@ -26,16 +26,16 @@ Evaluated on the [opendataloader-bench](https://github.com/opendataloader-projec

 | Engine | Overall | Reading Order (NID) | Tables (TEDS) | Headings (MHS) | Speed (200 docs) |
 |---|---|---|---|---|---|
-| pdf-inspector | 0.83 | 0.88 | 0.66 | 0.74 | 4s |
+| pdf-inspector | 0.78 | 0.87 | 0.59 | 0.57 | 4s |
 | opendataloader | 0.84 | 0.91 | 0.49 | 0.74 | 11s |
 | pymupdf4llm | 0.73 | 0.89 | 0.40 | 0.41 | 18s |
 | markitdown | 0.58 | 0.88 | 0.00 | 0.00 | 8s |

-For context, engines that use OCR/ML (docling, marker, mineru) score 0.83-0.88 overall but take 2-180 minutes on the same corpus — pdf-inspector reaches the low end of that range without any OCR, in 4 seconds.
+For context, engines that use OCR/ML (docling, marker, mineru) score 0.83-0.88 overall but take 2-180 minutes on the same corpus.

-**Where we do well:** Speed (fastest of all engines), the best table detection of any engine shown, and heading detection now on par with opendataloader. Overall lands within 0.01 of opendataloader at roughly 2.5× the speed.
+**Where we do well:** Speed (fastest of all engines), reading order, table detection vs other direct-text tools.

-**Where we lag:** Reading order still trails opendataloader slightly, and table structure trails OCR-based engines that can see visual layout.
+**Where we lag:** Heading detection trails opendataloader — many PDFs use bold text at body font size for headings, or headings that are only slightly larger than body text. Table detection trails OCR-based engines that can see visual table structure.

 ## Quick start

@@ -1,6 +1,6 @@
 {
  "name": "@firecrawl/pdf-inspector",
-  "version": "1.10.1",
+  "version": "1.10.0",
  "description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
  "main": "index.js",
  "types": "index.d.ts",