Replace ad-hoc bold/ratio heading checks with a unified scoring system
based on font size rarity. For each line, compute:
score = font_rarity * 0.5 + bold * 0.3 + standalone * 0.2
Font rarity measures how infrequently a font size appears across the
document — heading fonts are rare while body text is common. This
approach (from opendataloader's ModeWeightStatistics) naturally adapts
to each document's font distribution instead of relying on fixed
thresholds.
Guards: require font_size >= 0.95 * base_size (no small-font headings),
word_count >= 3, and standalone (paragraph break before).
Benchmark improvement: MHS 0.56→0.58, MHS-S 0.66→0.70, overall +0.003.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bold lines at body font size that are standalone (preceded by a paragraph
break) and have ≥3 words are promoted to headings. This catches the
common pattern in academic/technical PDFs where section headings use
bold text at the same size as body text.
Guards against false positives: minimum word count, colon-ending
exclusion (labels like "Table I:").
Benchmark improvement: MHS 0.37→0.50, overall 0.71→0.75.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix sort panics on NaN values from bogus PDF font metrics
Replace all `partial_cmp(...).unwrap_or(Ordering::Equal)` and bare
`partial_cmp(...).unwrap()` with `total_cmp()` across the codebase.
`partial_cmp` returns `None` for NaN, and mapping that to `Equal`
violates total ordering: `a == NaN` and `NaN == b` but `a != b`.
Rust 1.81+ detects this and panics in sort_by. `total_cmp` handles
NaN deterministically (sorts to end) and guarantees total ordering.
The critical crash was in `extract_text_in_regions` (lib.rs:478)
where PDFs with bogus font ascent/descent values produced NaN in
text item coordinates, causing process abort via NAPI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix missed partial_cmp in layout.rs and restore napi exports
- Convert two remaining b.y.partial_cmp(&a.y) calls to total_cmp
in group_single_column and column layout sorting
- Restore missing napi exports: detectPdf, extractText,
extractTextWithPositions, processPdf
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 5 stripped public-domain PDF fixtures and golden markdown snapshots
for CI regression testing. Fix non-deterministic output caused by
HashMap iteration order in font stats, table heuristics, and rect
clustering by adding deterministic tie-breaking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
