When the histogram-based column detector finds no valleys (common with
sidebar/asymmetric layouts), fall back to a simplified XY-cut: find the
largest horizontal gap between item edges and split there if both sides
have enough items with vertical overlap.
Inspired by opendataloader's XY-Cut++ algorithm but implemented as a
single-level fallback rather than full recursive segmentation.
Doc 156: NID 0.545→0.966, Doc 157: NID 0.564→0.962.
NID-S +0.007, TEDS-S +0.066 across 200 docs. No regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PDFs that draw table borders as thin filled rectangles (< 2pt, common in
spreadsheet exports) were invisible to both rect-based and line-based
table detectors. Now converts these thin rects to PdfLine objects and
runs line-based detection, but ONLY as a last resort after all other
methods (rect, line, heuristic, column-based) found nothing.
This avoids the regression from the earlier attempt which ran synthesis
at step 2, preempting the heuristic detector on PDFs where it worked
better.
Also relaxes uniform row spacing threshold (CV 0.05→0.02) to accept
spreadsheet-exported tables with even row heights.
Benchmark: TEDS 0.519→0.586 (+0.067), overall +0.006, no regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reverts commits 937311c, 1dcb0c6, 0300e96, 999f9a2. The thin-rect-to-line
synthesis and stacked table splitting improved extraction for specific
government PDFs but caused -0.05 TEDS regression on the benchmark by
preempting the heuristic detector with worse line-based grids.
These features need more targeted guards before re-enabling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rows that sit between horizontal rules but lack vertical border coverage
are not table cells — they're freestanding text (e.g. "Note: The cutoff
mark is out of 120"). These rows now split the grid into separate
sub-tables, with the unbounded text emitted as plain text between them.
Single-cell "tables" (from the split) render as plain text instead of
a degenerate 1x1 markdown table.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rows with 3+ short-valued cells (avg ≤10 chars) and an empty first cell
are column headers (e.g. "UR | SC | ST | OBC | EWS"), not text overflow
from the previous row. Prevents them from being merged into the
preceding section title row.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three fixes for better table extraction from spreadsheet-exported PDFs:
1. Convert thin filled rects (< 2pt) to PdfLine objects before line-based
table detection. Many PDFs draw table borders as narrow filled rectangles
instead of stroked paths — these were invisible to the line detector.
2. Relax uniform row spacing rejection (CV 0.05 → 0.02). Spreadsheet
exports have very even row heights that were being rejected as "chart
grids".
3. Fix continuation row merging: don't merge rows where the only non-first
cell content is a long label (section headers like "Category No. 03").
Don't merge first-cell-only rows with long text ("Note: ...").
Also adds multi-Y row splitting in line-based detection and column-aware
table detection skipping for multi-column pages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Many PDFs (especially spreadsheet exports) draw table borders as thin
filled rectangles (height/width < 2pt) instead of stroked paths. These
were invisible to our line-based table detector since only stroke
operations produced PdfLine objects.
Now synthesizes PdfLine from thin rects before line-based detection,
enabling table detection on border-drawn PDFs like government forms.
Also relaxes the uniform row spacing rejection threshold (CV 0.05→0.02)
to accept spreadsheet-exported tables with even row heights.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On pages where column detection finds 2+ columns, skip body-font
heuristic table detection in the merged-band retry path. This prevents
sidebar/two-column prose from being formatted as markdown tables.
The fix is targeted: per-band heuristic detection still runs (bands
are scoped to single columns), so real tables within columns are
still detected. Only the merged-band retry (which sees all items
across columns) is gated.
Also relaxes column validation to accept asymmetric layouts (sidebars)
where one side has fewer items, and tries center-based item assignment
before edge-based to improve column splitting for asymmetric layouts.
Benchmark: NID 0.865→0.869, NID-S 0.798→0.805, overall +0.002.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace ad-hoc bold/ratio heading checks with a unified scoring system
based on font size rarity. For each line, compute:
score = font_rarity * 0.5 + bold * 0.3 + standalone * 0.2
Font rarity measures how infrequently a font size appears across the
document — heading fonts are rare while body text is common. This
approach (from opendataloader's ModeWeightStatistics) naturally adapts
to each document's font distribution instead of relying on fixed
thresholds.
Guards: require font_size >= 0.95 * base_size (no small-font headings),
word_count >= 3, and standalone (paragraph break before).
Benchmark improvement: MHS 0.56→0.58, MHS-S 0.66→0.70, overall +0.003.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compare pdf-inspector against other direct text extraction engines
(no OCR/ML) on the opendataloader-bench corpus.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lines with font size 1.10-1.20x body text that are standalone and short
(1-8 words) are promoted to headings. This catches academic paper
headings where the font is only ~10% larger than body text, below the
previous 1.2x threshold.
Also syncs the simpler to_markdown_from_lines path to match the
table-aware path (removes stale colon exclusion).
Benchmark improvement: MHS 0.54→0.56, overall 0.761→0.766.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Caption detection was incorrectly classifying "Table of Contents" as a
caption because it starts with "Table ". Now "Table" and "Figure"
prefixes require a digit, parenthesis, or hash after them — matching
actual captions like "Table 1", "Figure 3.2" but not titles.
Also removes debug logging left from previous iteration.
Benchmark improvement: MHS 0.52→0.54, overall 0.757→0.761.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The colon exclusion was preventing legitimate headings like "Steps for
Using the Microscope:" and "Changing objectives:" from being detected.
The single edge case it was protecting (chart sub-headers) is less
impactful than the many headings it was blocking.
Benchmark improvement: MHS 0.51→0.52, MHS-S 0.61→0.62.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Lower minimum item count for body-font table candidates from 9 to 6,
allowing small 2-3 row tables to be detected.
- Allow 2-column body-font tables with short cells (avg ≤25 chars) to
bypass the "table-like content" validation. This catches text-only
definition/category tables (e.g., species lists) without false-positiving
on 2-column paragraph text (which has longer cells).
Benchmark improvement: TEDS 0.498→0.519, overall 0.750→0.754.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bold lines at body font size that are standalone (preceded by a paragraph
break) and have ≥3 words are promoted to headings. This catches the
common pattern in academic/technical PDFs where section headings use
bold text at the same size as body text.
Guards against false positives: minimum word count, colon-ending
exclusion (labels like "Table I:").
Benchmark improvement: MHS 0.37→0.50, overall 0.71→0.75.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align region filtering with rotated-page coordinate rewrites, switch region text assembly to the shared line-grouping pipeline, and retain edge-overlap text to avoid false empty regions that incorrectly trigger OCR fallback. Also make Python region inputs fail fast with clear ValueError messages for malformed boxes.
Made-with: Cursor
Two bugs in collect_text_in_region / extract_text_in_regions_mem:
1. The threshold-based sort comparator in collect_text_in_region was not
transitive, causing Rust's sort to panic on certain PDFs. Replaced with
strict total_cmp ordering — the line-grouping phase already handles
fuzzy Y matching via threshold.
2. The needs_ocr check was missing is_cid_garbage, so Identity-H fonts
with CID garbage (C1 control chars, high Latin mojibake) could pass
all quality checks and be served as real text with needs_ocr=false.
Also adds 7 integration tests for extract_text_in_regions_mem (previously
had zero coverage): basic extraction, Identity-H needs_ocr, multiple
regions, nonexistent page, empty region, invalid input, and a fast-vs-normal
comparison test across all text-based fixtures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
FontCMaps::from_doc can spend 4.5+ seconds decompressing and parsing
large embedded TrueType fonts for CID fonts with sparse ToUnicode
CMaps. For extract_text_in_regions (hybrid OCR pipeline), this is
unnecessary — fonts that can't be decoded cheaply will produce
empty/garbage text, triggering needs_ocr=true and GPU OCR fallback.
Changes:
- Add FontCMaps::from_doc_pages_fast() that skips TrueType font
fallback parsing (build_fallback_cmap_for_type0) and Identity-H/V
second pass entirely
- Add FontCMaps::from_doc_pages() for filtered page sets
- extract_text_in_regions_mem uses fast mode
- Restructure fallback chain: try cheap fallbacks first, only attempt
expensive TrueType parsing when needed and not in fast mode
Benchmark on nihms-1771367.pdf (19-page chemistry paper):
- FontCMaps fast: 201µs
- FontCMaps slow: 4.47s
- 22,000x speedup on font parsing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rust panics in NAPI modules abort the Node.js process with no chance
to report errors. This wraps every exported function in catch_unwind,
converting panics into JS Error exceptions that can be caught and
reported to Sentry.
Buffer data is extracted to Vec<u8> before the catch_unwind boundary
to satisfy UnwindSafe requirements.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
upload-artifact strips the common napi/ prefix, so files are at
artifacts/js-bindings/index.js not artifacts/js-bindings/napi/index.js.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
napi pre-publish expects a version-bump commit message convention.
Instead, upload index.js and index.d.ts generated by napi build
as artifacts and copy them into the publish step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The command is `napi pre-publish` (hyphenated), not `napi prepublish`,
and `--skip-gh-release` is not a valid flag.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
index.js and index.d.ts are generated by napi-rs. Generate them
at publish time via `napi prepublish` instead of checking them in.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix sort panics on NaN values from bogus PDF font metrics
Replace all `partial_cmp(...).unwrap_or(Ordering::Equal)` and bare
`partial_cmp(...).unwrap()` with `total_cmp()` across the codebase.
`partial_cmp` returns `None` for NaN, and mapping that to `Equal`
violates total ordering: `a == NaN` and `NaN == b` but `a != b`.
Rust 1.81+ detects this and panics in sort_by. `total_cmp` handles
NaN deterministically (sorts to end) and guarantees total ordering.
The critical crash was in `extract_text_in_regions` (lib.rs:478)
where PDFs with bogus font ascent/descent values produced NaN in
text item coordinates, causing process abort via NAPI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix missed partial_cmp in layout.rs and restore napi exports
- Convert two remaining b.y.partial_cmp(&a.y) calls to total_cmp
in group_single_column and column layout sorting
- Restore missing napi exports: detectPdf, extractText,
extractTextWithPositions, processPdf
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move detailed Python, Rust, and debugging docs into docs/ to keep
the main README focused on overview and quick start examples.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Drop platform-specific optional deps — ship all .node binaries in one
package (~4 MB total). Simplifies publishing and consumer install.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skip napi prepublish/artifacts commands that require GitHub API auth.
Instead, create platform package.json files and copy binaries directly.
Add optionalDependencies to main package so npm/bun auto-selects the
right binary.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Both bindings now expose the same 6 function families: process, detect,
classify, extractText, extractTextWithPositions, and extractTextInRegions.
Bumps PyO3 from 0.22 to 0.25 for Python 3.14 support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
