* tables: expand multi-row TSR cells in place
Recover row-under-counted TSR tables by splitting overstuffed cells with native PDF text bands before falling back to heuristic extraction.
Made-with: Cursor
* docs: note multi-row expansion scope
Clarify that the row-band cap intentionally keeps v1 focused on common small row-loss cases while larger compressions continue to use heuristic fallback.
Made-with: Cursor
Three fixes to extract_tables_with_structure_auto_mem (added in
1.7.1) caught by external review:
1. multi_row_in_cell over-triggered on legitimate multi-line cells.
The previous threshold (item span > 1.3× either smallest cell or
tallest item height) fires on any cell with 2+ y-separated text
items — including rowspan>1 cells, wrapped descriptions, and
superscript/subscript runs. Replaced with two gates:
- skip cells whose declared rowspan > 1 (intentional multi-line)
- require an actual whitespace gap (>~half a line height)
between the bottom of one item and the top of the next, in
PDF-native y-coordinates. Same-line items with tall glyphs or
superscripts have negative or near-zero gap; truly separate
visual rows have gap ≈ leading − line-height.
FNBO regression test still passes; new test covers a rowspan=2
cell with two visible text lines and verifies no fallback fires.
2. Heuristic returning empty silently replaced TSR markdown with
"". The auto wrapper now keeps the TSR markdown when the
heuristic markdown is empty/whitespace and tags fallback_reason
with `_heuristic_empty` suffix (e.g.
`multi_row_in_cell_heuristic_empty`). Worst case we ship the
same wrong-but-non-empty TSR output we'd have shipped before
1.7.1; we never replace useful output with literally nothing.
3. One bad input blanked the whole batch. Errors from
detect_tsr_quality_issue or extract_tables_in_regions_mem now
stay scoped to the single input — that input falls through to
raw TSR markdown with a `_error` reason label so callers can
metric on it. Other inputs in the batch are unaffected.
3 new integration tests:
- test_auto_does_not_fire_on_legit_rowspan_cell
- test_auto_keeps_tsr_markdown_when_heuristic_returns_empty
- test_auto_isolates_per_input_failures
All 6 auto tests + full 123-test suite pass. FNBO local replay
still triggers fallback (phantom_empty_row signal in this run) and
emits correct Shawnee/BVP/Sonoma rows with correct census tracts.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds extract_tables_with_structure_auto_mem (Rust) /
extractTablesWithStructureAuto (napi). Returns
TableExtractionResult { markdown, fallback_reason } per input.
The wrapper runs the existing TSR-hybrid path then checks the
resulting cells for two known SLANet detection pathologies:
* phantom_empty_row: empty row sandwiched between non-empty rows
(cheap, cell-metadata only).
* multi_row_in_cell: re-reads PDF text items, flags any cell whose
contained items span >1.3× either the smallest cell height or
the tallest contained item height. Catches the FNBO failure mode
where a tall TSR cell absorbs two adjacent PDF rows.
When either fires, extract_tables_in_regions_mem runs over the same
crop bbox and its markdown replaces the TSR markdown.
fallback_reason carries the diagnostic label so callers can emit
metrics and watch each pathology independently.
Validated on FNBO branches PDF page 1 (Kansas region):
- TSR-only: merges Shawnee into BVP, wrong census tract on Sonoma.
- Auto fallback (phantom_empty_row): each row separate, correct tracts.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the row-merge pattern observed against FNBO branch list at the
College/Fairway boundary: when SLANet emits cells whose y-extents
overlap between consecutive rows, an item whose center fell in the
overlap region got pulled into BOTH cells, producing run-on cells
(e.g. "Kansas Kansas" + concatenated addresses).
Cause: stage 1's "for each cell, gather items inside" loop allowed an
item to match multiple cells. `normalize_cell_bands` reduces overlap
but is biased when cell-mean-center is offset from actual text baseline
(SLANet bboxes are typically taller than their text content), so the
midpoint-clamp can land on the wrong side of the row boundary, and
items at the boundary still match two cells.
Fix: invert the matching. For each PDF text item, find candidate cells
(those whose bbox satisfies tsr_region_contains_item — center inside
OR >=60% overlap on both axes), and assign to the cell whose CENTER
is geometrically closest. Build per-cell text from the assigned items.
Stage 2 (orphan recovery) is unchanged.
Exclusivity prevents item duplication across cells. The closest-center
rule disambiguates the cell-overlap case naturally without aggressive
band clamping. normalize_cell_bands stays — it tightens cells before
matching (smaller overlap → fewer ambiguous candidates) but is no
longer load-bearing for correctness of the overlap case.
Local replay against FNBO via api/scripts/local-tsr-replay.ts (which
exercises the full layout-pod → table-pod → pdf-inspector chain
in-process):
Pre-1.6.4 (deployed 1.6.3):
|LITH West|Illinois Illinois|11700 S. IL Route 47, Huntley IL...
|College|||0534.03|
|Fairway|Kansas Kansas|4650 College Blvd... 2828 Shawnee Mission...
Post-1.6.4 (this change):
|Huntley|Illinois|11700 S. IL Route 47, Huntley IL...|8711.15|
|LITH West|Illinois|4520 W Algonquin Rd, Lake in the Hills...
|College|Kansas|4650 College Blvd, Overland Park KS...|0532.01|
|Fairway|Kansas|2828 Shawnee Mission Pkwy, Fairway KS...|0500.00|
One row (Shawnee in the Kansas section) can still get dropped under
SLANet detection variance — the model occasionally under-detects rows
and emits N structure rows for N+1 PDF rows. The squeezed row's text
gets routed to the structurally-nearest existing cell. This is a
SLANet limitation, not addressable in pdf-inspector without
synthesizing rows from PDF text geometry; deferred.
Tests: 416 lib + 120 integration + 2 doctests pass. clippy + fmt clean.
Bump @firecrawl/pdf-inspector to 1.6.4 (patch — refines stage 1's
matching strategy; no API changes).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the residual run-on-cell pattern observed on FNBO branch list
after 1.6.2 deployed:
|Shawnee Blue Valley Parkway|Kansas|6301 Pflumm, ...|0523.04|
|Sonoma Plaza|Kansas Kansas|<addr1> <addr2>|0531.05|
|Mitchell Woonsocket|South Dakota|<addr1>|9628.01|
Cause: when col-N cells across multiple consecutive rows are y-shifted
the same way (a local SLANet drift), the stage 2 orphan pass sees
multiple orphans qualifying for the same nearest empty cell. The cell
gets all of them appended in order, producing "RowA-text RowB-text".
Fix: track the y-coordinate of the first orphan that lands in each
cell. Subsequent orphans only join that cell if their y is within
half-a-row-height of the first orphan's y (same line). Cross-line
orphans skip that cell and look for the next-nearest empty cell on
their own line.
Same-line slack preserves multi-token branch names like
"Blue Valley Parkway" (3 PDF text items at the same y) — all three
stack into the same cell. Cross-row stacking is what gets rejected.
Two new tests:
- stage2_rejects_cross_line_stacking_into_same_cell
Two orphans on different rows, both equidistant to the same empty
cell. First wins; second routes to its own row's cell.
- stage2_allows_same_line_orphans_to_stack_into_one_cell
Three same-line orphans (multi-token branch name) all land in the
same empty cell, joined by spaces.
The existing four stage-2 tests + the cell-bleed regression test from
PR #62 + #63 all pass: 416 lib + 115 integration + 2 doctests.
clippy + fmt clean.
Bump @firecrawl/pdf-inspector to 1.6.3 (patch — refines 1.6.2; no API
changes).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #62 (1.6.1) closed the cell-bleed regression by clamping SLANet's
loose cell bboxes into non-overlapping row/column bands and tightening
text membership to center-containment OR >=60% overlap. That worked,
but exposed the opposite failure: legitimate native PDF text whose
center fell just outside the *clamped* cell bbox now had nowhere to go.
Two distinct failure modes observed against the FNBO branch-list PDF
after 1.6.1 deployed:
Symptom A — header text positioned at the LEFT of a column whose band
was derived from data-cell centers farther right. Header "Address"
PDF text at x=331..375 fell outside the clamped col band starting
at x=410. Strict membership rejected it (0% x-overlap, center
outside).
Symptom B — local SLANet row drift in col 0 over a 5-row stretch.
Cell bboxes sat just above the actual branch-name text items
(x-overlap 100% but y-overlap ~30-43%, below the 60% threshold).
Both share one root: post-normalization bboxes are too tight, and the
strict rule has no escape valve for legitimate edge text.
Fix: add a stage-2 orphan-recovery pass after the strict fill. Items
that NO cell claimed in stage 1 get re-assigned to their nearest
*empty* cell, distance-capped by `(median_col_width, median_row_height)`
so a far-orphan figure title can't get pulled into a faraway empty
cell. Stage 2 only fills empties — never overwrites stage 1 — so the
cell-bleed case PR #62 closed cannot regress.
Three new lib tests cover the bug shapes:
- stage2_recovers_left_aligned_header_text_outside_data_band
(Symptom A: header text left-of-band, data cells already filled,
stage 2 fills only the header)
- stage2_recovers_y_shifted_col0_in_consecutive_rows
(Symptom B: 3 col-0 cells shifted vs text, all 3 recovered)
- stage2_does_not_overwrite_filled_cells_or_admit_far_orphans
(cap rejects far figure titles; pre-filled cells untouched)
Plus tests for the cap-derivation helper:
- tsr_assignment_caps_uses_median_geometry
- tsr_assignment_caps_floor_protects_degenerate_input
The existing dense-overlapping-rows regression test (added in #62) still
passes, confirming no regression on the cell-bleed case.
Test results: 414 lib + 115 integration + 2 doctests pass. clippy + fmt
clean.
Bump @firecrawl/pdf-inspector to 1.6.2 (patch — fixes a regression
introduced by 1.6.1; no API changes).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: TSR-aware table extraction (extract_tables_with_structure_mem)
New public function that consumes raw structure-recovery output (HTML
structure tokens + per-cell bboxes from a model like SLANet) and assembles
markdown tables by pulling cell text from the native PDF — no OCR, no
geometry inference.
Why: the existing extract_tables_in_regions_mem infers grid geometry from
text positions only and can't distinguish merged cells from multiple narrow
columns. Pairing structure recovery from a layout/TSR model with native
PDF text gets perfect text quality with proper row/col/span structure.
- New module src/tables/structured.rs: token state machine, polygon→AABB,
crop-px→page-pt, rowspan/colspan-aware cell layout, markdown emitter.
Accepts both 4-element rects and 8-element 4-corner polygons.
- New public extract_tables_with_structure_mem in src/lib.rs that reuses
extract_page_text_items, region_overlaps_item, and the shared region
text-collection helper. No existing public function modified.
- napi binding extractTablesWithStructure mirroring the existing
extractTablesInRegions shape (f64 in JS → f32 internally).
- 14 unit tests + 5 integration tests, including a real-PDF gold-standard
match against bits_pilani_feedback.pdf.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TSR follow-ups: header-aware separator, cells API, v1.6.0
- cells_to_markdown emits the separator after the LAST row that contains
is_header=true cells, falling back to "after row 0" when no header is
flagged. Multi-row theads now render correctly. Three new unit tests
cover: multi-row header, header not on row 0, no headers (fallback).
- New public extract_tables_with_structure_cells_mem returning
Vec<Vec<StructuredCell>> so callers can drive their own rendering or
debug overlays without re-doing the parse + extraction. The markdown
variant now wraps it. The previously-unused page_pt_bbox field is
surfaced through this API.
- New napi binding extractTablesWithStructureCells + StructuredCellJs.
- Bump @firecrawl/pdf-inspector to 1.6.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix TSR cell text assignment for overlapping bboxes
Made-with: Cursor
* bump npm package version to 1.6.1
Made-with: Cursor
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: TSR-aware table extraction (extract_tables_with_structure_mem)
New public function that consumes raw structure-recovery output (HTML
structure tokens + per-cell bboxes from a model like SLANet) and assembles
markdown tables by pulling cell text from the native PDF — no OCR, no
geometry inference.
Why: the existing extract_tables_in_regions_mem infers grid geometry from
text positions only and can't distinguish merged cells from multiple narrow
columns. Pairing structure recovery from a layout/TSR model with native
PDF text gets perfect text quality with proper row/col/span structure.
- New module src/tables/structured.rs: token state machine, polygon→AABB,
crop-px→page-pt, rowspan/colspan-aware cell layout, markdown emitter.
Accepts both 4-element rects and 8-element 4-corner polygons.
- New public extract_tables_with_structure_mem in src/lib.rs that reuses
extract_page_text_items, region_overlaps_item, and the shared region
text-collection helper. No existing public function modified.
- napi binding extractTablesWithStructure mirroring the existing
extractTablesInRegions shape (f64 in JS → f32 internally).
- 14 unit tests + 5 integration tests, including a real-PDF gold-standard
match against bits_pilani_feedback.pdf.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TSR follow-ups: header-aware separator, cells API, v1.6.0
- cells_to_markdown emits the separator after the LAST row that contains
is_header=true cells, falling back to "after row 0" when no header is
flagged. Multi-row theads now render correctly. Three new unit tests
cover: multi-row header, header not on row 0, no headers (fallback).
- New public extract_tables_with_structure_cells_mem returning
Vec<Vec<StructuredCell>> so callers can drive their own rendering or
debug overlays without re-doing the parse + extraction. The markdown
variant now wraps it. The previously-unused page_pt_bbox field is
surfaced through this API.
- New napi binding extractTablesWithStructureCells + StructuredCellJs.
- Bump @firecrawl/pdf-inspector to 1.6.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The rect-based cell fallback in detect_row_stripe_table_from_cell_rects
derives columns purely from text X-position clustering. When prose
wraps inside a bounding-box rect (chat transcripts, stylized figures),
the word-boundary gaps cluster into many spurious columns, producing
a multi-column "table" that is just fragmented prose.
Count cells containing common English function words (articles,
prepositions, pronouns, common verbs) and reject the fallback when
20%+ of non-empty cells contain any such word. Real tabular data —
labels, units, numbers, short identifiers — rarely contains these.
Update the td9264 snapshot: the government document section that
previously rendered as a malformed table now renders as cleaner
prose + a proper CFR list.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: don't reclassify wrapped bold list leads as headings
When a numbered/bulleted list item's bold lead phrase wraps onto a
second visual line, that line is all_bold + standalone, which scored
above the rarity heading threshold and was emitted as #### in the
middle of the item. That reset in_list, so the body continuation
below picked up a stray `- ` bullet via the struct-tree LI path,
shattering a single item into heading + stray bullets.
Guard the font heuristic: when already inside a list, skip heading
classification for lines at the list continuation indent with a Y
gap within para_threshold. Structure-tree headings still win.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: tighten rect-row span check in propagate_merged_cells
propagate_merged_cells used an overlap-based predicate with ±tol
slop that returned true at shared row boundaries — a rect whose
top exactly equals row N's bottom lies entirely below the row, yet
the predicate considered it to span row N. When multiple background
rects aligned on a shared Y edge (e.g. consecutive row-stripe
shading), each adjacent rect would over-reach by one row, cascading
labels and data from unrelated rows into a single merged cell.
Replace the overlap predicate with a containment check: rect bottom
at or below row bottom, rect top at or above row top (each within
tol). Genuine merged-cell rects fully contain the rows they span;
tangent rects do not.
Update two snapshots that were encoding the old buggy output.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a numbered/bulleted list item's bold lead phrase wraps onto a
second visual line, that line is all_bold + standalone, which scored
above the rarity heading threshold and was emitted as #### in the
middle of the item. That reset in_list, so the body continuation
below picked up a stray `- ` bullet via the struct-tree LI path,
shattering a single item into heading + stray bullets.
Guard the font heuristic: when already inside a list, skip heading
classification for lines at the list continuation indent with a Y
gap within para_threshold. Structure-tree headings still win.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On PDFs where every list item starts with ● at the left margin and content
at a fixed offset, histogram column detection sees the gap between marker
and content as a gutter and splits each line across two phantom "columns,"
scrambling the reading order (Anthropic's Mythos system card p.73–74).
- layout: reject gutter candidates where the smaller side is ≥80%
standalone bullet-marker glyphs (•, ●, ○, ◦, ▪, ▫, ◆, ◇, ■, □)
- markdown/classify: add starts_with_bullet_marker helper (narrower than
is_list_item — excludes numbered/lettered patterns like 1. and a) so
numbered section headings stay as headings)
- markdown/convert: skip heuristic heading detection on lines that start
with a bullet marker
- markdown/classify: strip a leading bullet wrapped in a bold/italic run
(e.g. "**● Label:**" → "- **Label:**") — some PDFs put the marker inside
the same bold run as the label
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Some tagged PDFs use a flat style where each wrapped visual line of a list
item gets its own MCID tagged directly under /LI. The LI branch was
unconditionally prefixing every such line with "- ", turning continuation
lines into their own bullets. Only emit a new bullet when we're not already
inside a list; otherwise fall through to the existing continuation logic so
the text is appended to the previous item.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: expose per-page markdown extraction to Python and Node (#49)
Implements the feature requested in issue #49: a list-of-pages markdown output
from the Python API. Matching the existing project pattern, the feature lives
in the Rust core and is surfaced through every binding.
- Rust core: `extract_pages_markdown` (path) and `extract_pages_markdown_mem`
(bytes) now take `Option<&[u32]>` — `None` returns every page in document
order; a slice restricts and preserves caller order.
- Python: new `extract_pages_markdown(path, pages=None)` and
`extract_pages_markdown_bytes(data, pages=None)` functions plus
`PageMarkdown` / `PagesExtractionResult` classes; stub file updated.
- Node: `extractPagesMarkdown(buffer, pages?)` — `pages` is now optional.
- Tests: 2 new Rust integration tests, 9 new Python tests, 2 new Node
assertions. All 372 unit + 107 integration + 53 Python tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version from 1.3.0 to 1.4.0
Minor bump for the new per-page markdown extraction API exposed through
the Python and Node bindings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The TOC/data-table distinction was being recomputed at every consumer:
- format.rs::table_to_markdown ran is_table_of_contents to decide between
flat-list and markdown-table rendering.
- compute_layout_complexity ran it again to filter TOCs out of
pages_with_tables.
- detect_heuristic validations used it to decide whether to relax val 1/9.
Each caller had to remember tables can be either kind, which leaks the
TOC concept across the codebase.
Add `TableKind { Data, Toc }` and a `Table::new` constructor that classifies
once from the cells. All five detectors (heuristic, rect, line, struct,
columns) now go through `Table::new`. Consumers match on `kind` instead of
re-running classification.
Pure refactor — no behavior change. Verified: pdf-evals output is byte-for-
byte identical (0 changed snapshots).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: detect hierarchical-indent TOCs as tables
Validation 1 (≥25% of rows have first col) and validation 9 (paragraph
content) were rejecting TOC pages where top-level chapters indent at the
leftmost X but subsections cascade further right. The leftmost column ends
up sparse (only chapter rows land there) and most cells are empty, but the
structure is still an unambiguous TOC.
Skip both validations when cells form a TOC pattern AND the table is narrow
(≤5 cols). The width cap preserves existing handling of wide multi-column
TOCs (e.g. back-of-book 2-up indices) where format_toc_as_list would mash
adjacent visual entries together.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: exclude TOC pages from pages_with_tables metadata
TOC detection routes through the table pipeline (detected as a table, then
formatted as a flat list with tab-aligned page numbers via
format_toc_as_list). That made TOC pages appear in LayoutComplexity's
pages_with_tables, which:
- Misleads downstream consumers that read this field as "this page has a
data table".
- Trips the table-page guard in column detection, which switches to a
different valley-detection threshold for table pages.
Add an is_table_of_contents check in compute_layout_complexity so TOC-shaped
detections don't count toward the table flag. The TOC still renders correctly
as a flat list — only the metadata classification changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Validation 1 (≥25% of rows have first col) and validation 9 (paragraph
content) were rejecting TOC pages where top-level chapters indent at the
leftmost X but subsections cascade further right. The leftmost column ends
up sparse (only chapter rows land there) and most cells are empty, but the
structure is still an unambiguous TOC.
Skip both validations when cells form a TOC pattern AND the table is narrow
(≤5 cols). The width cap preserves existing handling of wide multi-column
TOCs (e.g. back-of-book 2-up indices) where format_toc_as_list would mash
adjacent visual entries together.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The subset-GID-mismatch heuristic tripped whenever the CIDFont's W array
started at CID 0 and the ToUnicode CMap's minimum source CID was > 2.
That's the normal sparse layout for a correctly-subsetted Identity-H
font with a .notdef at CID 0 and actual glyphs at high CIDs (Cyrillic,
Arabic, etc.). The spurious remap to sequential CIDs, combined with
score_text penalizing non-Latin letters as "other", caused the garbled
CMap to win over the correct primary — spaces turned into %, digits
shifted into punctuation, and Latin letters got rewritten to Cyrillic
codepoints.
Check whether the W array actually covers the CMap's maximum source CID.
If it does, the font and CMap agree, no renumbering happened, and the
remap stays off. True mismatches (CMap CIDs outside W coverage) still
trigger the remap.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds x86_64-pc-windows-msvc target to the napi package and CI build
matrix so the published @firecrawl/pdf-inspector npm package ships
native Windows binaries alongside Linux and macOS.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix: reject dot-less tables of contents as heuristic table false-positive
The body-font heuristic table detector was firing on tagged PDF TOCs
whose entries are laid out as multi-column rows (section number, title,
page number) without dot leaders. Extend is_table_of_contents to flag
this pattern by requiring:
- 2+ columns and 4+ rows
- >=60% of rows start with a dotted section number (e.g. "4.3.1")
- >=70% of filled last-column cells are all-digit page numbers
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* format TOC tables as flat per-row list instead of dropping them
Keeping the previous approach (rejecting TOCs at detection time) meant
dot-less TOCs fell back to the column-aware text reader, which stacked
every section title first and dumped all page numbers at the end of the
region — so a reader saw a wall of titles followed by a wall of numbers.
Keep the detected Table, and in the formatter interleave the cells into
one line per row with the page number appended after a tab. The raw
(pre-clean) cells are used for the TOC check because clean_table_cells
can collapse genuine data tables into a TOC-looking shape.
Tightened is_table_of_contents to require both dot leaders AND page
numbers (previously dot-ratio alone was enough, which misfired on
register tables that use "..." as a continuation marker).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* split TOC detection into dot-leader and tabular patterns
Previous is_table_of_contents rejected every TOC-shaped table at detect
time, dropping dot-leader TOCs that render well as flat-list output and
misclassifying data tables with trailing "..." cells as TOCs.
Now is_dot_leader_toc (structural + inline-leader) keeps per-row flat
layouts for the formatter, while only wide inline-leader indices are
rejected at detect time. row_cell_is_page_number accepts dashed
section-page IDs ("A-1", "5-21") and rejects decimals/thousands
separators; cell_has_trailing_leader requires alphabetic content so
numeric data-row labels ("1973 ... ") no longer qualify.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat: add CLI bin to npm package
Installing `firecrawl-pdf-inspector` now provides a `pdf-inspector` CLI command.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump npm package version to 0.8.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump npm package version to 1.0.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use first-glyph position for ActualText ligature items
The content stream parser emitted ActualText items (from BDC/Span
marked content) at the text matrix position captured at BDC entry.
When the content stream contains Td operators between BDC and the
first glyph Tj — common in PDFs generated by Google Docs, Figma,
and web-to-PDF tools — the BDC-entry position is on the previous
visual line while the actual glyph renders on the correct line.
This caused ligature glyphs (fi, fl, ff, etc.) wrapped in
ActualText spans to be positioned one text-leading offset above
their surrounding text. Downstream merge logic couldn't reconnect
them (different y-band), producing broken words like "fi" + "ndings"
instead of "findings" throughout the document.
Fix: capture the text matrix at the first Tj/TJ operation inside
the BDC block (after any Td repositioning), and use that as the
ActualText item's rendering position at EMC time. Falls back to the
BDC-entry position when no glyph ops occur inside the block.
One new variable, five insertion points, zero behavior change on
PDFs that don't use ActualText marked content.
* chore: allow clippy::collapsible_match for Rust 1.95
Rust 1.95 introduced the collapsible_match lint which flags `if`
blocks inside match arms that could be converted to match guards.
The content-stream parsers use this pattern extensively for
readability (match on PDF operator name, then check preconditions
like `in_text_block && !op.operands.is_empty()`). Allow crate-wide
rather than refactoring 21 match arms across the parser files.
* fix(detector): correct false flags for CID-encoded text and supplementary fonts (0.7.4)
The page classifier was over-aggressively flagging Mixed-PDF pages as
needing OCR in three distinct cases. Each is fixed at the root in
analyze_page_content / page_has_identity_h_no_tounicode / the
looks_like_scan check.
1. has_vector_text false positives on dense layouts
path_ops > text_ops*200 fired on pages with decorative paths
(column borders, dividers) alongside real selectable text. Added
a unique_alphanum_chars < 30 guard: real outlined-text pages have
very few unique alphanum chars (each glyph is a path), while
pages with real text + decorations have many.
2. Identity-H without ToUnicode flagged whole pages on supplementary fonts
page_has_identity_h_no_tounicode would flag a page if any single
Type0 font lacked ToUnicode and had no fallback CMap, even when
the page's actual text came from other decodable fonts (Type1
with ToUnicode, etc.). Rewrote to track both undecodable
Identity-H fonts AND other decodable fonts, only flagging when
no decodable text font is present.
3. CID-encoded text with ToUnicode misclassified as scan
looks_like_scan checked unique_alphanum_chars < 10 on raw string
operand bytes. CID-encoded fonts (Type0 with ToUnicode) emit
2-byte CID values that aren't ASCII alphanum, so the metric is
blind to them even when the text is fully decodable. Added a
has_decodable_text_fonts signal: when a page has decodable fonts
AND >= 10 text ops, the low alphanum count is treated as a CID
encoding artifact rather than evidence of a scan.
Validated against a broad PDF corpus:
- 6 known false-positive pages now correctly classified as text
- 22 previously-missed scan pages (cover/blank/photo) now correctly
flagged for OCR
- 0 regressions on truly-scanned PDFs (61/61 pages stay flagged)
- All 437 existing tests pass; clippy clean
Bumps NAPI package to 0.7.4.
* test(detector): add unit tests for the three classifier fixes
Adds 10 unit tests covering the heuristic changes:
- has_vector_text alphanum guard
- real text + decorative paths → not flagged
- true outlined glyphs (low alphanum) → still flagged
- page_has_identity_h_no_tounicode supplementary-font handling
- undecodable Identity-H + decodable Type1 → not flagged (new)
- undecodable Identity-H alone → still flagged (regression)
- page_has_decodable_text_fonts (new helper)
- Type1 → true
- Type0 with ToUnicode → true
- undecodable Identity-H only → false
- looks_like_scan with has_decodable_text_fonts override
- CID-encoded decodable text → not flagged as scan
- same metrics with no decodable fonts → still flagged
- decodable fonts but text_ops < 10 (page-number overlay) → still flagged
* fix(detector): make decodable-font checks usage-based and XObject-aware
Addresses two reviewer concerns on the previous heuristic fix:
P1 — resource-based check could create an inverse bug
page_has_identity_h_no_tounicode and page_has_decodable_text_fonts
iterated all fonts in the page Resources dict, including unused fonts.
A page whose actual text was rendered exclusively in an undecodable
Identity-H font but whose Resources also listed an unused decodable
Type1 would be wrongly unflagged.
Fix: parse Tf operator operands during content stream scanning to
collect the set of font names actually referenced. The font checks
now filter to only USED fonts via a new used_fonts_have_*
family of functions operating on (used_font_names, font_map).
P2 — checks didn't follow text into Form XObjects
analyze_page_content correctly recurses through Form XObjects via
scan_xobjects_in_resources, but the font checks only looked at the
page's top-level Resources/Font. Pages that render text through Form
XObjects (corporate templates, header/footer overlays) had their
XObject font resources missed entirely.
Fix: scan_xobjects_in_resources now propagates the used_font_names
set AND collects fonts from each Form XObject's own Resources into
the shared font_map. The usage-based check sees the full picture:
page-level fonts + every nested XObject's fonts, intersected with
fonts actually referenced by Tf operators anywhere in the content.
Implementation:
- New extract_font_name_before_tf helper (parses /Name immediately
preceding Tf).
- New FontInfo struct caches font properties per-name.
- New collect_fonts_from_resource_dict + new used_fonts_have_*
functions are pure filters over (used_names, font_map).
- analyze_page_content threads used_font_names + font_map through
page content scan and XObject recursion, then runs the new checks.
- Old resource-based functions kept as #[cfg(test)] for the existing
unit-test interface.
- Phase 3 uncached-page loop now goes through analyze_page_content
so it also gets the usage-based + XObject-aware behavior.
Tests added (8):
- extract_font_name_before_tf basic + long-name parsing
- scan_content_for_text_operators collects used font names
- P1 — unused decodable font in Resources doesn't save a page
whose used font is undecodable
- P1 — both fonts used → decodable font correctly prevents flag
- P2 — decodable font inside Form XObject correctly unflags
- P2 — undecodable font only in XObject still flags even with
unused decodable font at page level
- P2 — has_decodable_text_fonts populated from XObject fonts
Validation:
- 349 lib + 104 integration + 2 doc tests pass (was 341)
- cargo clippy --lib --bin detect-pdf -- -D warnings: clean
- External eval: 9/9 PDFs pass, 6/6 false positives resolved,
0 regressions, 61/61 scanned pages still correctly flagged
- No eval delta — confirms previous fix wasn't relying on the
resource-based bug for any of the eval PDFs
* fix(detector): scope font lookups by ObjectId + handle indirect Form Resources
Addresses two more reviewer findings on the previous decodable-font commit.
P1 — Resource-name scoping bug
The previous fix keyed used_font_names and font_map by raw resource
names like b"F1". PDF resource names are scoped to each resource
dictionary: a Form XObject can legally define its own /F1 that points
to a completely different font from the page's /F1. Because
collect_fonts_from_resource_dict skipped duplicates with
`if font_map.contains_key(name)`, the first definition won and later
Tf /F1 usages in different scopes resolved against the wrong font.
This could reintroduce both the undecodable-Identity-H false flag
and the decodable-CID false unflag depending on which side of the
collision happened to be inserted first.
Fix: switch the lookup mechanism from font names to font ObjectIds.
- font_map: HashMap<ObjectId, FontInfo> (was Vec<u8> keys)
- used_font_ids: HashSet<ObjectId> (was Vec<u8> names)
- new resolve_font_names_to_ids() runs immediately after each
content scan, against the resource dict in scope, to translate
the per-scope name set into ObjectIds.
Each Form XObject's content stream now resolves /F1 against THAT
XObject's own Resources, so name collisions are impossible by design.
Inline (no-ID) font dicts are skipped — extremely rare in practice
and have no stable key.
P2 — Indirect Form /Resources skipped
scan_xobjects_in_resources used `.as_dict()` on the Form's /Resources
entry, which returns None for indirect references. PDFs frequently
store /Resources as `X 0 R`, in which case font collection and
recursion were both skipped — even though the Tf usages inside the
XObject content had already been recorded.
Fix: handle Object::Reference(r) in addition to Object::Dictionary(d)
by resolving via doc.get_dictionary. Audited the rest of the file —
the other /Resources access points (analyze_page_images,
collect_images_from_resources) already handled both cases.
Tests added (4):
- P1 same-name-different-font (page undecodable, XObject decodable):
must NOT flag — XObject's text is decodable in its own scope.
- P1 inverse (page decodable, XObject undecodable, content uses
XObject /F1): MUST flag — undecodable text exists in real scope.
- P2 indirect Form /Resources: font discovery must still work when
/Resources is a `X 0 R` reference rather than inline.
- Combined regression: indirect Resources + name collision.
Validation:
- cargo test --release: 459 tests pass (353 lib + 104 integration + 2 doc)
- cargo clippy --lib --bin detect-pdf -- -D warnings: clean
- external eval (9 PDFs): 9/9 pass, 6/6 false positives resolved,
0 regressions, 61/61 truly-scanned pages still flagged
The behavior on the eval set is identical — confirms the correctness
fix isn't masking any change in classifier outcomes.
* fix(detector): respect resource shadowing when resolving page-content fonts
The previous ObjectId-based fix correctly scoped Form XObject fonts
but still violated PDF resource inheritance for page content. When a
page overrides /F1 from a parent /Pages node (different font dict for
the same name), get_page_resources returns the page's own /Resources
plus all ancestor /Resources dicts. The old code called
resolve_font_names_to_ids on each one and added every match to
used_font_ids — both font ObjectIds ended up in the used set even
though only the page's /F1 is actually visible to that page's content.
Per ISO 32000-1 §7.7.3.4, resource names are inherited with
shadowing semantics: the most-specific (deepest, closest to the page)
definition wins.
Fix:
- New lookup_font_id helper resolves a single name in a single dict.
- New resolve_with_shadowing iterates names, checking the page's own
/Resources first, then walking ancestors in most-specific-first
order (which is the order lopdf's get_page_resources returns).
First hit wins via a labeled `continue 'name` — subsequent
ancestors are skipped for that name.
- analyze_page_content's flat resolution loop replaced with one call
to resolve_with_shadowing.
Audit:
- XObject path is correct: each Form XObject already resolves names
against its OWN /Resources (XObjects don't inherit from page tree).
- font_map population is correct: keyed by ObjectId, so collecting
from all dicts builds the full available-fonts catalog. The bug
was only in the used-set resolution.
- Confirmed lopdf returns ancestors in most-specific-first order
(page → parent → grandparent → root), matching the shadowing
direction used here.
Tests added (3):
- page /F1 undecodable shadows parent's decodable /F1 → MUST flag
- page /F1 decodable shadows parent's undecodable /F1 → MUST NOT flag
- no override: page inherits parent's decodable /F1 → MUST NOT flag
Validation:
- cargo test --release: 462 tests pass (356 lib + 104 integration + 2 doc)
- cargo clippy --lib --bin detect-pdf -- -D warnings: clean
- external eval: 9/9, 0 regressions, 6/6 false positives resolved,
61/61 scanned pages still correctly flagged
The looks_like_scan check incorrectly used OR logic, causing any single
condition (image_count <= 1, text_ops < 50, alphanum < 10) to flag a page
as a scan. A real scan has ALL three: single full-page image AND low text
AND low alphanum. Text pages with one figure were falsely flagged for OCR.
Bump napi to 0.7.3.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: reduce false OCR recommendations for text PDFs with figure images
Two fixes in the detector:
1. Fix Tf operator parsing: some PDFs concatenate Tf directly with the
next operator (e.g. "25 Tf[<01>...") without whitespace. The scanner
now accepts [, (, <, / as valid followers, fixing font_changes being
reported as 0.
2. Distinguish text-with-figures from scanned-with-OCR: pages with
multiple images (image_count > 1) and strong text signals (text_ops
>= 50, alphanum >= 10) are recognized as text pages with figures,
not scanned templates. Scanned PDFs have exactly 1 full-page image.
This prevents academic papers, reports with charts, and similar PDFs
from being incorrectly classified as Mixed/OCR-needed when their text
is perfectly extractable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump napi version to 0.7.2
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove template image influence from page classification
Template images (large background/figure images) no longer affect
pages_needing_ocr. In the region-based pipeline, text regions are
extracted independently from image regions, and per-region needs_ocr
quality checks handle scanned-with-OCR garbage text.
Also makes the invisible text retry (for OCR text layers) trigger on
text quality rather than PDF type, so it works regardless of
classification.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Revert "fix: remove template image influence from page classification"
This reverts commit 100cbe5453.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: improve heuristic table detection for numeric columns and multi-line headers
Two fixes for tables that have clean extractable text but fail heuristic
structure detection:
1. Numeric column merge pass (grid.rs): After initial X-position
clustering, adjacent clusters are merged when one is sparse (header
text) and the other is dense with >50% numeric items (data column).
Multi-line wrapped headers often land slightly offset from their
data column — the merge closes gaps within 1.5× the clustering
threshold. New is_numeric_text() helper matches decimals, percentages,
negative numbers, and comma-separated thousands.
2. Duplicate-header skip (detect_heuristic.rs): Spanning super-headers
like "First Degree | First Degree | Higher Degree" contain duplicate
cells that trigger looks_like_partial_table_ex rejection. Now skips
rows with duplicate cells when a better header candidate exists
within the next 3 rows (higher fill ratio or numeric cells).
Tested on BITS Pilani university report (430 pages, 314 table pages).
Page 4 (multi-line header + numeric data) previously returned
needs_ocr=true; now correctly detects the table structure.
Eval: 197 PDFs, zero regressions, all 104+ tests pass, zero clippy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* bump version to 0.7.1
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace tag-based trigger with push-to-main trigger that detects
version changes in napi/package.json, removing the need for manual
git tags to publish new npm releases.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Combine per-page markdown extraction with layout classification into a
single parse. extractPagesMarkdown now returns PagesExtractionResult with
pages_with_tables, pages_with_columns, pages_needing_ocr, and is_complex
alongside the per-page markdown — eliminating redundant PDF parses for
callers that need both.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* add extract_pages_markdown_mem for per-page markdown extraction
Enables hybrid OCR pipelines to skip GPU render+layout for simple text
pages by providing per-page markdown with needs_ocr flags. Font stats
are computed document-wide for consistent header detection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* bump napi package version to 0.6.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace stringly-typed pdf_type and item_type fields with
#[napi(string_enum)] enums for proper TypeScript type checking.
Add link_url field to TextItem instead of encoding URL in the
item_type string.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two changes that reduce false needsOcr rejections without hurting quality:
1. Per-region GID check instead of per-page blanket rejection.
Previously, if ANY font on the page used GID-encoded glyphs (common
in logos, decorative fonts), ALL table and text regions on that page
were forced to GPU OCR via needsOcr=true. Now the page-level bail is
removed; per-region text quality checks (is_garbage_text, is_cid_garbage,
detect_encoding_issues) catch actual GID corruption in the extracted
content. Tables whose text is clean pass through even if an unrelated
font elsewhere on the page is GID-encoded.
2. Relaxed looks_like_partial_table for layout-assisted extraction.
When the layout model already identified a region as a table (i.e.,
extract_tables_in_regions_mem), boundary-detection heuristics are
less necessary — we're not guessing "is this a table?" anymore, only
"can we extract it correctly?". Relaxations:
- Numeric first header cell accepted (e.g., year "2024")
- 1 empty header cell allowed in 3+ column tables (merged headers)
- Sparse first data row threshold relaxed from 33% to 50%
Paragraph detection and duplicate-header checks remain strict.
Eval: 196/196 pass (full regression suite), 91/91 Rust tests pass
including 7 new layout-assisted validation tests. Zero regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a 5th failure-mode check to looks_like_partial_table: when the
heuristic mis-detects text-wrapped paragraph prose as a multi-column
table, cells in the same column tend to start with lowercase letters
or continuation punctuation (commas, closing quotes) — because they're
actually sentence fragments. Real tables almost never have most data
cells starting lowercase.
Trigger: ≥2 cols, ≥4 data rows, ≥60% of non-empty data cells start
with lowercase or continuation punctuation → return needs_ocr=true.
Caught in the eval as the next-largest failure mode after the 0.4.1 fix:
PDFs 088, 182, 090 — heuristic produced "tables" like:
|Approval is needed from the|Acquisitions of|
|Treasurer if the acquisition|residential and|
|constitutes a "significant|agricultural|
|action," including acquiring an|land by foreign|
Reading column 1 top-to-bottom: "Approval is needed from the Treasurer
if the acquisition constitutes a 'significant action,' including
acquiring an interest..." — a paragraph, not tabular data.
Tests: 2 new tests (the 088-style failure case + a real multi-word
table that must NOT be flagged). All 11 looks_like_partial_table tests
pass; 323 unit + 91 integration tests still green.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the heuristic returns markdown that looks like a partial / mis-detected
table, set needs_ocr=true so the caller falls back to GPU OCR. Previously the
same cases returned the broken table with needs_ocr=false, which produced
real-world TEDS=0 scores in fire-pdf evals (heuristic-built table didn't
match ground truth structure at all, but caller had no signal to fall back).
Four failure modes detected, all observed in opendataloader-bench eval losses:
1. **Header looks like a data row** — first cell of header is a bare number
(e.g. `|2|...`), suggesting the actual header row was skipped. Real
headers almost never start with just a number.
2. **Empty header cells in a multi-column table** — ≥3 cols, ≥1 empty cell
in the header row. Indicates poor column boundary detection.
3. **Duplicate header cells** — same non-empty value appearing twice in the
header (e.g. "Administration|Administration"). Means a multi-line header
was collapsed wrong.
4. **Sparse first data row** — ≥3 cols and ≥1/3 of first-data-row cells are
empty. Multi-row headers in the source PDF get smashed into header +
sparse data row by the heuristic; this catches that.
Tests: 9 new unit tests in `looks_like_partial_table_tests` cover each
failure mode plus realistic non-failures (well-formed table, single-column
list, two-col with a single empty cell). All 91 existing tests still pass.
Bumps `napi/package.json` to 0.4.1 since this changes the function's return
behaviour for callers (some inputs that returned needs_ocr=false now return
true). The output text field is also cleared on the new fallback path so
callers don't accidentally use the broken markdown.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a new function that takes a PDF buffer and page+bbox regions (same interface
as extractTextInRegions), runs heuristic table detection on items within each region,
and returns markdown pipe-tables. Falls back to needs_ocr=true when no table
structure is found or text quality is suspect.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Some PDFs (e.g. British Academy grant guidance, Carter BloodCare privacy
policy) have structure trees that incorrectly tag body text as H2 headings.
This caused every line within numbered paragraphs to render as a separate
## heading instead of being joined into flowing paragraph text.
Added detect_overused_struct_heading_levels() which pre-scans heading tag
frequency and suppresses levels appearing on >15% of tagged lines, allowing
those lines to fall through to normal paragraph joining.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The detector flagged pages for OCR whenever any font was Identity-H
without ToUnicode, even when the extraction pipeline could decode the
font via fallback paths (CID-as-Unicode passthrough or embedded TrueType
cmap). This caused false positives on PDFs from Chromium, wkhtmltopdf,
and other generators that use Identity-H with Unicode CID values.
Now checks DescendantFonts W array and embedded font cmap before
flagging. Fonts that are genuinely undecodable (stripped cmap, low GID
CIDs) are still correctly flagged.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Do invokes any XObject (Form or Image), but scan_content_for_text_operators
was counting every Do as an image. PDFs with Form XObjects (e.g. ACS
publisher watermark pages) were misclassified as ImageBased because the
inflated image_count raised the min text ops threshold above the actual
text operator count.
Image detection is already correctly handled by scan_xobjects_in_resources
(checks Subtype) and analyze_page_images (measures pixel area).
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
In multi-column PDFs, column switches break paragraph continuity,
making body text lines appear "standalone". Combined with moderate
font-size rarity from minor size variation between columns, this
caused hundreds of false heading classifications (e.g. 281 false ##
headings on a single academic paper).
Non-bold, non-isolated lines now require very high rarity (≥0.97)
and short word count (≤8) to qualify as headings via the rarity path.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When fire-pdf sends pre-segmented bboxes from the layout model,
pdf-inspector no longer runs column detection, stream-order heuristics,
or newspaper/tabular mode detection within the region. These heuristics
conflict with the layout model's decisions and cause wrong reading order.
Region extraction now simply: Y-sorts items, groups into lines, and
sorts within each line by X position. The heavy heuristics remain
available for standalone full-page extraction.
Eval showed pure OCR (0.2875 NED) beating native+heuristics (0.2916)
across all categories, especially multi-column (-0.08) and newspaper
(-0.16). This change should close that gap.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pre-scan lines to identify "isolated" ones — short lines (1-6 words)
with paragraph breaks both before AND after. These are heading
candidates even at body font size, common in academic papers
("Acknowledgements", "Limitations", "B.3 Prompt Engineering").
Inspired by opendataloader's HeadingProcessor which passes prevNode
and nextNode context to the heading probability scorer.
The isolated signal (+0.3) combines with rarity/bold/standalone
signals. A per-page density guard prevents false positives on
multi-column pages where many lines appear isolated. Continuation
word detection (ending in "the", "and", etc.) filters wrapped
paragraph lines.
MHS=0 docs: 18→13. MHS-S +0.004. No regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the histogram-based column detector finds no valleys (common with
sidebar/asymmetric layouts), fall back to a simplified XY-cut: find the
largest horizontal gap between item edges and split there if both sides
have enough items with vertical overlap.
Inspired by opendataloader's XY-Cut++ algorithm but implemented as a
single-level fallback rather than full recursive segmentation.
Doc 156: NID 0.545→0.966, Doc 157: NID 0.564→0.962.
NID-S +0.007, TEDS-S +0.066 across 200 docs. No regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PDFs that draw table borders as thin filled rectangles (< 2pt, common in
spreadsheet exports) were invisible to both rect-based and line-based
table detectors. Now converts these thin rects to PdfLine objects and
runs line-based detection, but ONLY as a last resort after all other
methods (rect, line, heuristic, column-based) found nothing.
This avoids the regression from the earlier attempt which ran synthesis
at step 2, preempting the heuristic detector on PDFs where it worked
better.
Also relaxes uniform row spacing threshold (CV 0.05→0.02) to accept
spreadsheet-exported tables with even row heights.
Benchmark: TEDS 0.519→0.586 (+0.067), overall +0.006, no regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reverts commits 937311c, 1dcb0c6, 0300e96, 999f9a2. The thin-rect-to-line
synthesis and stacked table splitting improved extraction for specific
government PDFs but caused -0.05 TEDS regression on the benchmark by
preempting the heuristic detector with worse line-based grids.
These features need more targeted guards before re-enabling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rows that sit between horizontal rules but lack vertical border coverage
are not table cells — they're freestanding text (e.g. "Note: The cutoff
mark is out of 120"). These rows now split the grid into separate
sub-tables, with the unbounded text emitted as plain text between them.
Single-cell "tables" (from the split) render as plain text instead of
a degenerate 1x1 markdown table.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rows with 3+ short-valued cells (avg ≤10 chars) and an empty first cell
are column headers (e.g. "UR | SC | ST | OBC | EWS"), not text overflow
from the previous row. Prevents them from being merged into the
preceding section title row.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three fixes for better table extraction from spreadsheet-exported PDFs:
1. Convert thin filled rects (< 2pt) to PdfLine objects before line-based
table detection. Many PDFs draw table borders as narrow filled rectangles
instead of stroked paths — these were invisible to the line detector.
2. Relax uniform row spacing rejection (CV 0.05 → 0.02). Spreadsheet
exports have very even row heights that were being rejected as "chart
grids".
3. Fix continuation row merging: don't merge rows where the only non-first
cell content is a long label (section headers like "Category No. 03").
Don't merge first-cell-only rows with long text ("Note: ...").
Also adds multi-Y row splitting in line-based detection and column-aware
table detection skipping for multi-column pages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Many PDFs (especially spreadsheet exports) draw table borders as thin
filled rectangles (height/width < 2pt) instead of stroked paths. These
were invisible to our line-based table detector since only stroke
operations produced PdfLine objects.
Now synthesizes PdfLine from thin rects before line-based detection,
enabling table detection on border-drawn PDFs like government forms.
Also relaxes the uniform row spacing rejection threshold (CV 0.05→0.02)
to accept spreadsheet-exported tables with even row heights.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On pages where column detection finds 2+ columns, skip body-font
heuristic table detection in the merged-band retry path. This prevents
sidebar/two-column prose from being formatted as markdown tables.
The fix is targeted: per-band heuristic detection still runs (bands
are scoped to single columns), so real tables within columns are
still detected. Only the merged-band retry (which sees all items
across columns) is gated.
Also relaxes column validation to accept asymmetric layouts (sidebars)
where one side has fewer items, and tries center-based item assignment
before edge-based to improve column splitting for asymmetric layouts.
Benchmark: NID 0.865→0.869, NID-S 0.798→0.805, overall +0.002.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace ad-hoc bold/ratio heading checks with a unified scoring system
based on font size rarity. For each line, compute:
score = font_rarity * 0.5 + bold * 0.3 + standalone * 0.2
Font rarity measures how infrequently a font size appears across the
document — heading fonts are rare while body text is common. This
approach (from opendataloader's ModeWeightStatistics) naturally adapts
to each document's font distribution instead of relying on fixed
thresholds.
Guards: require font_size >= 0.95 * base_size (no small-font headings),
word_count >= 3, and standalone (paragraph break before).
Benchmark improvement: MHS 0.56→0.58, MHS-S 0.66→0.70, overall +0.003.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compare pdf-inspector against other direct text extraction engines
(no OCR/ML) on the opendataloader-bench corpus.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lines with font size 1.10-1.20x body text that are standalone and short
(1-8 words) are promoted to headings. This catches academic paper
headings where the font is only ~10% larger than body text, below the
previous 1.2x threshold.
Also syncs the simpler to_markdown_from_lines path to match the
table-aware path (removes stale colon exclusion).
Benchmark improvement: MHS 0.54→0.56, overall 0.761→0.766.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Caption detection was incorrectly classifying "Table of Contents" as a
caption because it starts with "Table ". Now "Table" and "Figure"
prefixes require a digit, parenthesis, or hash after them — matching
actual captions like "Table 1", "Figure 3.2" but not titles.
Also removes debug logging left from previous iteration.
Benchmark improvement: MHS 0.52→0.54, overall 0.757→0.761.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The colon exclusion was preventing legitimate headings like "Steps for
Using the Microscope:" and "Changing objectives:" from being detected.
The single edge case it was protecting (chart sub-headers) is less
impactful than the many headings it was blocking.
Benchmark improvement: MHS 0.51→0.52, MHS-S 0.61→0.62.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Lower minimum item count for body-font table candidates from 9 to 6,
allowing small 2-3 row tables to be detected.
- Allow 2-column body-font tables with short cells (avg ≤25 chars) to
bypass the "table-like content" validation. This catches text-only
definition/category tables (e.g., species lists) without false-positiving
on 2-column paragraph text (which has longer cells).
Benchmark improvement: TEDS 0.498→0.519, overall 0.750→0.754.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bold lines at body font size that are standalone (preceded by a paragraph
break) and have ≥3 words are promoted to headings. This catches the
common pattern in academic/technical PDFs where section headings use
bold text at the same size as body text.
Guards against false positives: minimum word count, colon-ending
exclusion (labels like "Table I:").
Benchmark improvement: MHS 0.37→0.50, overall 0.71→0.75.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align region filtering with rotated-page coordinate rewrites, switch region text assembly to the shared line-grouping pipeline, and retain edge-overlap text to avoid false empty regions that incorrectly trigger OCR fallback. Also make Python region inputs fail fast with clear ValueError messages for malformed boxes.
Made-with: Cursor
Two bugs in collect_text_in_region / extract_text_in_regions_mem:
1. The threshold-based sort comparator in collect_text_in_region was not
transitive, causing Rust's sort to panic on certain PDFs. Replaced with
strict total_cmp ordering — the line-grouping phase already handles
fuzzy Y matching via threshold.
2. The needs_ocr check was missing is_cid_garbage, so Identity-H fonts
with CID garbage (C1 control chars, high Latin mojibake) could pass
all quality checks and be served as real text with needs_ocr=false.
Also adds 7 integration tests for extract_text_in_regions_mem (previously
had zero coverage): basic extraction, Identity-H needs_ocr, multiple
regions, nonexistent page, empty region, invalid input, and a fast-vs-normal
comparison test across all text-based fixtures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
FontCMaps::from_doc can spend 4.5+ seconds decompressing and parsing
large embedded TrueType fonts for CID fonts with sparse ToUnicode
CMaps. For extract_text_in_regions (hybrid OCR pipeline), this is
unnecessary — fonts that can't be decoded cheaply will produce
empty/garbage text, triggering needs_ocr=true and GPU OCR fallback.
Changes:
- Add FontCMaps::from_doc_pages_fast() that skips TrueType font
fallback parsing (build_fallback_cmap_for_type0) and Identity-H/V
second pass entirely
- Add FontCMaps::from_doc_pages() for filtered page sets
- extract_text_in_regions_mem uses fast mode
- Restructure fallback chain: try cheap fallbacks first, only attempt
expensive TrueType parsing when needed and not in fast mode
Benchmark on nihms-1771367.pdf (19-page chemistry paper):
- FontCMaps fast: 201µs
- FontCMaps slow: 4.47s
- 22,000x speedup on font parsing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rust panics in NAPI modules abort the Node.js process with no chance
to report errors. This wraps every exported function in catch_unwind,
converting panics into JS Error exceptions that can be caught and
reported to Sentry.
Buffer data is extracted to Vec<u8> before the catch_unwind boundary
to satisfy UnwindSafe requirements.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
upload-artifact strips the common napi/ prefix, so files are at
artifacts/js-bindings/index.js not artifacts/js-bindings/napi/index.js.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
napi pre-publish expects a version-bump commit message convention.
Instead, upload index.js and index.d.ts generated by napi build
as artifacts and copy them into the publish step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The command is `napi pre-publish` (hyphenated), not `napi prepublish`,
and `--skip-gh-release` is not a valid flag.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
index.js and index.d.ts are generated by napi-rs. Generate them
at publish time via `napi prepublish` instead of checking them in.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix sort panics on NaN values from bogus PDF font metrics
Replace all `partial_cmp(...).unwrap_or(Ordering::Equal)` and bare
`partial_cmp(...).unwrap()` with `total_cmp()` across the codebase.
`partial_cmp` returns `None` for NaN, and mapping that to `Equal`
violates total ordering: `a == NaN` and `NaN == b` but `a != b`.
Rust 1.81+ detects this and panics in sort_by. `total_cmp` handles
NaN deterministically (sorts to end) and guarantees total ordering.
The critical crash was in `extract_text_in_regions` (lib.rs:478)
where PDFs with bogus font ascent/descent values produced NaN in
text item coordinates, causing process abort via NAPI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix missed partial_cmp in layout.rs and restore napi exports
- Convert two remaining b.y.partial_cmp(&a.y) calls to total_cmp
in group_single_column and column layout sorting
- Restore missing napi exports: detectPdf, extractText,
extractTextWithPositions, processPdf
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move detailed Python, Rust, and debugging docs into docs/ to keep
the main README focused on overview and quick start examples.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
