Three rejection sites in detect_rects.rs killed any candidate grid
where a single cell exceeded 500 chars:
- detect_row_stripe_table (line 1399)
- detect_row_stripe_table_from_cell_rects (line 1729)
- detect_merged_cluster_table (line 2154)
The intent was to skip layout-background rects — sidebars, banners,
section bands — where one big rectangle wraps a paragraph of prose.
Those almost always present as ≤3 row stripes (header / body / footer
or single big block).
Multi-row key/value tables with paragraph-length values in one column
present the same cell-length signal but are legitimate tables.
Gating the rejection on `non_empty_rows < 4` preserves the
layout-background guard for narrow stripe layouts while letting
through multi-row tables with descriptive content.
Verified against a 9-row × 2-column key/value layout where the value
column has multi-line content (~1.4KB in the longest cell). Before:
rejected with `max cell length 1384 > 500`. After: accepted with 89%
density. The existing `test_row_stripe_rejects_layout_background_long_cells`
regression test for narrow stripe layouts still passes.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The page-spanning-frame guard rejected any line set whose bounding box
exceeded ~90% of a standard A4/Letter page in both axes. The intent was
to skip decorative outer borders, but it also threw away every real
full-page table — common in governmental ledgers, financial filings,
and dense report layouts.
Decorative borders have just 4 edges (top/bottom/left/right). Real
full-page tables have many internal row and column rules. Gate the
rejection on `horizontals.len() <= 4 && verticals.len() <= 4` so the
guard still catches bare frames but lets through line sets with real
internal grid structure.
Tested against a regione.lazio.it Estrazione-provvedimenti page (full
A4-width table, 14 rows × 6 cols): now ACCEPTED with 211/211 items
captured. Bare-frame regression test added.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a wire-bordered table has headers centered/right-aligned in their
cells but data left-aligned, cluster_x_positions can both merge adjacent
data columns (when the data-to-data gap is below the clamped threshold)
and drop the header-only x-positions in its singleton-filter pass. The
cell-rect fallback then used text-cluster column edges and lost a column
or fragmented neighbor cells.
Prefer rect-derived column edges when the rect grid has 3+ columns and
every rect column holds multiple text items. The all-cols-populated
check protects against decorative or background rects (prose laid out
in a frame, cell-fill rects with extra borders) that would otherwise
split a logical column into spurious sub-columns. The existing
prose-in-frame, well-distributed-columns, and wireless-prose guards
still fire for the cases they were built for.
Bump napi version 1.8.7 → 1.8.8.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: lift detection on shaded-header + alt-row tables (#wired-grids)
Production telemetry on `wired_high_confidence`-classified table regions
showed `detect_vector_grid_in_region_mem` returning a usable grid only
~27% of the time, with the rest falling through to GLM-OCR. Three
surgical fixes target the dominant production shapes:
* Path-fill cell backgrounds: when the page has no `re` rects but draws
cell backgrounds via `m`/`l`/`h`/`f*` sequences, prefer the fill-derived
rects over the few section-level `W*` clip paths that previously won
the priority gate. Activated when fill rects outnumber clip rects ≥3×.
* Dedup-induced cluster splits: page-background rects could pose as
containers in the sub-rect dedup and evict a slightly smaller
table-frame rect, breaking adjacency between column-cell groups so each
column became its own cluster. Origin-anchored containers are now
disqualified from sub-rect dedup. A separate exact-duplicate pass
collapses the cell-padding/text-bg/cell-border triple emissions some
PDFs produce, preserving original order to avoid reshuffling table
output on multi-table pages.
* Prose-words rejection: the `cell-rect` fallback's whole-grid prose
threshold also rejected real tables that include a description column.
Now relaxed when content is well-distributed (≥75% of cols filled),
while keeping the original strictness for prose-in-a-frame layouts.
Two regression fixtures from the opendataloader-bench corpus, covering
the dominant production failure categories:
* `greencomp_competence.pdf` — 2-col shaded-header + plain-body glossary.
Mirrors production crops #1 (Contractions glossary) and #6 (BIO 350
course header).
* `upstage_key_functions.pdf` — 4-col shaded-header + alt-row backgrounds
+ merged left column. Mirrors production crops #2 (Parameter/Value
alt-row), #7 (Spanish XML schema), and #8 (Córdoba multi-row header).
Existing fixtures stay green (doc 51 wrapped-label, doc 128 forecast
six-cols, td9264 snapshot). 133 unit + integration tests pass; clippy
clean.
Bumps napi/package.json 1.8.4 → 1.8.5.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: tighten prose-in-frame rejection — fixes pdf-evals #30 regression
PR #76's shaded-header detection lift surfaced a regression on
accessory_building_permit_application_1 (TEDS 0.10 → 0.05): a
paragraph of legal text laid out in a 2-column justified block was
being admitted as a 10×2 fake table where every cell holds a
sentence fragment ("I agree to comply...", "I", "It is the property
owner's responsibility..."). Per pdf-evals PR #30 review, this is
the kind of regression production users will notice — the markdown
is structurally and semantically misleading.
Root cause: PR #76's prose-rejection only fires for `num_cols >= 4`,
so the 2-col prose-in-a-frame case slipped past it entirely. The new
fill-priority + dedup changes started producing rects for this layout
that 1.8.4 correctly ignored.
Fix: tighten the prose-in-frame check.
- Lower the column-count guard from `>= 4` to `>= 2`.
- Add a content-length signal as the primary discriminator: when the
prose-words trigger fires AND mean non-empty cell length exceeds
65 chars, reject regardless of column distribution.
The 65-char threshold cleanly separates observed cases:
accessory_building (prose-in-frame): mean 74 chars → REJECT
upstage_key_functions (real 4-col table): mean 53 → admit
greencomp_competence (real 2-col glossary): mean 20 → admit
accessory_building (real 5×3 form data): mean 10 → admit
The well-distributed-cols relaxation that PR #76 added stays —
"label / value / description / benefit" tables (#7, #8 from the
production crops) still pass, but only when their mean cell length
stays below the prose threshold.
New regression test `accessory_building_rejects_prose_in_frame` asserts
both that the real 5×3 form data table survives AND the 10×2 prose
block is rejected. Snapshot test `test_snapshot_td9264` updated to
match new output — old snapshot captured the same prose-in-frame bug
on regulatory text (paragraphs emitted as 3-col `||text||` fake-table
rows). New snapshot emits clean prose paragraphs, which is correct.
Verification:
- cargo test --all: 424 lib + 133 integration + 2 doc tests pass
- cargo fmt --check clean
- cargo clippy -- -D warnings clean (lib-level; pre-existing
test-level clippy issues on the wired-grids branch unaffected)
- Existing fixtures stay green: forecast_table_chart_six_cols (PR
#72), bits_pilani_* (PR #73), greencomp_competence_two_cols and
upstage_key_functions_four_cols (PR #76).
This branch is based on abimaelmartell/wired-grids so it includes
PR #76's commits plus this fix on top. Suggest merging this and
closing #76, OR rebasing #76 to incorporate this fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: drop early-dedup atom — caused broad TOC + matrix corruption
Bisected PR #76's 4 atoms against the SEC 10-K
0001104659-25-093871_183e44ac.pdf which appeared as a TEDS regression
in pdf-evals PR #30. Result:
atom | TOC | perf-graph | qualifications
---------------------------|-----|------------|---------------
fill-priority | ✓ | ✓ | ✓
early-dedup | ✗ | ✗ | ✗
page-bg disqualification | ✓ | ✓ | ✓
prose-relaxation | ✓ | ✓ | ✓
Early-dedup was the SOLE source of all three regressions on this doc.
Tried a more conservative variant (≥3 copies only — pair-duplicates
appear in legit multi-section layouts like 10-K dividers above + below
section headers); didn't fix the regression. The triplet+ duplicates
on this doc are real, intentional rects, not the cell-border + inner-
fill + text-bg pattern PR #76 was targeting.
Drop early-dedup. Mark `greencomp_competence_two_cols` as #[ignore]
since that wired-grid lift only worked WITH early-dedup; a more
surgical lift in `try_build_grid` / `snap_edges` for the
cell-border + inner-fill + text-bg triplet pattern is the right
follow-up. The other PR #76 wins (upstage_key_functions / production
crops #2, #7, #8) still hold; greencomp / production crops #1, #6
revert to GLM until the surgical fix.
Validation on the regression doc:
0001104659 TOC PART II markers: 4 (matches main, was 2 with PR#76)
0001104659 perf-graph data row: 2 (matches main, was 1)
0001104659 qualifications rows: 9 (matches main, was 6)
Validation on the prose-frame doc:
accessory_building fake-table: 0 (matches main, was 1 with PR#76)
accessory_building prose intact: 1 (matches main)
cargo test --all clean, cargo fmt --check clean, cargo clippy --lib
-- -D warnings clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* extractor: drop Latin-1 mojibake on Type0/CID fonts; tokenize wide TSR items
Two text-extraction failure modes surfaced by table-candidate shadow
data; both also affect the existing TableFormer / vector-grid paths
since they share `extract_tables_with_structure_*_mem`'s downstream
cell-fill.
1. CJK / multi-byte mojibake. The bottom Latin-1 fallback in
`extract_text_from_operand` ran unconditionally. For a Type0/CID
(Identity-H) font whose ToUnicode CMap fails to parse, the bytes
are CIDs (font-internal indices), not character codes — per-byte
Latin-1 produces mojibake (e.g. 2-byte CID 0xCDD9 surfaces as "ÍÙ").
Gate that fallback on `FontWidthInfo.is_cid` (set by
`parse_type0_widths` for `/Subtype /Type0`). For Type0 fonts with
any non-ASCII byte, emit one U+FFFD per CID instead so
`detect_encoding_issues` still trips and the page is flagged for
OCR — preserving the existing OCR-routing path that the
high-Latin-1 garbage used to satisfy by accident. Type1 / TrueType
simple fonts retain the per-byte Latin-1 round-trip (it IS the
canonical interpretation for them; verified against an existing
pdf-evals fixture where bytes like 0xB6 are legitimate Latin-1).
Threaded `font_widths: &PageFontWidths` through
`extract_text_from_operand` and its 5 call sites in
`content_stream.rs` / `xobjects.rs`.
2. Dense-cell text collapse in `extract_tables_with_structure_cells_mem`.
Stage-1 routing did per-item assignment — each TextItem went into the
single cell whose bbox contained its center. When a row's text is
rendered as one wide Tj (e.g. "Marshall Islands 0.9 0.9 0.9"), the
whole row parks in one cell and the rest of the row stays empty.
New `split_item_into_token_subitems` helper splits each item into
per-token virtual sub-items with x positions estimated from
`effective_width / char_count` and the token's character offset.
Stage 1 then routes per-token. Single-token items collapse to a
one-element vector (no behavior change). Multi-token items spanning
multiple cells distribute correctly. Stage-2 orphan recovery now
operates on token-grain orphans rather than re-trying whole items.
Tests:
- `cid_font_with_unparseable_cmap_does_not_emit_latin1_mojibake` (unit)
exercises the Type0/CID + unparseable-CMap fallback path.
- `simple_font_latin1_fallback_passes_high_bytes_through` (unit)
guards the false-positive case where a Type1 font's `/ToUnicode`
reference is set but bytes are legitimate Latin-1 character codes.
- `test_extract_tables_with_structure_distributes_wide_item_across_cells`
(integration) builds a synthetic PDF with one wide Tj and asserts
each token lands in its own cell.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tests: pin CID mojibake fix mechanism with FFFD assertions
Two complementary tests for the Type0/CID Latin-1-fallback guard:
1. Tighten `test_identity_h_no_tounicode_suppresses_garbage` on the
existing real-PDF fixture `shinagawa_identity_h.pdf` to also assert
the pre-suppression text contains U+FFFD and contains no high-Latin-1
chars. Pins down WHICH mechanism is suppressing the garbage so a
future regression that re-enables Latin-1 mojibake fails loudly here
instead of silently switching the suppression chain back to
`is_cid_garbage` + high-Latin-1 detection.
2. Add `test_synthetic_type0_broken_tounicode_emits_fffd_not_latin1_mojibake`
with a fully-synthetic Type0 / Identity-H PDF built in process. We
control the malformed ToUnicode contents, the descendant CIDFontType2
shape (just enough for `parse_type0_widths` to set `is_cid=true`,
which is what the new guard keys off of), and the Tj byte stream.
No fixture file or external license needed. Reproduces the exact
"Type0 + non-ASCII bytes + unparseable ToUnicode" code path that
produced the production mojibake samples.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: expand multi-row TSR cells in place
Recover row-under-counted TSR tables by splitting overstuffed cells with native PDF text bands before falling back to heuristic extraction.
Made-with: Cursor
* docs: note multi-row expansion scope
Clarify that the row-band cap intentionally keeps v1 focused on common small row-loss cases while larger compressions continue to use heuristic fallback.
Made-with: Cursor
Three fixes to extract_tables_with_structure_auto_mem (added in
1.7.1) caught by external review:
1. multi_row_in_cell over-triggered on legitimate multi-line cells.
The previous threshold (item span > 1.3× either smallest cell or
tallest item height) fires on any cell with 2+ y-separated text
items — including rowspan>1 cells, wrapped descriptions, and
superscript/subscript runs. Replaced with two gates:
- skip cells whose declared rowspan > 1 (intentional multi-line)
- require an actual whitespace gap (>~half a line height)
between the bottom of one item and the top of the next, in
PDF-native y-coordinates. Same-line items with tall glyphs or
superscripts have negative or near-zero gap; truly separate
visual rows have gap ≈ leading − line-height.
FNBO regression test still passes; new test covers a rowspan=2
cell with two visible text lines and verifies no fallback fires.
2. Heuristic returning empty silently replaced TSR markdown with
"". The auto wrapper now keeps the TSR markdown when the
heuristic markdown is empty/whitespace and tags fallback_reason
with `_heuristic_empty` suffix (e.g.
`multi_row_in_cell_heuristic_empty`). Worst case we ship the
same wrong-but-non-empty TSR output we'd have shipped before
1.7.1; we never replace useful output with literally nothing.
3. One bad input blanked the whole batch. Errors from
detect_tsr_quality_issue or extract_tables_in_regions_mem now
stay scoped to the single input — that input falls through to
raw TSR markdown with a `_error` reason label so callers can
metric on it. Other inputs in the batch are unaffected.
3 new integration tests:
- test_auto_does_not_fire_on_legit_rowspan_cell
- test_auto_keeps_tsr_markdown_when_heuristic_returns_empty
- test_auto_isolates_per_input_failures
All 6 auto tests + full 123-test suite pass. FNBO local replay
still triggers fallback (phantom_empty_row signal in this run) and
emits correct Shawnee/BVP/Sonoma rows with correct census tracts.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds extract_tables_with_structure_auto_mem (Rust) /
extractTablesWithStructureAuto (napi). Returns
TableExtractionResult { markdown, fallback_reason } per input.
The wrapper runs the existing TSR-hybrid path then checks the
resulting cells for two known SLANet detection pathologies:
* phantom_empty_row: empty row sandwiched between non-empty rows
(cheap, cell-metadata only).
* multi_row_in_cell: re-reads PDF text items, flags any cell whose
contained items span >1.3× either the smallest cell height or
the tallest contained item height. Catches the FNBO failure mode
where a tall TSR cell absorbs two adjacent PDF rows.
When either fires, extract_tables_in_regions_mem runs over the same
crop bbox and its markdown replaces the TSR markdown.
fallback_reason carries the diagnostic label so callers can emit
metrics and watch each pathology independently.
Validated on FNBO branches PDF page 1 (Kansas region):
- TSR-only: merges Shawnee into BVP, wrong census tract on Sonoma.
- Auto fallback (phantom_empty_row): each row separate, correct tracts.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the row-merge pattern observed against FNBO branch list at the
College/Fairway boundary: when SLANet emits cells whose y-extents
overlap between consecutive rows, an item whose center fell in the
overlap region got pulled into BOTH cells, producing run-on cells
(e.g. "Kansas Kansas" + concatenated addresses).
Cause: stage 1's "for each cell, gather items inside" loop allowed an
item to match multiple cells. `normalize_cell_bands` reduces overlap
but is biased when cell-mean-center is offset from actual text baseline
(SLANet bboxes are typically taller than their text content), so the
midpoint-clamp can land on the wrong side of the row boundary, and
items at the boundary still match two cells.
Fix: invert the matching. For each PDF text item, find candidate cells
(those whose bbox satisfies tsr_region_contains_item — center inside
OR >=60% overlap on both axes), and assign to the cell whose CENTER
is geometrically closest. Build per-cell text from the assigned items.
Stage 2 (orphan recovery) is unchanged.
Exclusivity prevents item duplication across cells. The closest-center
rule disambiguates the cell-overlap case naturally without aggressive
band clamping. normalize_cell_bands stays — it tightens cells before
matching (smaller overlap → fewer ambiguous candidates) but is no
longer load-bearing for correctness of the overlap case.
Local replay against FNBO via api/scripts/local-tsr-replay.ts (which
exercises the full layout-pod → table-pod → pdf-inspector chain
in-process):
Pre-1.6.4 (deployed 1.6.3):
|LITH West|Illinois Illinois|11700 S. IL Route 47, Huntley IL...
|College|||0534.03|
|Fairway|Kansas Kansas|4650 College Blvd... 2828 Shawnee Mission...
Post-1.6.4 (this change):
|Huntley|Illinois|11700 S. IL Route 47, Huntley IL...|8711.15|
|LITH West|Illinois|4520 W Algonquin Rd, Lake in the Hills...
|College|Kansas|4650 College Blvd, Overland Park KS...|0532.01|
|Fairway|Kansas|2828 Shawnee Mission Pkwy, Fairway KS...|0500.00|
One row (Shawnee in the Kansas section) can still get dropped under
SLANet detection variance — the model occasionally under-detects rows
and emits N structure rows for N+1 PDF rows. The squeezed row's text
gets routed to the structurally-nearest existing cell. This is a
SLANet limitation, not addressable in pdf-inspector without
synthesizing rows from PDF text geometry; deferred.
Tests: 416 lib + 120 integration + 2 doctests pass. clippy + fmt clean.
Bump @firecrawl/pdf-inspector to 1.6.4 (patch — refines stage 1's
matching strategy; no API changes).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the residual run-on-cell pattern observed on FNBO branch list
after 1.6.2 deployed:
|Shawnee Blue Valley Parkway|Kansas|6301 Pflumm, ...|0523.04|
|Sonoma Plaza|Kansas Kansas|<addr1> <addr2>|0531.05|
|Mitchell Woonsocket|South Dakota|<addr1>|9628.01|
Cause: when col-N cells across multiple consecutive rows are y-shifted
the same way (a local SLANet drift), the stage 2 orphan pass sees
multiple orphans qualifying for the same nearest empty cell. The cell
gets all of them appended in order, producing "RowA-text RowB-text".
Fix: track the y-coordinate of the first orphan that lands in each
cell. Subsequent orphans only join that cell if their y is within
half-a-row-height of the first orphan's y (same line). Cross-line
orphans skip that cell and look for the next-nearest empty cell on
their own line.
Same-line slack preserves multi-token branch names like
"Blue Valley Parkway" (3 PDF text items at the same y) — all three
stack into the same cell. Cross-row stacking is what gets rejected.
Two new tests:
- stage2_rejects_cross_line_stacking_into_same_cell
Two orphans on different rows, both equidistant to the same empty
cell. First wins; second routes to its own row's cell.
- stage2_allows_same_line_orphans_to_stack_into_one_cell
Three same-line orphans (multi-token branch name) all land in the
same empty cell, joined by spaces.
The existing four stage-2 tests + the cell-bleed regression test from
PR #62 + #63 all pass: 416 lib + 115 integration + 2 doctests.
clippy + fmt clean.
Bump @firecrawl/pdf-inspector to 1.6.3 (patch — refines 1.6.2; no API
changes).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #62 (1.6.1) closed the cell-bleed regression by clamping SLANet's
loose cell bboxes into non-overlapping row/column bands and tightening
text membership to center-containment OR >=60% overlap. That worked,
but exposed the opposite failure: legitimate native PDF text whose
center fell just outside the *clamped* cell bbox now had nowhere to go.
Two distinct failure modes observed against the FNBO branch-list PDF
after 1.6.1 deployed:
Symptom A — header text positioned at the LEFT of a column whose band
was derived from data-cell centers farther right. Header "Address"
PDF text at x=331..375 fell outside the clamped col band starting
at x=410. Strict membership rejected it (0% x-overlap, center
outside).
Symptom B — local SLANet row drift in col 0 over a 5-row stretch.
Cell bboxes sat just above the actual branch-name text items
(x-overlap 100% but y-overlap ~30-43%, below the 60% threshold).
Both share one root: post-normalization bboxes are too tight, and the
strict rule has no escape valve for legitimate edge text.
Fix: add a stage-2 orphan-recovery pass after the strict fill. Items
that NO cell claimed in stage 1 get re-assigned to their nearest
*empty* cell, distance-capped by `(median_col_width, median_row_height)`
so a far-orphan figure title can't get pulled into a faraway empty
cell. Stage 2 only fills empties — never overwrites stage 1 — so the
cell-bleed case PR #62 closed cannot regress.
Three new lib tests cover the bug shapes:
- stage2_recovers_left_aligned_header_text_outside_data_band
(Symptom A: header text left-of-band, data cells already filled,
stage 2 fills only the header)
- stage2_recovers_y_shifted_col0_in_consecutive_rows
(Symptom B: 3 col-0 cells shifted vs text, all 3 recovered)
- stage2_does_not_overwrite_filled_cells_or_admit_far_orphans
(cap rejects far figure titles; pre-filled cells untouched)
Plus tests for the cap-derivation helper:
- tsr_assignment_caps_uses_median_geometry
- tsr_assignment_caps_floor_protects_degenerate_input
The existing dense-overlapping-rows regression test (added in #62) still
passes, confirming no regression on the cell-bleed case.
Test results: 414 lib + 115 integration + 2 doctests pass. clippy + fmt
clean.
Bump @firecrawl/pdf-inspector to 1.6.2 (patch — fixes a regression
introduced by 1.6.1; no API changes).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: TSR-aware table extraction (extract_tables_with_structure_mem)
New public function that consumes raw structure-recovery output (HTML
structure tokens + per-cell bboxes from a model like SLANet) and assembles
markdown tables by pulling cell text from the native PDF — no OCR, no
geometry inference.
Why: the existing extract_tables_in_regions_mem infers grid geometry from
text positions only and can't distinguish merged cells from multiple narrow
columns. Pairing structure recovery from a layout/TSR model with native
PDF text gets perfect text quality with proper row/col/span structure.
- New module src/tables/structured.rs: token state machine, polygon→AABB,
crop-px→page-pt, rowspan/colspan-aware cell layout, markdown emitter.
Accepts both 4-element rects and 8-element 4-corner polygons.
- New public extract_tables_with_structure_mem in src/lib.rs that reuses
extract_page_text_items, region_overlaps_item, and the shared region
text-collection helper. No existing public function modified.
- napi binding extractTablesWithStructure mirroring the existing
extractTablesInRegions shape (f64 in JS → f32 internally).
- 14 unit tests + 5 integration tests, including a real-PDF gold-standard
match against bits_pilani_feedback.pdf.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TSR follow-ups: header-aware separator, cells API, v1.6.0
- cells_to_markdown emits the separator after the LAST row that contains
is_header=true cells, falling back to "after row 0" when no header is
flagged. Multi-row theads now render correctly. Three new unit tests
cover: multi-row header, header not on row 0, no headers (fallback).
- New public extract_tables_with_structure_cells_mem returning
Vec<Vec<StructuredCell>> so callers can drive their own rendering or
debug overlays without re-doing the parse + extraction. The markdown
variant now wraps it. The previously-unused page_pt_bbox field is
surfaced through this API.
- New napi binding extractTablesWithStructureCells + StructuredCellJs.
- Bump @firecrawl/pdf-inspector to 1.6.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix TSR cell text assignment for overlapping bboxes
Made-with: Cursor
* bump npm package version to 1.6.1
Made-with: Cursor
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: TSR-aware table extraction (extract_tables_with_structure_mem)
New public function that consumes raw structure-recovery output (HTML
structure tokens + per-cell bboxes from a model like SLANet) and assembles
markdown tables by pulling cell text from the native PDF — no OCR, no
geometry inference.
Why: the existing extract_tables_in_regions_mem infers grid geometry from
text positions only and can't distinguish merged cells from multiple narrow
columns. Pairing structure recovery from a layout/TSR model with native
PDF text gets perfect text quality with proper row/col/span structure.
- New module src/tables/structured.rs: token state machine, polygon→AABB,
crop-px→page-pt, rowspan/colspan-aware cell layout, markdown emitter.
Accepts both 4-element rects and 8-element 4-corner polygons.
- New public extract_tables_with_structure_mem in src/lib.rs that reuses
extract_page_text_items, region_overlaps_item, and the shared region
text-collection helper. No existing public function modified.
- napi binding extractTablesWithStructure mirroring the existing
extractTablesInRegions shape (f64 in JS → f32 internally).
- 14 unit tests + 5 integration tests, including a real-PDF gold-standard
match against bits_pilani_feedback.pdf.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TSR follow-ups: header-aware separator, cells API, v1.6.0
- cells_to_markdown emits the separator after the LAST row that contains
is_header=true cells, falling back to "after row 0" when no header is
flagged. Multi-row theads now render correctly. Three new unit tests
cover: multi-row header, header not on row 0, no headers (fallback).
- New public extract_tables_with_structure_cells_mem returning
Vec<Vec<StructuredCell>> so callers can drive their own rendering or
debug overlays without re-doing the parse + extraction. The markdown
variant now wraps it. The previously-unused page_pt_bbox field is
surfaced through this API.
- New napi binding extractTablesWithStructureCells + StructuredCellJs.
- Bump @firecrawl/pdf-inspector to 1.6.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The rect-based cell fallback in detect_row_stripe_table_from_cell_rects
derives columns purely from text X-position clustering. When prose
wraps inside a bounding-box rect (chat transcripts, stylized figures),
the word-boundary gaps cluster into many spurious columns, producing
a multi-column "table" that is just fragmented prose.
Count cells containing common English function words (articles,
prepositions, pronouns, common verbs) and reject the fallback when
20%+ of non-empty cells contain any such word. Real tabular data —
labels, units, numbers, short identifiers — rarely contains these.
Update the td9264 snapshot: the government document section that
previously rendered as a malformed table now renders as cleaner
prose + a proper CFR list.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: don't reclassify wrapped bold list leads as headings
When a numbered/bulleted list item's bold lead phrase wraps onto a
second visual line, that line is all_bold + standalone, which scored
above the rarity heading threshold and was emitted as #### in the
middle of the item. That reset in_list, so the body continuation
below picked up a stray `- ` bullet via the struct-tree LI path,
shattering a single item into heading + stray bullets.
Guard the font heuristic: when already inside a list, skip heading
classification for lines at the list continuation indent with a Y
gap within para_threshold. Structure-tree headings still win.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: tighten rect-row span check in propagate_merged_cells
propagate_merged_cells used an overlap-based predicate with ±tol
slop that returned true at shared row boundaries — a rect whose
top exactly equals row N's bottom lies entirely below the row, yet
the predicate considered it to span row N. When multiple background
rects aligned on a shared Y edge (e.g. consecutive row-stripe
shading), each adjacent rect would over-reach by one row, cascading
labels and data from unrelated rows into a single merged cell.
Replace the overlap predicate with a containment check: rect bottom
at or below row bottom, rect top at or above row top (each within
tol). Genuine merged-cell rects fully contain the rows they span;
tangent rects do not.
Update two snapshots that were encoding the old buggy output.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a numbered/bulleted list item's bold lead phrase wraps onto a
second visual line, that line is all_bold + standalone, which scored
above the rarity heading threshold and was emitted as #### in the
middle of the item. That reset in_list, so the body continuation
below picked up a stray `- ` bullet via the struct-tree LI path,
shattering a single item into heading + stray bullets.
Guard the font heuristic: when already inside a list, skip heading
classification for lines at the list continuation indent with a Y
gap within para_threshold. Structure-tree headings still win.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On PDFs where every list item starts with ● at the left margin and content
at a fixed offset, histogram column detection sees the gap between marker
and content as a gutter and splits each line across two phantom "columns,"
scrambling the reading order (Anthropic's Mythos system card p.73–74).
- layout: reject gutter candidates where the smaller side is ≥80%
standalone bullet-marker glyphs (•, ●, ○, ◦, ▪, ▫, ◆, ◇, ■, □)
- markdown/classify: add starts_with_bullet_marker helper (narrower than
is_list_item — excludes numbered/lettered patterns like 1. and a) so
numbered section headings stay as headings)
- markdown/convert: skip heuristic heading detection on lines that start
with a bullet marker
- markdown/classify: strip a leading bullet wrapped in a bold/italic run
(e.g. "**● Label:**" → "- **Label:**") — some PDFs put the marker inside
the same bold run as the label
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Some tagged PDFs use a flat style where each wrapped visual line of a list
item gets its own MCID tagged directly under /LI. The LI branch was
unconditionally prefixing every such line with "- ", turning continuation
lines into their own bullets. Only emit a new bullet when we're not already
inside a list; otherwise fall through to the existing continuation logic so
the text is appended to the previous item.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: expose per-page markdown extraction to Python and Node (#49)
Implements the feature requested in issue #49: a list-of-pages markdown output
from the Python API. Matching the existing project pattern, the feature lives
in the Rust core and is surfaced through every binding.
- Rust core: `extract_pages_markdown` (path) and `extract_pages_markdown_mem`
(bytes) now take `Option<&[u32]>` — `None` returns every page in document
order; a slice restricts and preserves caller order.
- Python: new `extract_pages_markdown(path, pages=None)` and
`extract_pages_markdown_bytes(data, pages=None)` functions plus
`PageMarkdown` / `PagesExtractionResult` classes; stub file updated.
- Node: `extractPagesMarkdown(buffer, pages?)` — `pages` is now optional.
- Tests: 2 new Rust integration tests, 9 new Python tests, 2 new Node
assertions. All 372 unit + 107 integration + 53 Python tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version from 1.3.0 to 1.4.0
Minor bump for the new per-page markdown extraction API exposed through
the Python and Node bindings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The TOC/data-table distinction was being recomputed at every consumer:
- format.rs::table_to_markdown ran is_table_of_contents to decide between
flat-list and markdown-table rendering.
- compute_layout_complexity ran it again to filter TOCs out of
pages_with_tables.
- detect_heuristic validations used it to decide whether to relax val 1/9.
Each caller had to remember tables can be either kind, which leaks the
TOC concept across the codebase.
Add `TableKind { Data, Toc }` and a `Table::new` constructor that classifies
once from the cells. All five detectors (heuristic, rect, line, struct,
columns) now go through `Table::new`. Consumers match on `kind` instead of
re-running classification.
Pure refactor — no behavior change. Verified: pdf-evals output is byte-for-
byte identical (0 changed snapshots).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: detect hierarchical-indent TOCs as tables
Validation 1 (≥25% of rows have first col) and validation 9 (paragraph
content) were rejecting TOC pages where top-level chapters indent at the
leftmost X but subsections cascade further right. The leftmost column ends
up sparse (only chapter rows land there) and most cells are empty, but the
structure is still an unambiguous TOC.
Skip both validations when cells form a TOC pattern AND the table is narrow
(≤5 cols). The width cap preserves existing handling of wide multi-column
TOCs (e.g. back-of-book 2-up indices) where format_toc_as_list would mash
adjacent visual entries together.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: exclude TOC pages from pages_with_tables metadata
TOC detection routes through the table pipeline (detected as a table, then
formatted as a flat list with tab-aligned page numbers via
format_toc_as_list). That made TOC pages appear in LayoutComplexity's
pages_with_tables, which:
- Misleads downstream consumers that read this field as "this page has a
data table".
- Trips the table-page guard in column detection, which switches to a
different valley-detection threshold for table pages.
Add an is_table_of_contents check in compute_layout_complexity so TOC-shaped
detections don't count toward the table flag. The TOC still renders correctly
as a flat list — only the metadata classification changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Validation 1 (≥25% of rows have first col) and validation 9 (paragraph
content) were rejecting TOC pages where top-level chapters indent at the
leftmost X but subsections cascade further right. The leftmost column ends
up sparse (only chapter rows land there) and most cells are empty, but the
structure is still an unambiguous TOC.
Skip both validations when cells form a TOC pattern AND the table is narrow
(≤5 cols). The width cap preserves existing handling of wide multi-column
TOCs (e.g. back-of-book 2-up indices) where format_toc_as_list would mash
adjacent visual entries together.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The subset-GID-mismatch heuristic tripped whenever the CIDFont's W array
started at CID 0 and the ToUnicode CMap's minimum source CID was > 2.
That's the normal sparse layout for a correctly-subsetted Identity-H
font with a .notdef at CID 0 and actual glyphs at high CIDs (Cyrillic,
Arabic, etc.). The spurious remap to sequential CIDs, combined with
score_text penalizing non-Latin letters as "other", caused the garbled
CMap to win over the correct primary — spaces turned into %, digits
shifted into punctuation, and Latin letters got rewritten to Cyrillic
codepoints.
Check whether the W array actually covers the CMap's maximum source CID.
If it does, the font and CMap agree, no renumbering happened, and the
remap stays off. True mismatches (CMap CIDs outside W coverage) still
trigger the remap.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds x86_64-pc-windows-msvc target to the napi package and CI build
matrix so the published @firecrawl/pdf-inspector npm package ships
native Windows binaries alongside Linux and macOS.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix: reject dot-less tables of contents as heuristic table false-positive
The body-font heuristic table detector was firing on tagged PDF TOCs
whose entries are laid out as multi-column rows (section number, title,
page number) without dot leaders. Extend is_table_of_contents to flag
this pattern by requiring:
- 2+ columns and 4+ rows
- >=60% of rows start with a dotted section number (e.g. "4.3.1")
- >=70% of filled last-column cells are all-digit page numbers
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* format TOC tables as flat per-row list instead of dropping them
Keeping the previous approach (rejecting TOCs at detection time) meant
dot-less TOCs fell back to the column-aware text reader, which stacked
every section title first and dumped all page numbers at the end of the
region — so a reader saw a wall of titles followed by a wall of numbers.
Keep the detected Table, and in the formatter interleave the cells into
one line per row with the page number appended after a tab. The raw
(pre-clean) cells are used for the TOC check because clean_table_cells
can collapse genuine data tables into a TOC-looking shape.
Tightened is_table_of_contents to require both dot leaders AND page
numbers (previously dot-ratio alone was enough, which misfired on
register tables that use "..." as a continuation marker).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* split TOC detection into dot-leader and tabular patterns
Previous is_table_of_contents rejected every TOC-shaped table at detect
time, dropping dot-leader TOCs that render well as flat-list output and
misclassifying data tables with trailing "..." cells as TOCs.
Now is_dot_leader_toc (structural + inline-leader) keeps per-row flat
layouts for the formatter, while only wide inline-leader indices are
rejected at detect time. row_cell_is_page_number accepts dashed
section-page IDs ("A-1", "5-21") and rejects decimals/thousands
separators; cell_has_trailing_leader requires alphabetic content so
numeric data-row labels ("1973 ... ") no longer qualify.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat: add CLI bin to npm package
Installing `firecrawl-pdf-inspector` now provides a `pdf-inspector` CLI command.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump npm package version to 0.8.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump npm package version to 1.0.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use first-glyph position for ActualText ligature items
The content stream parser emitted ActualText items (from BDC/Span
marked content) at the text matrix position captured at BDC entry.
When the content stream contains Td operators between BDC and the
first glyph Tj — common in PDFs generated by Google Docs, Figma,
and web-to-PDF tools — the BDC-entry position is on the previous
visual line while the actual glyph renders on the correct line.
This caused ligature glyphs (fi, fl, ff, etc.) wrapped in
ActualText spans to be positioned one text-leading offset above
their surrounding text. Downstream merge logic couldn't reconnect
them (different y-band), producing broken words like "fi" + "ndings"
instead of "findings" throughout the document.
Fix: capture the text matrix at the first Tj/TJ operation inside
the BDC block (after any Td repositioning), and use that as the
ActualText item's rendering position at EMC time. Falls back to the
BDC-entry position when no glyph ops occur inside the block.
One new variable, five insertion points, zero behavior change on
PDFs that don't use ActualText marked content.
* chore: allow clippy::collapsible_match for Rust 1.95
Rust 1.95 introduced the collapsible_match lint which flags `if`
blocks inside match arms that could be converted to match guards.
The content-stream parsers use this pattern extensively for
readability (match on PDF operator name, then check preconditions
like `in_text_block && !op.operands.is_empty()`). Allow crate-wide
rather than refactoring 21 match arms across the parser files.
* fix(detector): correct false flags for CID-encoded text and supplementary fonts (0.7.4)
The page classifier was over-aggressively flagging Mixed-PDF pages as
needing OCR in three distinct cases. Each is fixed at the root in
analyze_page_content / page_has_identity_h_no_tounicode / the
looks_like_scan check.
1. has_vector_text false positives on dense layouts
path_ops > text_ops*200 fired on pages with decorative paths
(column borders, dividers) alongside real selectable text. Added
a unique_alphanum_chars < 30 guard: real outlined-text pages have
very few unique alphanum chars (each glyph is a path), while
pages with real text + decorations have many.
2. Identity-H without ToUnicode flagged whole pages on supplementary fonts
page_has_identity_h_no_tounicode would flag a page if any single
Type0 font lacked ToUnicode and had no fallback CMap, even when
the page's actual text came from other decodable fonts (Type1
with ToUnicode, etc.). Rewrote to track both undecodable
Identity-H fonts AND other decodable fonts, only flagging when
no decodable text font is present.
3. CID-encoded text with ToUnicode misclassified as scan
looks_like_scan checked unique_alphanum_chars < 10 on raw string
operand bytes. CID-encoded fonts (Type0 with ToUnicode) emit
2-byte CID values that aren't ASCII alphanum, so the metric is
blind to them even when the text is fully decodable. Added a
has_decodable_text_fonts signal: when a page has decodable fonts
AND >= 10 text ops, the low alphanum count is treated as a CID
encoding artifact rather than evidence of a scan.
Validated against a broad PDF corpus:
- 6 known false-positive pages now correctly classified as text
- 22 previously-missed scan pages (cover/blank/photo) now correctly
flagged for OCR
- 0 regressions on truly-scanned PDFs (61/61 pages stay flagged)
- All 437 existing tests pass; clippy clean
Bumps NAPI package to 0.7.4.
* test(detector): add unit tests for the three classifier fixes
Adds 10 unit tests covering the heuristic changes:
- has_vector_text alphanum guard
- real text + decorative paths → not flagged
- true outlined glyphs (low alphanum) → still flagged
- page_has_identity_h_no_tounicode supplementary-font handling
- undecodable Identity-H + decodable Type1 → not flagged (new)
- undecodable Identity-H alone → still flagged (regression)
- page_has_decodable_text_fonts (new helper)
- Type1 → true
- Type0 with ToUnicode → true
- undecodable Identity-H only → false
- looks_like_scan with has_decodable_text_fonts override
- CID-encoded decodable text → not flagged as scan
- same metrics with no decodable fonts → still flagged
- decodable fonts but text_ops < 10 (page-number overlay) → still flagged
* fix(detector): make decodable-font checks usage-based and XObject-aware
Addresses two reviewer concerns on the previous heuristic fix:
P1 — resource-based check could create an inverse bug
page_has_identity_h_no_tounicode and page_has_decodable_text_fonts
iterated all fonts in the page Resources dict, including unused fonts.
A page whose actual text was rendered exclusively in an undecodable
Identity-H font but whose Resources also listed an unused decodable
Type1 would be wrongly unflagged.
Fix: parse Tf operator operands during content stream scanning to
collect the set of font names actually referenced. The font checks
now filter to only USED fonts via a new used_fonts_have_*
family of functions operating on (used_font_names, font_map).
P2 — checks didn't follow text into Form XObjects
analyze_page_content correctly recurses through Form XObjects via
scan_xobjects_in_resources, but the font checks only looked at the
page's top-level Resources/Font. Pages that render text through Form
XObjects (corporate templates, header/footer overlays) had their
XObject font resources missed entirely.
Fix: scan_xobjects_in_resources now propagates the used_font_names
set AND collects fonts from each Form XObject's own Resources into
the shared font_map. The usage-based check sees the full picture:
page-level fonts + every nested XObject's fonts, intersected with
fonts actually referenced by Tf operators anywhere in the content.
Implementation:
- New extract_font_name_before_tf helper (parses /Name immediately
preceding Tf).
- New FontInfo struct caches font properties per-name.
- New collect_fonts_from_resource_dict + new used_fonts_have_*
functions are pure filters over (used_names, font_map).
- analyze_page_content threads used_font_names + font_map through
page content scan and XObject recursion, then runs the new checks.
- Old resource-based functions kept as #[cfg(test)] for the existing
unit-test interface.
- Phase 3 uncached-page loop now goes through analyze_page_content
so it also gets the usage-based + XObject-aware behavior.
Tests added (8):
- extract_font_name_before_tf basic + long-name parsing
- scan_content_for_text_operators collects used font names
- P1 — unused decodable font in Resources doesn't save a page
whose used font is undecodable
- P1 — both fonts used → decodable font correctly prevents flag
- P2 — decodable font inside Form XObject correctly unflags
- P2 — undecodable font only in XObject still flags even with
unused decodable font at page level
- P2 — has_decodable_text_fonts populated from XObject fonts
Validation:
- 349 lib + 104 integration + 2 doc tests pass (was 341)
- cargo clippy --lib --bin detect-pdf -- -D warnings: clean
- External eval: 9/9 PDFs pass, 6/6 false positives resolved,
0 regressions, 61/61 scanned pages still correctly flagged
- No eval delta — confirms previous fix wasn't relying on the
resource-based bug for any of the eval PDFs
* fix(detector): scope font lookups by ObjectId + handle indirect Form Resources
Addresses two more reviewer findings on the previous decodable-font commit.
P1 — Resource-name scoping bug
The previous fix keyed used_font_names and font_map by raw resource
names like b"F1". PDF resource names are scoped to each resource
dictionary: a Form XObject can legally define its own /F1 that points
to a completely different font from the page's /F1. Because
collect_fonts_from_resource_dict skipped duplicates with
`if font_map.contains_key(name)`, the first definition won and later
Tf /F1 usages in different scopes resolved against the wrong font.
This could reintroduce both the undecodable-Identity-H false flag
and the decodable-CID false unflag depending on which side of the
collision happened to be inserted first.
Fix: switch the lookup mechanism from font names to font ObjectIds.
- font_map: HashMap<ObjectId, FontInfo> (was Vec<u8> keys)
- used_font_ids: HashSet<ObjectId> (was Vec<u8> names)
- new resolve_font_names_to_ids() runs immediately after each
content scan, against the resource dict in scope, to translate
the per-scope name set into ObjectIds.
Each Form XObject's content stream now resolves /F1 against THAT
XObject's own Resources, so name collisions are impossible by design.
Inline (no-ID) font dicts are skipped — extremely rare in practice
and have no stable key.
P2 — Indirect Form /Resources skipped
scan_xobjects_in_resources used `.as_dict()` on the Form's /Resources
entry, which returns None for indirect references. PDFs frequently
store /Resources as `X 0 R`, in which case font collection and
recursion were both skipped — even though the Tf usages inside the
XObject content had already been recorded.
Fix: handle Object::Reference(r) in addition to Object::Dictionary(d)
by resolving via doc.get_dictionary. Audited the rest of the file —
the other /Resources access points (analyze_page_images,
collect_images_from_resources) already handled both cases.
Tests added (4):
- P1 same-name-different-font (page undecodable, XObject decodable):
must NOT flag — XObject's text is decodable in its own scope.
- P1 inverse (page decodable, XObject undecodable, content uses
XObject /F1): MUST flag — undecodable text exists in real scope.
- P2 indirect Form /Resources: font discovery must still work when
/Resources is a `X 0 R` reference rather than inline.
- Combined regression: indirect Resources + name collision.
Validation:
- cargo test --release: 459 tests pass (353 lib + 104 integration + 2 doc)
- cargo clippy --lib --bin detect-pdf -- -D warnings: clean
- external eval (9 PDFs): 9/9 pass, 6/6 false positives resolved,
0 regressions, 61/61 truly-scanned pages still flagged
The behavior on the eval set is identical — confirms the correctness
fix isn't masking any change in classifier outcomes.
* fix(detector): respect resource shadowing when resolving page-content fonts
The previous ObjectId-based fix correctly scoped Form XObject fonts
but still violated PDF resource inheritance for page content. When a
page overrides /F1 from a parent /Pages node (different font dict for
the same name), get_page_resources returns the page's own /Resources
plus all ancestor /Resources dicts. The old code called
resolve_font_names_to_ids on each one and added every match to
used_font_ids — both font ObjectIds ended up in the used set even
though only the page's /F1 is actually visible to that page's content.
Per ISO 32000-1 §7.7.3.4, resource names are inherited with
shadowing semantics: the most-specific (deepest, closest to the page)
definition wins.
Fix:
- New lookup_font_id helper resolves a single name in a single dict.
- New resolve_with_shadowing iterates names, checking the page's own
/Resources first, then walking ancestors in most-specific-first
order (which is the order lopdf's get_page_resources returns).
First hit wins via a labeled `continue 'name` — subsequent
ancestors are skipped for that name.
- analyze_page_content's flat resolution loop replaced with one call
to resolve_with_shadowing.
Audit:
- XObject path is correct: each Form XObject already resolves names
against its OWN /Resources (XObjects don't inherit from page tree).
- font_map population is correct: keyed by ObjectId, so collecting
from all dicts builds the full available-fonts catalog. The bug
was only in the used-set resolution.
- Confirmed lopdf returns ancestors in most-specific-first order
(page → parent → grandparent → root), matching the shadowing
direction used here.
Tests added (3):
- page /F1 undecodable shadows parent's decodable /F1 → MUST flag
- page /F1 decodable shadows parent's undecodable /F1 → MUST NOT flag
- no override: page inherits parent's decodable /F1 → MUST NOT flag
Validation:
- cargo test --release: 462 tests pass (356 lib + 104 integration + 2 doc)
- cargo clippy --lib --bin detect-pdf -- -D warnings: clean
- external eval: 9/9, 0 regressions, 6/6 false positives resolved,
61/61 scanned pages still correctly flagged
The looks_like_scan check incorrectly used OR logic, causing any single
condition (image_count <= 1, text_ops < 50, alphanum < 10) to flag a page
as a scan. A real scan has ALL three: single full-page image AND low text
AND low alphanum. Text pages with one figure were falsely flagged for OCR.
Bump napi to 0.7.3.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: reduce false OCR recommendations for text PDFs with figure images
Two fixes in the detector:
1. Fix Tf operator parsing: some PDFs concatenate Tf directly with the
next operator (e.g. "25 Tf[<01>...") without whitespace. The scanner
now accepts [, (, <, / as valid followers, fixing font_changes being
reported as 0.
2. Distinguish text-with-figures from scanned-with-OCR: pages with
multiple images (image_count > 1) and strong text signals (text_ops
>= 50, alphanum >= 10) are recognized as text pages with figures,
not scanned templates. Scanned PDFs have exactly 1 full-page image.
This prevents academic papers, reports with charts, and similar PDFs
from being incorrectly classified as Mixed/OCR-needed when their text
is perfectly extractable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump napi version to 0.7.2
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove template image influence from page classification
Template images (large background/figure images) no longer affect
pages_needing_ocr. In the region-based pipeline, text regions are
extracted independently from image regions, and per-region needs_ocr
quality checks handle scanned-with-OCR garbage text.
Also makes the invisible text retry (for OCR text layers) trigger on
text quality rather than PDF type, so it works regardless of
classification.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Revert "fix: remove template image influence from page classification"
This reverts commit 100cbe5453.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: improve heuristic table detection for numeric columns and multi-line headers
Two fixes for tables that have clean extractable text but fail heuristic
structure detection:
1. Numeric column merge pass (grid.rs): After initial X-position
clustering, adjacent clusters are merged when one is sparse (header
text) and the other is dense with >50% numeric items (data column).
Multi-line wrapped headers often land slightly offset from their
data column — the merge closes gaps within 1.5× the clustering
threshold. New is_numeric_text() helper matches decimals, percentages,
negative numbers, and comma-separated thousands.
2. Duplicate-header skip (detect_heuristic.rs): Spanning super-headers
like "First Degree | First Degree | Higher Degree" contain duplicate
cells that trigger looks_like_partial_table_ex rejection. Now skips
rows with duplicate cells when a better header candidate exists
within the next 3 rows (higher fill ratio or numeric cells).
Tested on BITS Pilani university report (430 pages, 314 table pages).
Page 4 (multi-line header + numeric data) previously returned
needs_ocr=true; now correctly detects the table structure.
Eval: 197 PDFs, zero regressions, all 104+ tests pass, zero clippy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* bump version to 0.7.1
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
