* feat(markdown): underline emission, Unicode scripts, style-preserving merges (ENG-5015 2b)
Three formatting losses in the direct-extraction markdown path:
1. text_with_formatting gains <u> run emission (detect_underline option,
default on) using the geometric is_underline flag from 1.9.9.
Underline runs stay free of nested bold/italic markers — consumers
match tag content literally. Heading lines keep plain text for
bold/italic but preserve <u>: the tag carries meaning `#` doesn't.
2. merge_subscript_items now maps absorbed digit scripts to Unicode
sub/superscript forms with direction from the baseline offset
("H"+"2" -> "H₂", "word"+raised "2" -> "word²", "m"+"3" -> "m³").
NFKC/NFKD folds these back to plain digits so text matching
downstream is unaffected; renderers keep the script semantics.
3. merge_text_items no longer merges across bold/italic boundaries —
absorbing a styled run into a plain neighbor erased the styling
before markdown emission ever saw it. On eval docs this recovers
20-82 italic runs per document that previously emitted as plain.
Snapshots regenerated (diffs are the features: CCl₂F₂, m³, underlined
legal section headings, finer bold runs). pdf-evals regression suite:
202/202 real PDFs pass. napi 1.9.9 -> 1.9.10.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): break merges at underline boundaries too (review)
OR-merging underline stretched the eventual <u> span over neighboring
plain fragments. Merge runs now break on any style-flag change, the
redundant accumulator is gone, and format_list_item learned to move
bullet markers outside <u> wrappers so fully-underlined bullet lines
still render as markdown lists. td9264 snapshot regenerated — spans are
tighter (trailing periods correctly outside the tag).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(markdown): strip stray spaces before sentence punctuation (review)
Style-boundary item splits can strand a trailing period in its own
fragment, and multiple assembly paths join fragments with spaces,
yielding "word ." artifacts. Rather than chasing every join site, a
postprocess pass removes a space before `.`/`,`/`;` when the mark ends
its token (whitespace, cell boundary `|`, or end of text follows).
Dot leaders/ellipses and mid-token periods are untouched.
Fixes the td9264 "companies ." artifacts and two pre-existing
"armoring ," artifacts in the 2013-app2 snapshot. pdf-evals: zero
markdown diffs across all 203 corpus PDFs vs committed baselines.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tables): trim spaces inside parenthetical cell fragments
* fix(tables): reject sparse prose row-stripe tables
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(extractor): geometric underline detection on TextItem (ENG-5015)
PDFs carry no underline font flag — underlines are stroked horizontal
lines or thin filled rects drawn under the baseline. Correlate those
graphics (already parsed from the content stream) with text items in a
post-pass: a rule within ~0.35em below the baseline covering >=60% of
an item's width marks is_underline.
Exposed through the napi and python bindings. Verified on real docs:
4/4 underlined sentences flagged on a Japanese report, links/headings
flagged on 8 of 10 underline-bearing eval docs, zero flags on docs
without underlines. Known FP source (table cell borders) documented —
downstream applies inline styling only to plain-text regions.
napi 1.9.8 -> 1.9.9.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): underline rules only from painted rects, normalized extents (review)
Two review fixes: (1) normalize rect extents before the thickness/width
checks — `re` operands pass through the CTM so width/height can be
negative, which missed negative-width rules and let negative-height
bands pass as thin; (2) only feed painted rects to underline detection —
`re` rects now wait in a pending list until a paint operator (S/s, f/F/
f*, B/B*/b/b*) confirms them, and `re W n` clip-only paths are discarded
at `n`, so invisible clip boundaries no longer underline nearby text.
Marking moved into content_stream where paint state lives (pre-rotation,
consistent device space).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): harden underline detection
* feat(cli): export positioned text item json
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* emit ItemType::Image bboxes for Image XObjects (was: silently dropped)
Background. ItemType::Image, MarkdownOptions::include_images, and the
markdown emitter's image-collection path have all been in the tree
for a while, but no producer ever populated them — content_stream.rs
explicitly `// Skip images — text extraction only` at the Do
operator, and the nested Form-XObject walker in xobjects.rs only
matched XObjectType::Form, silently dropping Image entries. The
declared types were dead code.
This PR lights them up. At every Do that resolves to an Image
XObject (both top-level and nested inside Form XObjects), we now
compute the page-space bbox from the current CTM via a new
`image_bbox_from_ctm` helper — handling both axis-aligned and
rotated/sheared placements via 4-corner AABB — and emit a TextItem
with `item_type: ItemType::Image` and the legacy `[Image: <name>]`
text payload that the markdown emitter already knows how to render.
Callers can now find raster figures via `extract_text_with_positions`
(and the `_mem` variant, newly re-exported at the crate root) without
needing to re-parse the PDF or run a vision/layout model. The intended
consumer is layout-aware text pipelines that want to crop figures and
caption them out-of-band.
Two backstops to avoid silent breakage for existing callers:
1. `MarkdownOptions::include_images` default flipped `true → false`.
If it stayed at `true`, every existing user of
`extract_pages_markdown` would suddenly see ``
placeholders inserted throughout their output the moment they
upgraded. Image data is still available structurally via
`extract_text_with_positions`; rendering it into markdown is now
an opt-in. New regression test asserts `extract_pages_markdown`
output is unchanged for the image-bearing fixture.
2. Image items now also skip the layout heuristics
(`detect_columns`, `detect_tables_from_rects`) via a new
`is_text_layout_item` predicate. Without this filter, an image's
left edge would land in the column-projection profile and skew
table column detection — surfaced by
`vector_grid_tests::upstage_key_functions_four_cols` going from 4
detected columns to 5 in CI before the filter was added.
Re-exporting `extract_text_with_positions_mem` at the crate root —
strictly additive; mirrors how `extract_pages_markdown_mem` is already
available there.
Tests:
- test_extract_text_with_positions_emits_image_bboxes — minimal PDF
with one 200×100 image at (50, 600); asserts one Image item with
correct bbox + page + text.
- test_image_xobject_bbox_handles_rotated_ctm — 90° rotated image
via shear-component CTM; asserts AABB is correct (handles non-
axis-aligned placements via 4-corner clamp).
- test_image_emission_does_not_change_default_markdown — asserts no
`Image:` token leaks into default markdown output, regression
guard for the include_images flip.
- test_markdown_options_default_has_include_images_false — explicit
sentinel so anyone flipping it back catches it in CI.
* Bump version from 1.8.15 to 1.9.0
text_cluster_column_undercount previously fired only on tables with
6+ markdown columns. That missed a common production failure shape:
4-column page geometry where the heuristic detector's x-position
clustering collapses 2 narrow numeric columns (dates, IDs, amounts)
into adjacent wider columns, producing 2-column markdown.
The original 6-column floor existed because raw x-cluster count is
noisy on small tables — wrapped continuations, bullet indents, and
within-cell text variation produce many small x-clusters that don't
correspond to real columns. Examples:
- Pcmso-style 2-column key/value layout with multi-line values
shows 10 raw x-clusters but only 1 has more than a single item.
- Yale-style archival catalog with 3 columns shows 6 raw
x-clusters but most clusters are single-item continuations.
Replace the raw cluster count with a "significant cluster" count:
clusters whose item count is at least 1/4 of the dominant cluster
(and ≥2 items). That filters the within-cell-variation noise while
preserving signal from real columns, which consistently have one
item per row.
With significant-cluster counting:
- Narrow-undercount fires when significant_clusters >= 3 AND
significant_clusters >= table_cols * 2 (catches 4-col-collapsed-
to-2 cases without misfiring on pcmso 10-clusters or yale 6
where the significant subset matches markdown).
- Legacy wide-undercount path (table_cols >= 6) keeps the same
+2 / 1.2x thresholds, now applied to significant clusters
instead of raw clusters.
Verified against representative regression and must-not-regress
cases from prior PR cycles: narrow-undercount catches genuine
column-drop cases (significant=4, markdown=2 → routes to OCR);
must-not-regress fixtures (italian-gov 6520 chars, pcmso 1733
chars, yale 1754 chars, doc200/182/189 still routing to OCR via
PR #91 guards) unchanged.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Some PDFs have fonts whose ToUnicode CMap is missing or broken —
Identity-H fonts without unicode metadata, Type-3 fonts where every
glyph maps to garbage. The page extractor returns punctuation-only
fragments or single-glyph repeats; the rendered image still carries
the visible text, so the region should fall back to OCR rather than
serve a partial table.
The existing captured_only_a_fragment guard can't catch this case
because region_text_chars itself collapses under font-decode failure —
captured vs extracted is symmetrically low, and the ratio still looks
acceptable.
Add a complementary area-based density guard: when a region has lots
of pixel real estate but very few text chars, the page extractor
hit a font failure. Bbox area is independent of extraction success,
so the symmetry breaks.
Threshold 0.003 chars/sq pt sits between observed clean extractions
(≥0.005 on full-page A4 ledgers, key/value layouts, archival
catalogs) and observed font-decode failures (≤0.0014 on prod-traffic
samples). Three guards keep it from misfiring:
- text_chars < 20 skipped: synthetic / fragmentary fixtures
- area < 30,000 sq pt skipped: tiny stat blocks
- area > 400,000 sq pt skipped: near-whole-A4 bboxes where
density is unreliable (large white-space margins)
Verified against three reproducible cases from prod shadow logs
that previously served partial output:
- Cyrillic page with punctuation-only decode (46 chars, density
0.00045) → flagged, routes to OCR
- Cyrillic page where every glyph collapsed to one letter (89
chars, density 0.00025) → flagged, routes to OCR
- Materials-test region where text extracted fine but the table
body extends beyond the bbox (96 chars, density 0.00131)
→ flagged, routes to OCR
Existing fixtures (full-page A4 ledger, multi-row key/value with
paragraph values, archival catalog, bits_pilani whole-page tests,
synthetic line-grid test) all retain identical behavior.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After enabling the vector-grid detectors on the extract path (#85)
and broadening detection to full-page grids (#83), long-cell tables
(#84), and segment-only layouts (#86), one residual failure shape
remained: detectors finding a valid grid but only capturing a small
fraction of the region's actual text. Two recurring sub-shapes:
- "header-only": detector captured the column-header band (often a
multi-line year/units block) but missed every data row below.
Common in financial statements, securities tables, budget
appendices.
- "sparse": detector returned a handful of fragmentary cells from a
content-rich region, missing the bulk of the page.
Both pass the existing needs_ocr quality gates — the captured cells
are well-formed markdown — but the customer would receive a 5-row
fragment of a 50-row table. Today these regions fell back to GLM-OCR
by default; flipping `__nativeTableExtraction=true` would start
serving the partials.
Add `captured_only_a_fragment(md, region_text_chars)`: rejects when
the captured non-delimiter character count is less than 25% of the
text the page extractor saw inside the region. The 200-char region
floor keeps short legitimate tables (units, axis labels) from being
mis-flagged. Wired into the existing `evaluate` quality gate
alongside is_garbage_text / is_cid_garbage / detect_encoding_issues
/ looks_like_partial_table_ex.
Verified against three representative residual cases from shadow
logs (financial-statement header band, securities-table fragment,
ESIA sparse region): all flip from `needs_ocr=false` with partial
output to `needs_ocr=true` so GLM takes over. Existing full-table
fixtures (governmental ledger, PPRA-style key/value, archival
catalog) still pass through unchanged.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Some catalog and archival-finding-aid tables draw each row's
horizontal rule as N segments (one segment per cell) with no vertical
lines at all. The previous detector rejected these outright at
`verticals.len() < 2`, even though the segment break points encoded
the column boundaries unambiguously.
When the vertical-line count is below the existing threshold, walk
the horizontal-segment x-endpoints and cluster them with the same
snap_edges path used for vertical-line columns. Accept the derived
edges only when ≥3 distinct x-positions each appear on ≥50% of the
unique horizontal-line rows — that consistency guard distinguishes
real per-cell segments from decorative rules with varying widths
(which never share endpoints across many rows).
When columns come from segment endpoints, skip the downstream
"spanning_v / partial_v" gate (there are no vertical lines to
validate against). All other gates — horizontal-span coverage,
content density, capture ratio, multi-column distribution, the
uniform-spacing chart-grid rejector — still apply.
Verified on a 7-row × 3-col archival catalog page that previously
extracted 98 chars (a 2-row fragment via the heuristic fallback); now
extracts 1754 chars with all rows + multi-line cells.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* extract_tables: try vector-grid detectors before text heuristic
extract_tables_in_regions_mem previously ran only the text-only
heuristic detector (tables::detect_tables) on the items inside each
region, discarding the rects and lines that extract_page_text_items
returned. That left the rect-backed and line-backed detectors
(detect_tables_from_rects, detect_tables_from_lines) unused by the
public region-scoped extraction path — they only ran through
detect_vector_grid_in_region_mem, which most callers don't use.
Keep the rects and lines, filter them to each region, and try in
order: rect detector → line detector → heuristic. Each candidate's
markdown is quality-gated by the existing needs_ocr checks
(is_garbage_text, is_cid_garbage, detect_encoding_issues,
looks_like_partial_table_ex); only the first clean output wins.
If all three produce empty or noisy output we still return
needs_ocr=true, matching prior behavior.
Effect on real prod-shape inputs from shadow logs:
Full-page ruled ledger, 6 cols x ~15 rows:
before: heuristic emits a 355-char two-row fragment
after: line detector emits the full 6520-char table
Multi-row key/value layout with paragraph values:
before: heuristic emits a 188-char header-only fragment
after: rect detector emits the full 1733-char table including
the multi-bullet description cell
Existing fixtures that already passed via the heuristic continue to
pass: the quality gate rejects partial vector-grid output and falls
through, so the heuristic still wins where it produced the cleaner
result.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Bump version from 1.8.9 to 1.8.10
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a wire-bordered table has headers centered/right-aligned in their
cells but data left-aligned, cluster_x_positions can both merge adjacent
data columns (when the data-to-data gap is below the clamped threshold)
and drop the header-only x-positions in its singleton-filter pass. The
cell-rect fallback then used text-cluster column edges and lost a column
or fragmented neighbor cells.
Prefer rect-derived column edges when the rect grid has 3+ columns and
every rect column holds multiple text items. The all-cols-populated
check protects against decorative or background rects (prose laid out
in a frame, cell-fill rects with extra borders) that would otherwise
split a logical column into spurious sub-columns. The existing
prose-in-frame, well-distributed-columns, and wireless-prose guards
still fire for the cases they were built for.
Bump napi version 1.8.7 → 1.8.8.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: lift detection on shaded-header + alt-row tables (#wired-grids)
Production telemetry on `wired_high_confidence`-classified table regions
showed `detect_vector_grid_in_region_mem` returning a usable grid only
~27% of the time, with the rest falling through to GLM-OCR. Three
surgical fixes target the dominant production shapes:
* Path-fill cell backgrounds: when the page has no `re` rects but draws
cell backgrounds via `m`/`l`/`h`/`f*` sequences, prefer the fill-derived
rects over the few section-level `W*` clip paths that previously won
the priority gate. Activated when fill rects outnumber clip rects ≥3×.
* Dedup-induced cluster splits: page-background rects could pose as
containers in the sub-rect dedup and evict a slightly smaller
table-frame rect, breaking adjacency between column-cell groups so each
column became its own cluster. Origin-anchored containers are now
disqualified from sub-rect dedup. A separate exact-duplicate pass
collapses the cell-padding/text-bg/cell-border triple emissions some
PDFs produce, preserving original order to avoid reshuffling table
output on multi-table pages.
* Prose-words rejection: the `cell-rect` fallback's whole-grid prose
threshold also rejected real tables that include a description column.
Now relaxed when content is well-distributed (≥75% of cols filled),
while keeping the original strictness for prose-in-a-frame layouts.
Two regression fixtures from the opendataloader-bench corpus, covering
the dominant production failure categories:
* `greencomp_competence.pdf` — 2-col shaded-header + plain-body glossary.
Mirrors production crops #1 (Contractions glossary) and #6 (BIO 350
course header).
* `upstage_key_functions.pdf` — 4-col shaded-header + alt-row backgrounds
+ merged left column. Mirrors production crops #2 (Parameter/Value
alt-row), #7 (Spanish XML schema), and #8 (Córdoba multi-row header).
Existing fixtures stay green (doc 51 wrapped-label, doc 128 forecast
six-cols, td9264 snapshot). 133 unit + integration tests pass; clippy
clean.
Bumps napi/package.json 1.8.4 → 1.8.5.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: tighten prose-in-frame rejection — fixes pdf-evals #30 regression
PR #76's shaded-header detection lift surfaced a regression on
accessory_building_permit_application_1 (TEDS 0.10 → 0.05): a
paragraph of legal text laid out in a 2-column justified block was
being admitted as a 10×2 fake table where every cell holds a
sentence fragment ("I agree to comply...", "I", "It is the property
owner's responsibility..."). Per pdf-evals PR #30 review, this is
the kind of regression production users will notice — the markdown
is structurally and semantically misleading.
Root cause: PR #76's prose-rejection only fires for `num_cols >= 4`,
so the 2-col prose-in-a-frame case slipped past it entirely. The new
fill-priority + dedup changes started producing rects for this layout
that 1.8.4 correctly ignored.
Fix: tighten the prose-in-frame check.
- Lower the column-count guard from `>= 4` to `>= 2`.
- Add a content-length signal as the primary discriminator: when the
prose-words trigger fires AND mean non-empty cell length exceeds
65 chars, reject regardless of column distribution.
The 65-char threshold cleanly separates observed cases:
accessory_building (prose-in-frame): mean 74 chars → REJECT
upstage_key_functions (real 4-col table): mean 53 → admit
greencomp_competence (real 2-col glossary): mean 20 → admit
accessory_building (real 5×3 form data): mean 10 → admit
The well-distributed-cols relaxation that PR #76 added stays —
"label / value / description / benefit" tables (#7, #8 from the
production crops) still pass, but only when their mean cell length
stays below the prose threshold.
New regression test `accessory_building_rejects_prose_in_frame` asserts
both that the real 5×3 form data table survives AND the 10×2 prose
block is rejected. Snapshot test `test_snapshot_td9264` updated to
match new output — old snapshot captured the same prose-in-frame bug
on regulatory text (paragraphs emitted as 3-col `||text||` fake-table
rows). New snapshot emits clean prose paragraphs, which is correct.
Verification:
- cargo test --all: 424 lib + 133 integration + 2 doc tests pass
- cargo fmt --check clean
- cargo clippy -- -D warnings clean (lib-level; pre-existing
test-level clippy issues on the wired-grids branch unaffected)
- Existing fixtures stay green: forecast_table_chart_six_cols (PR
#72), bits_pilani_* (PR #73), greencomp_competence_two_cols and
upstage_key_functions_four_cols (PR #76).
This branch is based on abimaelmartell/wired-grids so it includes
PR #76's commits plus this fix on top. Suggest merging this and
closing #76, OR rebasing #76 to incorporate this fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: drop early-dedup atom — caused broad TOC + matrix corruption
Bisected PR #76's 4 atoms against the SEC 10-K
0001104659-25-093871_183e44ac.pdf which appeared as a TEDS regression
in pdf-evals PR #30. Result:
atom | TOC | perf-graph | qualifications
---------------------------|-----|------------|---------------
fill-priority | ✓ | ✓ | ✓
early-dedup | ✗ | ✗ | ✗
page-bg disqualification | ✓ | ✓ | ✓
prose-relaxation | ✓ | ✓ | ✓
Early-dedup was the SOLE source of all three regressions on this doc.
Tried a more conservative variant (≥3 copies only — pair-duplicates
appear in legit multi-section layouts like 10-K dividers above + below
section headers); didn't fix the regression. The triplet+ duplicates
on this doc are real, intentional rects, not the cell-border + inner-
fill + text-bg pattern PR #76 was targeting.
Drop early-dedup. Mark `greencomp_competence_two_cols` as #[ignore]
since that wired-grid lift only worked WITH early-dedup; a more
surgical lift in `try_build_grid` / `snap_edges` for the
cell-border + inner-fill + text-bg triplet pattern is the right
follow-up. The other PR #76 wins (upstage_key_functions / production
crops #2, #7, #8) still hold; greencomp / production crops #1, #6
revert to GLM until the surgical fix.
Validation on the regression doc:
0001104659 TOC PART II markers: 4 (matches main, was 2 with PR#76)
0001104659 perf-graph data row: 2 (matches main, was 1)
0001104659 qualifications rows: 9 (matches main, was 6)
Validation on the prose-frame doc:
accessory_building fake-table: 0 (matches main, was 1 with PR#76)
accessory_building prose intact: 1 (matches main)
cargo test --all clean, cargo fmt --check clean, cargo clippy --lib
-- -D warnings clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three fixes to extract_tables_with_structure_auto_mem (added in
1.7.1) caught by external review:
1. multi_row_in_cell over-triggered on legitimate multi-line cells.
The previous threshold (item span > 1.3× either smallest cell or
tallest item height) fires on any cell with 2+ y-separated text
items — including rowspan>1 cells, wrapped descriptions, and
superscript/subscript runs. Replaced with two gates:
- skip cells whose declared rowspan > 1 (intentional multi-line)
- require an actual whitespace gap (>~half a line height)
between the bottom of one item and the top of the next, in
PDF-native y-coordinates. Same-line items with tall glyphs or
superscripts have negative or near-zero gap; truly separate
visual rows have gap ≈ leading − line-height.
FNBO regression test still passes; new test covers a rowspan=2
cell with two visible text lines and verifies no fallback fires.
2. Heuristic returning empty silently replaced TSR markdown with
"". The auto wrapper now keeps the TSR markdown when the
heuristic markdown is empty/whitespace and tags fallback_reason
with `_heuristic_empty` suffix (e.g.
`multi_row_in_cell_heuristic_empty`). Worst case we ship the
same wrong-but-non-empty TSR output we'd have shipped before
1.7.1; we never replace useful output with literally nothing.
3. One bad input blanked the whole batch. Errors from
detect_tsr_quality_issue or extract_tables_in_regions_mem now
stay scoped to the single input — that input falls through to
raw TSR markdown with a `_error` reason label so callers can
metric on it. Other inputs in the batch are unaffected.
3 new integration tests:
- test_auto_does_not_fire_on_legit_rowspan_cell
- test_auto_keeps_tsr_markdown_when_heuristic_returns_empty
- test_auto_isolates_per_input_failures
All 6 auto tests + full 123-test suite pass. FNBO local replay
still triggers fallback (phantom_empty_row signal in this run) and
emits correct Shawnee/BVP/Sonoma rows with correct census tracts.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds extract_tables_with_structure_auto_mem (Rust) /
extractTablesWithStructureAuto (napi). Returns
TableExtractionResult { markdown, fallback_reason } per input.
The wrapper runs the existing TSR-hybrid path then checks the
resulting cells for two known SLANet detection pathologies:
* phantom_empty_row: empty row sandwiched between non-empty rows
(cheap, cell-metadata only).
* multi_row_in_cell: re-reads PDF text items, flags any cell whose
contained items span >1.3× either the smallest cell height or
the tallest contained item height. Catches the FNBO failure mode
where a tall TSR cell absorbs two adjacent PDF rows.
When either fires, extract_tables_in_regions_mem runs over the same
crop bbox and its markdown replaces the TSR markdown.
fallback_reason carries the diagnostic label so callers can emit
metrics and watch each pathology independently.
Validated on FNBO branches PDF page 1 (Kansas region):
- TSR-only: merges Shawnee into BVP, wrong census tract on Sonoma.
- Auto fallback (phantom_empty_row): each row separate, correct tracts.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the row-merge pattern observed against FNBO branch list at the
College/Fairway boundary: when SLANet emits cells whose y-extents
overlap between consecutive rows, an item whose center fell in the
overlap region got pulled into BOTH cells, producing run-on cells
(e.g. "Kansas Kansas" + concatenated addresses).
Cause: stage 1's "for each cell, gather items inside" loop allowed an
item to match multiple cells. `normalize_cell_bands` reduces overlap
but is biased when cell-mean-center is offset from actual text baseline
(SLANet bboxes are typically taller than their text content), so the
midpoint-clamp can land on the wrong side of the row boundary, and
items at the boundary still match two cells.
Fix: invert the matching. For each PDF text item, find candidate cells
(those whose bbox satisfies tsr_region_contains_item — center inside
OR >=60% overlap on both axes), and assign to the cell whose CENTER
is geometrically closest. Build per-cell text from the assigned items.
Stage 2 (orphan recovery) is unchanged.
Exclusivity prevents item duplication across cells. The closest-center
rule disambiguates the cell-overlap case naturally without aggressive
band clamping. normalize_cell_bands stays — it tightens cells before
matching (smaller overlap → fewer ambiguous candidates) but is no
longer load-bearing for correctness of the overlap case.
Local replay against FNBO via api/scripts/local-tsr-replay.ts (which
exercises the full layout-pod → table-pod → pdf-inspector chain
in-process):
Pre-1.6.4 (deployed 1.6.3):
|LITH West|Illinois Illinois|11700 S. IL Route 47, Huntley IL...
|College|||0534.03|
|Fairway|Kansas Kansas|4650 College Blvd... 2828 Shawnee Mission...
Post-1.6.4 (this change):
|Huntley|Illinois|11700 S. IL Route 47, Huntley IL...|8711.15|
|LITH West|Illinois|4520 W Algonquin Rd, Lake in the Hills...
|College|Kansas|4650 College Blvd, Overland Park KS...|0532.01|
|Fairway|Kansas|2828 Shawnee Mission Pkwy, Fairway KS...|0500.00|
One row (Shawnee in the Kansas section) can still get dropped under
SLANet detection variance — the model occasionally under-detects rows
and emits N structure rows for N+1 PDF rows. The squeezed row's text
gets routed to the structurally-nearest existing cell. This is a
SLANet limitation, not addressable in pdf-inspector without
synthesizing rows from PDF text geometry; deferred.
Tests: 416 lib + 120 integration + 2 doctests pass. clippy + fmt clean.
Bump @firecrawl/pdf-inspector to 1.6.4 (patch — refines stage 1's
matching strategy; no API changes).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the residual run-on-cell pattern observed on FNBO branch list
after 1.6.2 deployed:
|Shawnee Blue Valley Parkway|Kansas|6301 Pflumm, ...|0523.04|
|Sonoma Plaza|Kansas Kansas|<addr1> <addr2>|0531.05|
|Mitchell Woonsocket|South Dakota|<addr1>|9628.01|
Cause: when col-N cells across multiple consecutive rows are y-shifted
the same way (a local SLANet drift), the stage 2 orphan pass sees
multiple orphans qualifying for the same nearest empty cell. The cell
gets all of them appended in order, producing "RowA-text RowB-text".
Fix: track the y-coordinate of the first orphan that lands in each
cell. Subsequent orphans only join that cell if their y is within
half-a-row-height of the first orphan's y (same line). Cross-line
orphans skip that cell and look for the next-nearest empty cell on
their own line.
Same-line slack preserves multi-token branch names like
"Blue Valley Parkway" (3 PDF text items at the same y) — all three
stack into the same cell. Cross-row stacking is what gets rejected.
Two new tests:
- stage2_rejects_cross_line_stacking_into_same_cell
Two orphans on different rows, both equidistant to the same empty
cell. First wins; second routes to its own row's cell.
- stage2_allows_same_line_orphans_to_stack_into_one_cell
Three same-line orphans (multi-token branch name) all land in the
same empty cell, joined by spaces.
The existing four stage-2 tests + the cell-bleed regression test from
PR #62 + #63 all pass: 416 lib + 115 integration + 2 doctests.
clippy + fmt clean.
Bump @firecrawl/pdf-inspector to 1.6.3 (patch — refines 1.6.2; no API
changes).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #62 (1.6.1) closed the cell-bleed regression by clamping SLANet's
loose cell bboxes into non-overlapping row/column bands and tightening
text membership to center-containment OR >=60% overlap. That worked,
but exposed the opposite failure: legitimate native PDF text whose
center fell just outside the *clamped* cell bbox now had nowhere to go.
Two distinct failure modes observed against the FNBO branch-list PDF
after 1.6.1 deployed:
Symptom A — header text positioned at the LEFT of a column whose band
was derived from data-cell centers farther right. Header "Address"
PDF text at x=331..375 fell outside the clamped col band starting
at x=410. Strict membership rejected it (0% x-overlap, center
outside).
Symptom B — local SLANet row drift in col 0 over a 5-row stretch.
Cell bboxes sat just above the actual branch-name text items
(x-overlap 100% but y-overlap ~30-43%, below the 60% threshold).
Both share one root: post-normalization bboxes are too tight, and the
strict rule has no escape valve for legitimate edge text.
Fix: add a stage-2 orphan-recovery pass after the strict fill. Items
that NO cell claimed in stage 1 get re-assigned to their nearest
*empty* cell, distance-capped by `(median_col_width, median_row_height)`
so a far-orphan figure title can't get pulled into a faraway empty
cell. Stage 2 only fills empties — never overwrites stage 1 — so the
cell-bleed case PR #62 closed cannot regress.
Three new lib tests cover the bug shapes:
- stage2_recovers_left_aligned_header_text_outside_data_band
(Symptom A: header text left-of-band, data cells already filled,
stage 2 fills only the header)
- stage2_recovers_y_shifted_col0_in_consecutive_rows
(Symptom B: 3 col-0 cells shifted vs text, all 3 recovered)
- stage2_does_not_overwrite_filled_cells_or_admit_far_orphans
(cap rejects far figure titles; pre-filled cells untouched)
Plus tests for the cap-derivation helper:
- tsr_assignment_caps_uses_median_geometry
- tsr_assignment_caps_floor_protects_degenerate_input
The existing dense-overlapping-rows regression test (added in #62) still
passes, confirming no regression on the cell-bleed case.
Test results: 414 lib + 115 integration + 2 doctests pass. clippy + fmt
clean.
Bump @firecrawl/pdf-inspector to 1.6.2 (patch — fixes a regression
introduced by 1.6.1; no API changes).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: TSR-aware table extraction (extract_tables_with_structure_mem)
New public function that consumes raw structure-recovery output (HTML
structure tokens + per-cell bboxes from a model like SLANet) and assembles
markdown tables by pulling cell text from the native PDF — no OCR, no
geometry inference.
Why: the existing extract_tables_in_regions_mem infers grid geometry from
text positions only and can't distinguish merged cells from multiple narrow
columns. Pairing structure recovery from a layout/TSR model with native
PDF text gets perfect text quality with proper row/col/span structure.
- New module src/tables/structured.rs: token state machine, polygon→AABB,
crop-px→page-pt, rowspan/colspan-aware cell layout, markdown emitter.
Accepts both 4-element rects and 8-element 4-corner polygons.
- New public extract_tables_with_structure_mem in src/lib.rs that reuses
extract_page_text_items, region_overlaps_item, and the shared region
text-collection helper. No existing public function modified.
- napi binding extractTablesWithStructure mirroring the existing
extractTablesInRegions shape (f64 in JS → f32 internally).
- 14 unit tests + 5 integration tests, including a real-PDF gold-standard
match against bits_pilani_feedback.pdf.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TSR follow-ups: header-aware separator, cells API, v1.6.0
- cells_to_markdown emits the separator after the LAST row that contains
is_header=true cells, falling back to "after row 0" when no header is
flagged. Multi-row theads now render correctly. Three new unit tests
cover: multi-row header, header not on row 0, no headers (fallback).
- New public extract_tables_with_structure_cells_mem returning
Vec<Vec<StructuredCell>> so callers can drive their own rendering or
debug overlays without re-doing the parse + extraction. The markdown
variant now wraps it. The previously-unused page_pt_bbox field is
surfaced through this API.
- New napi binding extractTablesWithStructureCells + StructuredCellJs.
- Bump @firecrawl/pdf-inspector to 1.6.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix TSR cell text assignment for overlapping bboxes
Made-with: Cursor
* bump npm package version to 1.6.1
Made-with: Cursor
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: TSR-aware table extraction (extract_tables_with_structure_mem)
New public function that consumes raw structure-recovery output (HTML
structure tokens + per-cell bboxes from a model like SLANet) and assembles
markdown tables by pulling cell text from the native PDF — no OCR, no
geometry inference.
Why: the existing extract_tables_in_regions_mem infers grid geometry from
text positions only and can't distinguish merged cells from multiple narrow
columns. Pairing structure recovery from a layout/TSR model with native
PDF text gets perfect text quality with proper row/col/span structure.
- New module src/tables/structured.rs: token state machine, polygon→AABB,
crop-px→page-pt, rowspan/colspan-aware cell layout, markdown emitter.
Accepts both 4-element rects and 8-element 4-corner polygons.
- New public extract_tables_with_structure_mem in src/lib.rs that reuses
extract_page_text_items, region_overlaps_item, and the shared region
text-collection helper. No existing public function modified.
- napi binding extractTablesWithStructure mirroring the existing
extractTablesInRegions shape (f64 in JS → f32 internally).
- 14 unit tests + 5 integration tests, including a real-PDF gold-standard
match against bits_pilani_feedback.pdf.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TSR follow-ups: header-aware separator, cells API, v1.6.0
- cells_to_markdown emits the separator after the LAST row that contains
is_header=true cells, falling back to "after row 0" when no header is
flagged. Multi-row theads now render correctly. Three new unit tests
cover: multi-row header, header not on row 0, no headers (fallback).
- New public extract_tables_with_structure_cells_mem returning
Vec<Vec<StructuredCell>> so callers can drive their own rendering or
debug overlays without re-doing the parse + extraction. The markdown
variant now wraps it. The previously-unused page_pt_bbox field is
surfaced through this API.
- New napi binding extractTablesWithStructureCells + StructuredCellJs.
- Bump @firecrawl/pdf-inspector to 1.6.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: expose per-page markdown extraction to Python and Node (#49)
Implements the feature requested in issue #49: a list-of-pages markdown output
from the Python API. Matching the existing project pattern, the feature lives
in the Rust core and is surfaced through every binding.
- Rust core: `extract_pages_markdown` (path) and `extract_pages_markdown_mem`
(bytes) now take `Option<&[u32]>` — `None` returns every page in document
order; a slice restricts and preserves caller order.
- Python: new `extract_pages_markdown(path, pages=None)` and
`extract_pages_markdown_bytes(data, pages=None)` functions plus
`PageMarkdown` / `PagesExtractionResult` classes; stub file updated.
- Node: `extractPagesMarkdown(buffer, pages?)` — `pages` is now optional.
- Tests: 2 new Rust integration tests, 9 new Python tests, 2 new Node
assertions. All 372 unit + 107 integration + 53 Python tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version from 1.3.0 to 1.4.0
Minor bump for the new per-page markdown extraction API exposed through
the Python and Node bindings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds x86_64-pc-windows-msvc target to the napi package and CI build
matrix so the published @firecrawl/pdf-inspector npm package ships
native Windows binaries alongside Linux and macOS.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat: add CLI bin to npm package
Installing `firecrawl-pdf-inspector` now provides a `pdf-inspector` CLI command.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump npm package version to 0.8.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump npm package version to 1.0.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(detector): correct false flags for CID-encoded text and supplementary fonts (0.7.4)
The page classifier was over-aggressively flagging Mixed-PDF pages as
needing OCR in three distinct cases. Each is fixed at the root in
analyze_page_content / page_has_identity_h_no_tounicode / the
looks_like_scan check.
1. has_vector_text false positives on dense layouts
path_ops > text_ops*200 fired on pages with decorative paths
(column borders, dividers) alongside real selectable text. Added
a unique_alphanum_chars < 30 guard: real outlined-text pages have
very few unique alphanum chars (each glyph is a path), while
pages with real text + decorations have many.
2. Identity-H without ToUnicode flagged whole pages on supplementary fonts
page_has_identity_h_no_tounicode would flag a page if any single
Type0 font lacked ToUnicode and had no fallback CMap, even when
the page's actual text came from other decodable fonts (Type1
with ToUnicode, etc.). Rewrote to track both undecodable
Identity-H fonts AND other decodable fonts, only flagging when
no decodable text font is present.
3. CID-encoded text with ToUnicode misclassified as scan
looks_like_scan checked unique_alphanum_chars < 10 on raw string
operand bytes. CID-encoded fonts (Type0 with ToUnicode) emit
2-byte CID values that aren't ASCII alphanum, so the metric is
blind to them even when the text is fully decodable. Added a
has_decodable_text_fonts signal: when a page has decodable fonts
AND >= 10 text ops, the low alphanum count is treated as a CID
encoding artifact rather than evidence of a scan.
Validated against a broad PDF corpus:
- 6 known false-positive pages now correctly classified as text
- 22 previously-missed scan pages (cover/blank/photo) now correctly
flagged for OCR
- 0 regressions on truly-scanned PDFs (61/61 pages stay flagged)
- All 437 existing tests pass; clippy clean
Bumps NAPI package to 0.7.4.
* test(detector): add unit tests for the three classifier fixes
Adds 10 unit tests covering the heuristic changes:
- has_vector_text alphanum guard
- real text + decorative paths → not flagged
- true outlined glyphs (low alphanum) → still flagged
- page_has_identity_h_no_tounicode supplementary-font handling
- undecodable Identity-H + decodable Type1 → not flagged (new)
- undecodable Identity-H alone → still flagged (regression)
- page_has_decodable_text_fonts (new helper)
- Type1 → true
- Type0 with ToUnicode → true
- undecodable Identity-H only → false
- looks_like_scan with has_decodable_text_fonts override
- CID-encoded decodable text → not flagged as scan
- same metrics with no decodable fonts → still flagged
- decodable fonts but text_ops < 10 (page-number overlay) → still flagged
* fix(detector): make decodable-font checks usage-based and XObject-aware
Addresses two reviewer concerns on the previous heuristic fix:
P1 — resource-based check could create an inverse bug
page_has_identity_h_no_tounicode and page_has_decodable_text_fonts
iterated all fonts in the page Resources dict, including unused fonts.
A page whose actual text was rendered exclusively in an undecodable
Identity-H font but whose Resources also listed an unused decodable
Type1 would be wrongly unflagged.
Fix: parse Tf operator operands during content stream scanning to
collect the set of font names actually referenced. The font checks
now filter to only USED fonts via a new used_fonts_have_*
family of functions operating on (used_font_names, font_map).
P2 — checks didn't follow text into Form XObjects
analyze_page_content correctly recurses through Form XObjects via
scan_xobjects_in_resources, but the font checks only looked at the
page's top-level Resources/Font. Pages that render text through Form
XObjects (corporate templates, header/footer overlays) had their
XObject font resources missed entirely.
Fix: scan_xobjects_in_resources now propagates the used_font_names
set AND collects fonts from each Form XObject's own Resources into
the shared font_map. The usage-based check sees the full picture:
page-level fonts + every nested XObject's fonts, intersected with
fonts actually referenced by Tf operators anywhere in the content.
Implementation:
- New extract_font_name_before_tf helper (parses /Name immediately
preceding Tf).
- New FontInfo struct caches font properties per-name.
- New collect_fonts_from_resource_dict + new used_fonts_have_*
functions are pure filters over (used_names, font_map).
- analyze_page_content threads used_font_names + font_map through
page content scan and XObject recursion, then runs the new checks.
- Old resource-based functions kept as #[cfg(test)] for the existing
unit-test interface.
- Phase 3 uncached-page loop now goes through analyze_page_content
so it also gets the usage-based + XObject-aware behavior.
Tests added (8):
- extract_font_name_before_tf basic + long-name parsing
- scan_content_for_text_operators collects used font names
- P1 — unused decodable font in Resources doesn't save a page
whose used font is undecodable
- P1 — both fonts used → decodable font correctly prevents flag
- P2 — decodable font inside Form XObject correctly unflags
- P2 — undecodable font only in XObject still flags even with
unused decodable font at page level
- P2 — has_decodable_text_fonts populated from XObject fonts
Validation:
- 349 lib + 104 integration + 2 doc tests pass (was 341)
- cargo clippy --lib --bin detect-pdf -- -D warnings: clean
- External eval: 9/9 PDFs pass, 6/6 false positives resolved,
0 regressions, 61/61 scanned pages still correctly flagged
- No eval delta — confirms previous fix wasn't relying on the
resource-based bug for any of the eval PDFs
* fix(detector): scope font lookups by ObjectId + handle indirect Form Resources
Addresses two more reviewer findings on the previous decodable-font commit.
P1 — Resource-name scoping bug
The previous fix keyed used_font_names and font_map by raw resource
names like b"F1". PDF resource names are scoped to each resource
dictionary: a Form XObject can legally define its own /F1 that points
to a completely different font from the page's /F1. Because
collect_fonts_from_resource_dict skipped duplicates with
`if font_map.contains_key(name)`, the first definition won and later
Tf /F1 usages in different scopes resolved against the wrong font.
This could reintroduce both the undecodable-Identity-H false flag
and the decodable-CID false unflag depending on which side of the
collision happened to be inserted first.
Fix: switch the lookup mechanism from font names to font ObjectIds.
- font_map: HashMap<ObjectId, FontInfo> (was Vec<u8> keys)
- used_font_ids: HashSet<ObjectId> (was Vec<u8> names)
- new resolve_font_names_to_ids() runs immediately after each
content scan, against the resource dict in scope, to translate
the per-scope name set into ObjectIds.
Each Form XObject's content stream now resolves /F1 against THAT
XObject's own Resources, so name collisions are impossible by design.
Inline (no-ID) font dicts are skipped — extremely rare in practice
and have no stable key.
P2 — Indirect Form /Resources skipped
scan_xobjects_in_resources used `.as_dict()` on the Form's /Resources
entry, which returns None for indirect references. PDFs frequently
store /Resources as `X 0 R`, in which case font collection and
recursion were both skipped — even though the Tf usages inside the
XObject content had already been recorded.
Fix: handle Object::Reference(r) in addition to Object::Dictionary(d)
by resolving via doc.get_dictionary. Audited the rest of the file —
the other /Resources access points (analyze_page_images,
collect_images_from_resources) already handled both cases.
Tests added (4):
- P1 same-name-different-font (page undecodable, XObject decodable):
must NOT flag — XObject's text is decodable in its own scope.
- P1 inverse (page decodable, XObject undecodable, content uses
XObject /F1): MUST flag — undecodable text exists in real scope.
- P2 indirect Form /Resources: font discovery must still work when
/Resources is a `X 0 R` reference rather than inline.
- Combined regression: indirect Resources + name collision.
Validation:
- cargo test --release: 459 tests pass (353 lib + 104 integration + 2 doc)
- cargo clippy --lib --bin detect-pdf -- -D warnings: clean
- external eval (9 PDFs): 9/9 pass, 6/6 false positives resolved,
0 regressions, 61/61 truly-scanned pages still flagged
The behavior on the eval set is identical — confirms the correctness
fix isn't masking any change in classifier outcomes.
* fix(detector): respect resource shadowing when resolving page-content fonts
The previous ObjectId-based fix correctly scoped Form XObject fonts
but still violated PDF resource inheritance for page content. When a
page overrides /F1 from a parent /Pages node (different font dict for
the same name), get_page_resources returns the page's own /Resources
plus all ancestor /Resources dicts. The old code called
resolve_font_names_to_ids on each one and added every match to
used_font_ids — both font ObjectIds ended up in the used set even
though only the page's /F1 is actually visible to that page's content.
Per ISO 32000-1 §7.7.3.4, resource names are inherited with
shadowing semantics: the most-specific (deepest, closest to the page)
definition wins.
Fix:
- New lookup_font_id helper resolves a single name in a single dict.
- New resolve_with_shadowing iterates names, checking the page's own
/Resources first, then walking ancestors in most-specific-first
order (which is the order lopdf's get_page_resources returns).
First hit wins via a labeled `continue 'name` — subsequent
ancestors are skipped for that name.
- analyze_page_content's flat resolution loop replaced with one call
to resolve_with_shadowing.
Audit:
- XObject path is correct: each Form XObject already resolves names
against its OWN /Resources (XObjects don't inherit from page tree).
- font_map population is correct: keyed by ObjectId, so collecting
from all dicts builds the full available-fonts catalog. The bug
was only in the used-set resolution.
- Confirmed lopdf returns ancestors in most-specific-first order
(page → parent → grandparent → root), matching the shadowing
direction used here.
Tests added (3):
- page /F1 undecodable shadows parent's decodable /F1 → MUST flag
- page /F1 decodable shadows parent's undecodable /F1 → MUST NOT flag
- no override: page inherits parent's decodable /F1 → MUST NOT flag
Validation:
- cargo test --release: 462 tests pass (356 lib + 104 integration + 2 doc)
- cargo clippy --lib --bin detect-pdf -- -D warnings: clean
- external eval: 9/9, 0 regressions, 6/6 false positives resolved,
61/61 scanned pages still correctly flagged
The looks_like_scan check incorrectly used OR logic, causing any single
condition (image_count <= 1, text_ops < 50, alphanum < 10) to flag a page
as a scan. A real scan has ALL three: single full-page image AND low text
AND low alphanum. Text pages with one figure were falsely flagged for OCR.
Bump napi to 0.7.3.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: reduce false OCR recommendations for text PDFs with figure images
Two fixes in the detector:
1. Fix Tf operator parsing: some PDFs concatenate Tf directly with the
next operator (e.g. "25 Tf[<01>...") without whitespace. The scanner
now accepts [, (, <, / as valid followers, fixing font_changes being
reported as 0.
2. Distinguish text-with-figures from scanned-with-OCR: pages with
multiple images (image_count > 1) and strong text signals (text_ops
>= 50, alphanum >= 10) are recognized as text pages with figures,
not scanned templates. Scanned PDFs have exactly 1 full-page image.
This prevents academic papers, reports with charts, and similar PDFs
from being incorrectly classified as Mixed/OCR-needed when their text
is perfectly extractable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump napi version to 0.7.2
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove template image influence from page classification
Template images (large background/figure images) no longer affect
pages_needing_ocr. In the region-based pipeline, text regions are
extracted independently from image regions, and per-region needs_ocr
quality checks handle scanned-with-OCR garbage text.
Also makes the invisible text retry (for OCR text layers) trigger on
text quality rather than PDF type, so it works regardless of
classification.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Revert "fix: remove template image influence from page classification"
This reverts commit 100cbe5453.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: improve heuristic table detection for numeric columns and multi-line headers
Two fixes for tables that have clean extractable text but fail heuristic
structure detection:
1. Numeric column merge pass (grid.rs): After initial X-position
clustering, adjacent clusters are merged when one is sparse (header
text) and the other is dense with >50% numeric items (data column).
Multi-line wrapped headers often land slightly offset from their
data column — the merge closes gaps within 1.5× the clustering
threshold. New is_numeric_text() helper matches decimals, percentages,
negative numbers, and comma-separated thousands.
2. Duplicate-header skip (detect_heuristic.rs): Spanning super-headers
like "First Degree | First Degree | Higher Degree" contain duplicate
cells that trigger looks_like_partial_table_ex rejection. Now skips
rows with duplicate cells when a better header candidate exists
within the next 3 rows (higher fill ratio or numeric cells).
Tested on BITS Pilani university report (430 pages, 314 table pages).
Page 4 (multi-line header + numeric data) previously returned
needs_ocr=true; now correctly detects the table structure.
Eval: 197 PDFs, zero regressions, all 104+ tests pass, zero clippy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* bump version to 0.7.1
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Combine per-page markdown extraction with layout classification into a
single parse. extractPagesMarkdown now returns PagesExtractionResult with
pages_with_tables, pages_with_columns, pages_needing_ocr, and is_complex
alongside the per-page markdown — eliminating redundant PDF parses for
callers that need both.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* add extract_pages_markdown_mem for per-page markdown extraction
Enables hybrid OCR pipelines to skip GPU render+layout for simple text
pages by providing per-page markdown with needs_ocr flags. Font stats
are computed document-wide for consistent header detection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* bump napi package version to 0.6.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace stringly-typed pdf_type and item_type fields with
#[napi(string_enum)] enums for proper TypeScript type checking.
Add link_url field to TextItem instead of encoding URL in the
item_type string.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
