Releases fix(regions) #351 — invisible (Tr 3) OCR text layers served from
the region extractor instead of falling back to GPU OCR.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Tagged PDFs carry a structure tree with real heading roles (H1..H6), and
the core already parses it (structure_tree::StructTree) and threads MCIDs
onto TextItem — but neither surfaced through the bindings.
- Expose TextItem.mcid (Option<i64>) through the napi and pyo3 bindings,
matching the core field added with the marked-content extractor.
- Add StructRole::name(), the inverse of from_name, so roles have a
stable string form.
- Add extract_structure_elements / extract_structure_elements_mem to the
core: one (page, mcid, role) entry per marked-content reference, sorted
by (page, mcid), empty for untagged PDFs. Pages are 1-indexed to match
TextItem.page, so results join directly against
extract_text_with_positions output.
- Bind it as extractStructureElements (napi) and
extract_structure_elements / extract_structure_elements_bytes (pyo3),
with type-stub updates in pdf_inspector.pyi.
- Cover the join in Rust integration tests, napi test.mjs, and pytest,
using the existing firecrawl_docs_tagged.pdf fixture (tagged) and
thermo-freon12.pdf (untagged).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(napi): add processPdfAsync, classifyPdfAsync, extractPagesMarkdownAsync
The Node bindings are synchronous, so every call parses on the event
loop thread — up to hundreds of milliseconds of dead loop per document
in a server. Add additive AsyncTask-based variants that run the same
shared implementations on the libuv thread pool and return promises.
The existing synchronous exports keep their names, signatures, and
behaviour; each sync/async pair shares one implementation. Panics in
compute() are caught and surfaced as rejections, matching the sync
error contract.
Closes#336
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(napi): read async task buffers in place instead of copying
Review feedback on #337: buffer.to_vec() copied the whole PDF on the
event loop before the task was queued, so large inputs still stalled
the loop and doubled peak memory. The tasks now hold the napi Buffer
itself — its ref pins the JS allocation for the task's lifetime and
the backing store is stable, so compute() reads it directly from the
worker thread. Callers must not mutate the buffer until the promise
settles (same contract as Node's async fs APIs); documented on each
export and in the README.
The suggested removal of ts_return_type was checked and rejected:
without it napi-rs generates Promise<unknown> for AsyncTask returns.
A comment now records that finding.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(napi): copy async task input on the JS thread for soundness
Review feedback on #337: holding the napi Buffer and reading it from
the libuv worker was unsound. Buffer derefs straight to the JS-side
allocation, so a caller mutating it before the promise settled would
race the worker's reads — undefined behavior, not a recoverable error,
and the documented don't-mutate contract was unenforceable. Deferring
the copy to compute() would not help: any off-thread read races the
same way. The JS thread is the only race-free place to take the copy,
because JS is single-threaded and nothing can mutate the buffer during
the synchronous part of the call.
Revert to an owned Vec<u8> copied at call time. The cost is one memcpy,
negligible next to the parse the async variants exist to unblock. Docs
now state the buffer may be reused or mutated immediately, and a test
locks in the copy semantics by mutating the input while a parse is in
flight.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
Adds x86_64-unknown-linux-musl, aarch64-unknown-linux-gnu, and
aarch64-unknown-linux-musl to the napi build targets so
@firecrawl/pdf-inspector works on Alpine and ARM64 Linux deployments.
- gnu arm64 cross-compiles with --use-napi-cross (old-glibc sysroot),
musl targets with -x (zig + cargo-zigbuild), per the napi-rs template
- new platform packages carry npm libc metadata (glibc/musl)
- smoke-test job runs napi/test.mjs on all six targets before publish
(Alpine containers for musl, ubuntu-24.04-arm runners for ARM64)
- bump to 1.12.0 to trigger publishing of all platform packages
Closes#216
Concise Features list and the opendataloader-bench comparison table on
each registry readme, adapted per ecosystem. Bump all three versions
(crate 0.1.6, python 0.2.5, npm 1.11.1) to republish the pages.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(npm): split platform binaries into optionalDependencies (1.11.0)
The single package bundled all three .node binaries (17.6 MB unpacked)
so every install downloaded every platform. Publish one package per
platform (@firecrawl/pdf-inspector-{linux-x64-gnu,darwin-arm64,
win32-x64-msvc}) holding just its binary; the napi-generated loader
already falls back to exactly these names. Main package drops *.node
from files (8.5 kB tarball) and pins the platform packages as
optionalDependencies, re-stamped to the exact version at publish time.
Publish workflow gains a workflow_dispatch fallback and per-package
already-published checks so partial releases can be retried.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(npm): document Windows support and platform packages; drop stale napi.package.name
Review follow-ups: the README claimed only linux-x64 and macOS ARM64
despite the win32-x64-msvc binary shipping, and napi.package.name
(@firecrawl/pdf-inspector-js) contradicts the real platform package
prefix — the loader and workflow derive it from the root package name.
Verified the generated loader is unchanged without the config.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Ships #145: exclusive item->region assignment in extract_text_in_regions
(overlapping layout regions no longer double-extract shared items —
duplicated lines on 21% of a 2,078-doc bench corpus, with occasional
content loss when downstream dedup kept the wrong variant).
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
New in this release (#125): isStrikeout on TextItem (geometric
detection sharing the underline rules pipeline), descriptor/embedded-
font bold+italic recall for subset fonts (FontDescriptor flags,
ttf-parser OS/2+post, bare-CFF Name INDEX), quote-operator advance
width, Ts text-rise handling, ActualText rise/position fixes, and a
document-scoped font style cache.
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(extractor): descriptor/embedded-font style flags + geometric strikeout detection
Two style-recall gaps, both invisible to the existing name-based
heuristics:
1. Subset fonts with opaque BaseFont names ("Tc1", "AAAAAB+Amplitude")
defeat is_italic_font/is_bold_font. New descriptor_style_flags reads
the FontDescriptor (ItalicAngle beyond 4 degrees, Flags bit 7 Italic,
bit 19 ForceBold) and, when the descriptor claims upright, falls back
to the embedded font file: ttf-parser's OS/2 fsSelection + post
italicAngle for sfnt fonts, and the CFF Name INDEX PostScript name
for bare-CFF FontFile3 (descriptor rewritten to ItalicAngle 0 while
embedding "Amplitude-LightItalic" was observed in the wild).
ORed into is_bold/is_italic at item creation (content streams and
form XObjects).
2. No strikeout signal existed. New is_strikeout on TextItem, detected
in the same pass as underline: same rules pipeline (stroked lines /
thin filled rects, table-ruling suppression), different vertical
window — a rule crossing the glyphs at 12-55% of the em above the
baseline instead of sitting at it. Exposed through napi and python
bindings and pdf2md --items-json.
Verified on public ParseBench corpus docs: previously-missed italic
council titles and bold CJK itinerary headings now flagged (render-
checked); 24/508 docs gain flags, none lose any; 35 strikeout items
detected corpus-wide, disjoint from underline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): quote-op advance width, Ts text rise, doc-level font style cache (PR #125 review)
Address three valid findings from review:
- The ' (move-to-next-line-and-show-text) operator emitted zero-width
items and never advanced the text matrix, so geometric underline/
strikeout detection (which requires width > 0) could never mark its
text, and following show ops overlapped it. Reuse Tj's advance-width
computation and matrix advance.
- Ts (text rise) was dropped entirely: raised/lowered runs kept the
unshifted baseline, so rules drawn at the risen glyph position missed
the strike/underline windows. Track rise in the text state (saved and
restored with q/Q) and shift the rendering position through the text
matrix's y column; advances stay on the unshifted matrix per spec.
- descriptor_style_flags re-decompressed and re-parsed the same embedded
font program on every page whenever the descriptor left a style flag
unset (the common case). Add a document-scoped FontStyleCache keyed by
the FontFile2/FontFile3 object id, threaded through page and form
extraction alongside the existing CMapDecisionCache.
The fourth finding (Form XObject rules never reach geometric detection)
is real but pre-existing for underline and needs the form walker to grow
path/paint tracking plus a new return type; deferred as a follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): ActualText items render at their glyphs' text rise (PR #125 review)
The EMC-built ActualText item used the captured text matrix without the
rise adjustment the ordinary Tj/TJ/' emission sites apply, so a tagged
run shown with Ts landed on the unshifted baseline — off the strikeout/
underline windows and inconsistent with untagged runs. The rise is
captured together with the first-glyph matrix (and at BDC for the
entry-position fallback): the item must render at the rise of its
GLYPHS, not whatever rise is set by EMC time.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): capture ActualText glyph position after the quote op's line move (PR #125 review)
The `'` handler skipped the entire suppressed-extraction block, so a
tagged span whose show op is `'` never captured its glyph matrix/rise —
the EMC item fell back to the BDC-entry matrix, which sits on the
PREVIOUS line (the `'` line move happens after BDC) with no rise. The
capture now happens right after the line move, matching the Tj/TJ
paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): style-boundary gate on subscript merge + strikeout suppression coverage (PR #125 review)
merge_subscript_items absorbed a script digit into its parent
regardless of underline/strikeout flags — dropping the digit's own mark
or widening the parent's over it. The merged item carries one flag, so
differing marks now break the merge, mirroring merge_text_items'
style-boundary rule (pre-existing for underline as well).
Also extends the table-suppression test to assert is_strikeout is
cleared alongside is_underline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(lib): detect substitution-cipher garbled text from broken ToUnicode CMaps
ParseBench text_simple__att10k.pdf (issue #118) ships Type0/Identity-H
fonts whose ToUnicode CMaps are authored garbled: every bfrange maps with
a wrong constant delta, so text extracts as pure-ASCII ciphertext
("Certificate" -> "8VceZWZTReV"). The embedded subset font has no cmap
table and no glyph names, so no decode source can recover the real text
(poppler and mupdf emit the same ciphertext). The only correct behavior
is to flag the page for OCR instead of serving the garbage silently --
but the text is 100% printable ASCII with word-like tokens, so it slipped
past is_garbage_text and detect_encoding_issues.
Add CipherGarbleStats, a letter-statistics discriminator that flags a
Latin-dominant sample (>=200 ASCII letters) when vowels are starved
(<=30% of letters) AND either:
- lowercase->uppercase transitions inside words exceed 10% of letter
bigrams (a shifted lowercase alphabet straddles the ASCII uppercase
block), or
- the letter histogram's cosine similarity against English letter
frequencies drops below 0.60 (catches shifts that stay within case
blocks).
Wired into analyze_text_quality (per-page, item-level) and
detect_encoding_issues (markdown-level), so extract_pages_markdown
reports needs_ocr + suspected_garbled_text and suppresses the garbage.
Thresholds validated against the 380-document pdf-evals snapshot corpus
(Swedish, Finnish, Turkish, German, romaji, schematics, all-caps and
camelCase-heavy docs): zero false positives, and byte-identical eval
output vs main. Garbled page measures vowel ratio 0.245 / case-shift
rate 0.225 / cosine 0.532; closest legitimate document on each axis is
0.264 / 0.021 / 0.801.
Fixes#118
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump pdf-inspector to 0.1.4, npm package to 1.9.11
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lib): exempt uniform-case structured content from cipher detection
Address PR review (cubic P2): the frequency branch (english_cosine < 0.60)
fired on any Latin-dominant, low-vowel letter distribution unlike English,
so non-linguistic ASCII — DNA/protein sequences, ticker symbols, hex dumps —
could be suppressed and routed to OCR despite not being garbled. Measured:
DNA cosine 0.428 / vowel ratio 0.260, protein 0.738, tickers 0.747, hex
0.549 — all would have flagged.
Add a mixed-case guard to looks_garbled: garbled English is a permutation of
natural language and carries sentence capitalization (block-straddling shifts
invert the ratio — att10k is 60% uppercase; in-case Caesar shifts preserve it
at ~3%), so both keep some of each case. The exempted structured content is
uniform case (all upper or all lower). Requiring the minority case to be >=1%
of ASCII letters exempts single-case sequences while preserving both garble
signals, including the in-case-shift scenario the frequency branch exists for.
Strictly tightens the detector: it can only remove flags, so the eval corpus
stays at zero false positives (verified byte-identical to a baseline main
binary across all 185 PDFs) and att10k remains flagged. Adds regression tests
for DNA, protein, tickers, and an in-case Caesar shift.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lib): make cipher detection case-agnostic via sorted-histogram shape
Address PR review follow-up: the mixed-case guard from the previous commit
returned before the vowel/frequency checks, creating a blind spot — a
uniform-case (all-lower or all-upper) substitution cipher is a plausible
broken-CMap output and would bypass OCR entirely.
Replace the case proxy with the actual invariant. A substitution cipher is
a bijection over a real language's alphabet, so it preserves the frequency
SHAPE (the sorted histogram) while scrambling letter POSITIONS (the unsorted
histogram). Signal 2 now flags when english_cosine < 0.60 (positions unlike
English) AND english_shape_cosine >= 0.90 (profile is still English-shaped).
This is independent of case, so it catches all-lower, all-upper, and
case-straddling shifts alike.
The exempted structured content fails one half: DNA/hex dumps have too steep
a profile (shape cosine 0.74 / 0.81 < 0.90), while protein sequences, ticker
symbols and base64 are not sufficiently unlike English in position (unsorted
cosine 0.74 / 0.75 / 0.77 >= 0.60). All stay out of OCR.
Still strictly corpus-safe: every real Latin document scores unsorted cosine
>= 0.70 (min 0.80), far above the 0.60 gate, so none can reach Signal 2.
Re-verified byte-identical to a baseline main binary across all 185 eval
PDFs; att10k remains flagged. Drops the now-unused case counters and adds
all-lowercase / all-uppercase shifted-prose regression tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: source Python package version from Cargo.toml via maturin
Address PR review (cubic P2): pyproject.toml pinned version = "0.1.0",
which overrides Cargo.toml, so a maturin build produced a 0.1.0 Python
artifact regardless of the crate version (it had drifted since the PyO3
bindings were added). Switch to dynamic = ["version"] so maturin sources
the version from Cargo.toml [package] version and the two can no longer
diverge. No workflow auto-publishes the Python package, so this is metadata
hygiene rather than a release-path fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(markdown): underline emission, Unicode scripts, style-preserving merges (ENG-5015 2b)
Three formatting losses in the direct-extraction markdown path:
1. text_with_formatting gains <u> run emission (detect_underline option,
default on) using the geometric is_underline flag from 1.9.9.
Underline runs stay free of nested bold/italic markers — consumers
match tag content literally. Heading lines keep plain text for
bold/italic but preserve <u>: the tag carries meaning `#` doesn't.
2. merge_subscript_items now maps absorbed digit scripts to Unicode
sub/superscript forms with direction from the baseline offset
("H"+"2" -> "H₂", "word"+raised "2" -> "word²", "m"+"3" -> "m³").
NFKC/NFKD folds these back to plain digits so text matching
downstream is unaffected; renderers keep the script semantics.
3. merge_text_items no longer merges across bold/italic boundaries —
absorbing a styled run into a plain neighbor erased the styling
before markdown emission ever saw it. On eval docs this recovers
20-82 italic runs per document that previously emitted as plain.
Snapshots regenerated (diffs are the features: CCl₂F₂, m³, underlined
legal section headings, finer bold runs). pdf-evals regression suite:
202/202 real PDFs pass. napi 1.9.9 -> 1.9.10.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): break merges at underline boundaries too (review)
OR-merging underline stretched the eventual <u> span over neighboring
plain fragments. Merge runs now break on any style-flag change, the
redundant accumulator is gone, and format_list_item learned to move
bullet markers outside <u> wrappers so fully-underlined bullet lines
still render as markdown lists. td9264 snapshot regenerated — spans are
tighter (trailing periods correctly outside the tag).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(markdown): strip stray spaces before sentence punctuation (review)
Style-boundary item splits can strand a trailing period in its own
fragment, and multiple assembly paths join fragments with spaces,
yielding "word ." artifacts. Rather than chasing every join site, a
postprocess pass removes a space before `.`/`,`/`;` when the mark ends
its token (whitespace, cell boundary `|`, or end of text follows).
Dot leaders/ellipses and mid-token periods are untouched.
Fixes the td9264 "companies ." artifacts and two pre-existing
"armoring ," artifacts in the 2013-app2 snapshot. pdf-evals: zero
markdown diffs across all 203 corpus PDFs vs committed baselines.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tables): trim spaces inside parenthetical cell fragments
* fix(tables): reject sparse prose row-stripe tables
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(extractor): geometric underline detection on TextItem (ENG-5015)
PDFs carry no underline font flag — underlines are stroked horizontal
lines or thin filled rects drawn under the baseline. Correlate those
graphics (already parsed from the content stream) with text items in a
post-pass: a rule within ~0.35em below the baseline covering >=60% of
an item's width marks is_underline.
Exposed through the napi and python bindings. Verified on real docs:
4/4 underlined sentences flagged on a Japanese report, links/headings
flagged on 8 of 10 underline-bearing eval docs, zero flags on docs
without underlines. Known FP source (table cell borders) documented —
downstream applies inline styling only to plain-text regions.
napi 1.9.8 -> 1.9.9.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): underline rules only from painted rects, normalized extents (review)
Two review fixes: (1) normalize rect extents before the thickness/width
checks — `re` operands pass through the CTM so width/height can be
negative, which missed negative-width rules and let negative-height
bands pass as thin; (2) only feed painted rects to underline detection —
`re` rects now wait in a pending list until a paint operator (S/s, f/F/
f*, B/B*/b/b*) confirms them, and `re W n` clip-only paths are discarded
at `n`, so invisible clip boundaries no longer underline nearby text.
Marking moved into content_stream where paint state lives (pre-rotation,
consistent device space).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): harden underline detection
* feat(cli): export positioned text item json
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* emit ItemType::Image bboxes for Image XObjects (was: silently dropped)
Background. ItemType::Image, MarkdownOptions::include_images, and the
markdown emitter's image-collection path have all been in the tree
for a while, but no producer ever populated them — content_stream.rs
explicitly `// Skip images — text extraction only` at the Do
operator, and the nested Form-XObject walker in xobjects.rs only
matched XObjectType::Form, silently dropping Image entries. The
declared types were dead code.
This PR lights them up. At every Do that resolves to an Image
XObject (both top-level and nested inside Form XObjects), we now
compute the page-space bbox from the current CTM via a new
`image_bbox_from_ctm` helper — handling both axis-aligned and
rotated/sheared placements via 4-corner AABB — and emit a TextItem
with `item_type: ItemType::Image` and the legacy `[Image: <name>]`
text payload that the markdown emitter already knows how to render.
Callers can now find raster figures via `extract_text_with_positions`
(and the `_mem` variant, newly re-exported at the crate root) without
needing to re-parse the PDF or run a vision/layout model. The intended
consumer is layout-aware text pipelines that want to crop figures and
caption them out-of-band.
Two backstops to avoid silent breakage for existing callers:
1. `MarkdownOptions::include_images` default flipped `true → false`.
If it stayed at `true`, every existing user of
`extract_pages_markdown` would suddenly see ``
placeholders inserted throughout their output the moment they
upgraded. Image data is still available structurally via
`extract_text_with_positions`; rendering it into markdown is now
an opt-in. New regression test asserts `extract_pages_markdown`
output is unchanged for the image-bearing fixture.
2. Image items now also skip the layout heuristics
(`detect_columns`, `detect_tables_from_rects`) via a new
`is_text_layout_item` predicate. Without this filter, an image's
left edge would land in the column-projection profile and skew
table column detection — surfaced by
`vector_grid_tests::upstage_key_functions_four_cols` going from 4
detected columns to 5 in CI before the filter was added.
Re-exporting `extract_text_with_positions_mem` at the crate root —
strictly additive; mirrors how `extract_pages_markdown_mem` is already
available there.
Tests:
- test_extract_text_with_positions_emits_image_bboxes — minimal PDF
with one 200×100 image at (50, 600); asserts one Image item with
correct bbox + page + text.
- test_image_xobject_bbox_handles_rotated_ctm — 90° rotated image
via shear-component CTM; asserts AABB is correct (handles non-
axis-aligned placements via 4-corner clamp).
- test_image_emission_does_not_change_default_markdown — asserts no
`Image:` token leaks into default markdown output, regression
guard for the include_images flip.
- test_markdown_options_default_has_include_images_false — explicit
sentinel so anyone flipping it back catches it in CI.
* Bump version from 1.8.15 to 1.9.0
text_cluster_column_undercount previously fired only on tables with
6+ markdown columns. That missed a common production failure shape:
4-column page geometry where the heuristic detector's x-position
clustering collapses 2 narrow numeric columns (dates, IDs, amounts)
into adjacent wider columns, producing 2-column markdown.
The original 6-column floor existed because raw x-cluster count is
noisy on small tables — wrapped continuations, bullet indents, and
within-cell text variation produce many small x-clusters that don't
correspond to real columns. Examples:
- Pcmso-style 2-column key/value layout with multi-line values
shows 10 raw x-clusters but only 1 has more than a single item.
- Yale-style archival catalog with 3 columns shows 6 raw
x-clusters but most clusters are single-item continuations.
Replace the raw cluster count with a "significant cluster" count:
clusters whose item count is at least 1/4 of the dominant cluster
(and ≥2 items). That filters the within-cell-variation noise while
preserving signal from real columns, which consistently have one
item per row.
With significant-cluster counting:
- Narrow-undercount fires when significant_clusters >= 3 AND
significant_clusters >= table_cols * 2 (catches 4-col-collapsed-
to-2 cases without misfiring on pcmso 10-clusters or yale 6
where the significant subset matches markdown).
- Legacy wide-undercount path (table_cols >= 6) keeps the same
+2 / 1.2x thresholds, now applied to significant clusters
instead of raw clusters.
Verified against representative regression and must-not-regress
cases from prior PR cycles: narrow-undercount catches genuine
column-drop cases (significant=4, markdown=2 → routes to OCR);
must-not-regress fixtures (italian-gov 6520 chars, pcmso 1733
chars, yale 1754 chars, doc200/182/189 still routing to OCR via
PR #91 guards) unchanged.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Some PDFs have fonts whose ToUnicode CMap is missing or broken —
Identity-H fonts without unicode metadata, Type-3 fonts where every
glyph maps to garbage. The page extractor returns punctuation-only
fragments or single-glyph repeats; the rendered image still carries
the visible text, so the region should fall back to OCR rather than
serve a partial table.
The existing captured_only_a_fragment guard can't catch this case
because region_text_chars itself collapses under font-decode failure —
captured vs extracted is symmetrically low, and the ratio still looks
acceptable.
Add a complementary area-based density guard: when a region has lots
of pixel real estate but very few text chars, the page extractor
hit a font failure. Bbox area is independent of extraction success,
so the symmetry breaks.
Threshold 0.003 chars/sq pt sits between observed clean extractions
(≥0.005 on full-page A4 ledgers, key/value layouts, archival
catalogs) and observed font-decode failures (≤0.0014 on prod-traffic
samples). Three guards keep it from misfiring:
- text_chars < 20 skipped: synthetic / fragmentary fixtures
- area < 30,000 sq pt skipped: tiny stat blocks
- area > 400,000 sq pt skipped: near-whole-A4 bboxes where
density is unreliable (large white-space margins)
Verified against three reproducible cases from prod shadow logs
that previously served partial output:
- Cyrillic page with punctuation-only decode (46 chars, density
0.00045) → flagged, routes to OCR
- Cyrillic page where every glyph collapsed to one letter (89
chars, density 0.00025) → flagged, routes to OCR
- Materials-test region where text extracted fine but the table
body extends beyond the bbox (96 chars, density 0.00131)
→ flagged, routes to OCR
Existing fixtures (full-page A4 ledger, multi-row key/value with
paragraph values, archival catalog, bits_pilani whole-page tests,
synthetic line-grid test) all retain identical behavior.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After enabling the vector-grid detectors on the extract path (#85)
and broadening detection to full-page grids (#83), long-cell tables
(#84), and segment-only layouts (#86), one residual failure shape
remained: detectors finding a valid grid but only capturing a small
fraction of the region's actual text. Two recurring sub-shapes:
- "header-only": detector captured the column-header band (often a
multi-line year/units block) but missed every data row below.
Common in financial statements, securities tables, budget
appendices.
- "sparse": detector returned a handful of fragmentary cells from a
content-rich region, missing the bulk of the page.
Both pass the existing needs_ocr quality gates — the captured cells
are well-formed markdown — but the customer would receive a 5-row
fragment of a 50-row table. Today these regions fell back to GLM-OCR
by default; flipping `__nativeTableExtraction=true` would start
serving the partials.
Add `captured_only_a_fragment(md, region_text_chars)`: rejects when
the captured non-delimiter character count is less than 25% of the
text the page extractor saw inside the region. The 200-char region
floor keeps short legitimate tables (units, axis labels) from being
mis-flagged. Wired into the existing `evaluate` quality gate
alongside is_garbage_text / is_cid_garbage / detect_encoding_issues
/ looks_like_partial_table_ex.
Verified against three representative residual cases from shadow
logs (financial-statement header band, securities-table fragment,
ESIA sparse region): all flip from `needs_ocr=false` with partial
output to `needs_ocr=true` so GLM takes over. Existing full-table
fixtures (governmental ledger, PPRA-style key/value, archival
catalog) still pass through unchanged.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Some catalog and archival-finding-aid tables draw each row's
horizontal rule as N segments (one segment per cell) with no vertical
lines at all. The previous detector rejected these outright at
`verticals.len() < 2`, even though the segment break points encoded
the column boundaries unambiguously.
When the vertical-line count is below the existing threshold, walk
the horizontal-segment x-endpoints and cluster them with the same
snap_edges path used for vertical-line columns. Accept the derived
edges only when ≥3 distinct x-positions each appear on ≥50% of the
unique horizontal-line rows — that consistency guard distinguishes
real per-cell segments from decorative rules with varying widths
(which never share endpoints across many rows).
When columns come from segment endpoints, skip the downstream
"spanning_v / partial_v" gate (there are no vertical lines to
validate against). All other gates — horizontal-span coverage,
content density, capture ratio, multi-column distribution, the
uniform-spacing chart-grid rejector — still apply.
Verified on a 7-row × 3-col archival catalog page that previously
extracted 98 chars (a 2-row fragment via the heuristic fallback); now
extracts 1754 chars with all rows + multi-line cells.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* extract_tables: try vector-grid detectors before text heuristic
extract_tables_in_regions_mem previously ran only the text-only
heuristic detector (tables::detect_tables) on the items inside each
region, discarding the rects and lines that extract_page_text_items
returned. That left the rect-backed and line-backed detectors
(detect_tables_from_rects, detect_tables_from_lines) unused by the
public region-scoped extraction path — they only ran through
detect_vector_grid_in_region_mem, which most callers don't use.
Keep the rects and lines, filter them to each region, and try in
order: rect detector → line detector → heuristic. Each candidate's
markdown is quality-gated by the existing needs_ocr checks
(is_garbage_text, is_cid_garbage, detect_encoding_issues,
looks_like_partial_table_ex); only the first clean output wins.
If all three produce empty or noisy output we still return
needs_ocr=true, matching prior behavior.
Effect on real prod-shape inputs from shadow logs:
Full-page ruled ledger, 6 cols x ~15 rows:
before: heuristic emits a 355-char two-row fragment
after: line detector emits the full 6520-char table
Multi-row key/value layout with paragraph values:
before: heuristic emits a 188-char header-only fragment
after: rect detector emits the full 1733-char table including
the multi-bullet description cell
Existing fixtures that already passed via the heuristic continue to
pass: the quality gate rejects partial vector-grid output and falls
through, so the heuristic still wins where it produced the cleaner
result.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Bump version from 1.8.9 to 1.8.10
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a wire-bordered table has headers centered/right-aligned in their
cells but data left-aligned, cluster_x_positions can both merge adjacent
data columns (when the data-to-data gap is below the clamped threshold)
and drop the header-only x-positions in its singleton-filter pass. The
cell-rect fallback then used text-cluster column edges and lost a column
or fragmented neighbor cells.
Prefer rect-derived column edges when the rect grid has 3+ columns and
every rect column holds multiple text items. The all-cols-populated
check protects against decorative or background rects (prose laid out
in a frame, cell-fill rects with extra borders) that would otherwise
split a logical column into spurious sub-columns. The existing
prose-in-frame, well-distributed-columns, and wireless-prose guards
still fire for the cases they were built for.
Bump napi version 1.8.7 → 1.8.8.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: lift detection on shaded-header + alt-row tables (#wired-grids)
Production telemetry on `wired_high_confidence`-classified table regions
showed `detect_vector_grid_in_region_mem` returning a usable grid only
~27% of the time, with the rest falling through to GLM-OCR. Three
surgical fixes target the dominant production shapes:
* Path-fill cell backgrounds: when the page has no `re` rects but draws
cell backgrounds via `m`/`l`/`h`/`f*` sequences, prefer the fill-derived
rects over the few section-level `W*` clip paths that previously won
the priority gate. Activated when fill rects outnumber clip rects ≥3×.
* Dedup-induced cluster splits: page-background rects could pose as
containers in the sub-rect dedup and evict a slightly smaller
table-frame rect, breaking adjacency between column-cell groups so each
column became its own cluster. Origin-anchored containers are now
disqualified from sub-rect dedup. A separate exact-duplicate pass
collapses the cell-padding/text-bg/cell-border triple emissions some
PDFs produce, preserving original order to avoid reshuffling table
output on multi-table pages.
* Prose-words rejection: the `cell-rect` fallback's whole-grid prose
threshold also rejected real tables that include a description column.
Now relaxed when content is well-distributed (≥75% of cols filled),
while keeping the original strictness for prose-in-a-frame layouts.
Two regression fixtures from the opendataloader-bench corpus, covering
the dominant production failure categories:
* `greencomp_competence.pdf` — 2-col shaded-header + plain-body glossary.
Mirrors production crops #1 (Contractions glossary) and #6 (BIO 350
course header).
* `upstage_key_functions.pdf` — 4-col shaded-header + alt-row backgrounds
+ merged left column. Mirrors production crops #2 (Parameter/Value
alt-row), #7 (Spanish XML schema), and #8 (Córdoba multi-row header).
Existing fixtures stay green (doc 51 wrapped-label, doc 128 forecast
six-cols, td9264 snapshot). 133 unit + integration tests pass; clippy
clean.
Bumps napi/package.json 1.8.4 → 1.8.5.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: tighten prose-in-frame rejection — fixes pdf-evals #30 regression
PR #76's shaded-header detection lift surfaced a regression on
accessory_building_permit_application_1 (TEDS 0.10 → 0.05): a
paragraph of legal text laid out in a 2-column justified block was
being admitted as a 10×2 fake table where every cell holds a
sentence fragment ("I agree to comply...", "I", "It is the property
owner's responsibility..."). Per pdf-evals PR #30 review, this is
the kind of regression production users will notice — the markdown
is structurally and semantically misleading.
Root cause: PR #76's prose-rejection only fires for `num_cols >= 4`,
so the 2-col prose-in-a-frame case slipped past it entirely. The new
fill-priority + dedup changes started producing rects for this layout
that 1.8.4 correctly ignored.
Fix: tighten the prose-in-frame check.
- Lower the column-count guard from `>= 4` to `>= 2`.
- Add a content-length signal as the primary discriminator: when the
prose-words trigger fires AND mean non-empty cell length exceeds
65 chars, reject regardless of column distribution.
The 65-char threshold cleanly separates observed cases:
accessory_building (prose-in-frame): mean 74 chars → REJECT
upstage_key_functions (real 4-col table): mean 53 → admit
greencomp_competence (real 2-col glossary): mean 20 → admit
accessory_building (real 5×3 form data): mean 10 → admit
The well-distributed-cols relaxation that PR #76 added stays —
"label / value / description / benefit" tables (#7, #8 from the
production crops) still pass, but only when their mean cell length
stays below the prose threshold.
New regression test `accessory_building_rejects_prose_in_frame` asserts
both that the real 5×3 form data table survives AND the 10×2 prose
block is rejected. Snapshot test `test_snapshot_td9264` updated to
match new output — old snapshot captured the same prose-in-frame bug
on regulatory text (paragraphs emitted as 3-col `||text||` fake-table
rows). New snapshot emits clean prose paragraphs, which is correct.
Verification:
- cargo test --all: 424 lib + 133 integration + 2 doc tests pass
- cargo fmt --check clean
- cargo clippy -- -D warnings clean (lib-level; pre-existing
test-level clippy issues on the wired-grids branch unaffected)
- Existing fixtures stay green: forecast_table_chart_six_cols (PR
#72), bits_pilani_* (PR #73), greencomp_competence_two_cols and
upstage_key_functions_four_cols (PR #76).
This branch is based on abimaelmartell/wired-grids so it includes
PR #76's commits plus this fix on top. Suggest merging this and
closing #76, OR rebasing #76 to incorporate this fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: drop early-dedup atom — caused broad TOC + matrix corruption
Bisected PR #76's 4 atoms against the SEC 10-K
0001104659-25-093871_183e44ac.pdf which appeared as a TEDS regression
in pdf-evals PR #30. Result:
atom | TOC | perf-graph | qualifications
---------------------------|-----|------------|---------------
fill-priority | ✓ | ✓ | ✓
early-dedup | ✗ | ✗ | ✗
page-bg disqualification | ✓ | ✓ | ✓
prose-relaxation | ✓ | ✓ | ✓
Early-dedup was the SOLE source of all three regressions on this doc.
Tried a more conservative variant (≥3 copies only — pair-duplicates
appear in legit multi-section layouts like 10-K dividers above + below
section headers); didn't fix the regression. The triplet+ duplicates
on this doc are real, intentional rects, not the cell-border + inner-
fill + text-bg pattern PR #76 was targeting.
Drop early-dedup. Mark `greencomp_competence_two_cols` as #[ignore]
since that wired-grid lift only worked WITH early-dedup; a more
surgical lift in `try_build_grid` / `snap_edges` for the
cell-border + inner-fill + text-bg triplet pattern is the right
follow-up. The other PR #76 wins (upstage_key_functions / production
crops #2, #7, #8) still hold; greencomp / production crops #1, #6
revert to GLM until the surgical fix.
Validation on the regression doc:
0001104659 TOC PART II markers: 4 (matches main, was 2 with PR#76)
0001104659 perf-graph data row: 2 (matches main, was 1)
0001104659 qualifications rows: 9 (matches main, was 6)
Validation on the prose-frame doc:
accessory_building fake-table: 0 (matches main, was 1 with PR#76)
accessory_building prose intact: 1 (matches main)
cargo test --all clean, cargo fmt --check clean, cargo clippy --lib
-- -D warnings clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Abimael Martell