The isolated-line density guard wiped every isolated line on a page
where they exceeded 25% of lines. On sparse pages (covers, ToC pages
with a lone "CONTENTS" title, section-divider pages) a single heading
is trivially >25%, so the guard erased exactly the line it exists to
find. Require the page to have >=10 lines before the guard runs — the
25% ratio only signals a multi-column misfire on a dense page.
That let more isolated lines through, exposing that the visual heading
heuristic could promote lines already tagged with a non-heading struct
role (list item, blockquote, code, caption, ToC) or set in a monospace
font. Gate the heuristic on those in both converter paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(markdown): ToC-page suppression, wrapped bold headings, math fragments
Three heading-classification improvements:
- After emitting a "Contents"/"Table of Contents" heading, suppress
heading promotion for the rest of that page: ToC entries are section
titles that look exactly like headings ("1. Overview of OCR Pack")
and whole contents pages came out as stacks of ##.
- merge_heading_lines only merged font-size-tier and struct-tree
headings, so bold-at-body-size headings that wrap emitted two
separate ## lines. Merge a fully-bold line into the previous
fully-bold line when it reads as a wrap continuation (starts
lowercase, tiny Y gap, no terminal punctuation on the previous line).
- Reject display-math fragments from the bold/rarity heading heuristic:
equations ending in an equation number ("S = kB ln W, (2)") and
lead-ins referencing one ("Rearranging Equation (8) gives:"). A bare
trailing colon is deliberately NOT a signal — real headings often end
with colons ("Procedure:").
The p1244 snapshot change is the bold-merge working as intended:
stacked form labels "**Subtotals** **from pages**" now read
"**Subtotals from pages**".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: guard tier path, protect prev headings in merge, narrow (N) rule
All three review findings applied, calibrated against the corpora:
- is_heading_fragment now gates the font-size-tier path too, not just
the rarity heuristic.
- The bold wrap-merge requires the previous line to be tier-less as
suggested; corpus diff confirmed the old behavior was absorbing a
wrapped list-item fragment into a real heading.
- The bare "(N)" suffix rule suppressed real headings ("Nicaea (325)",
appendix numbering). It now requires math evidence: an operator
(=, <=, <<, ...) in the line or ,/: immediately before the number.
Page-of-total running headers ("PM 2 (10)") get an explicit rule
since the old blanket suffix check had been catching them only by
accident.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(extractor): descriptor/embedded-font style flags + geometric strikeout detection
Two style-recall gaps, both invisible to the existing name-based
heuristics:
1. Subset fonts with opaque BaseFont names ("Tc1", "AAAAAB+Amplitude")
defeat is_italic_font/is_bold_font. New descriptor_style_flags reads
the FontDescriptor (ItalicAngle beyond 4 degrees, Flags bit 7 Italic,
bit 19 ForceBold) and, when the descriptor claims upright, falls back
to the embedded font file: ttf-parser's OS/2 fsSelection + post
italicAngle for sfnt fonts, and the CFF Name INDEX PostScript name
for bare-CFF FontFile3 (descriptor rewritten to ItalicAngle 0 while
embedding "Amplitude-LightItalic" was observed in the wild).
ORed into is_bold/is_italic at item creation (content streams and
form XObjects).
2. No strikeout signal existed. New is_strikeout on TextItem, detected
in the same pass as underline: same rules pipeline (stroked lines /
thin filled rects, table-ruling suppression), different vertical
window — a rule crossing the glyphs at 12-55% of the em above the
baseline instead of sitting at it. Exposed through napi and python
bindings and pdf2md --items-json.
Verified on public ParseBench corpus docs: previously-missed italic
council titles and bold CJK itinerary headings now flagged (render-
checked); 24/508 docs gain flags, none lose any; 35 strikeout items
detected corpus-wide, disjoint from underline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): quote-op advance width, Ts text rise, doc-level font style cache (PR #125 review)
Address three valid findings from review:
- The ' (move-to-next-line-and-show-text) operator emitted zero-width
items and never advanced the text matrix, so geometric underline/
strikeout detection (which requires width > 0) could never mark its
text, and following show ops overlapped it. Reuse Tj's advance-width
computation and matrix advance.
- Ts (text rise) was dropped entirely: raised/lowered runs kept the
unshifted baseline, so rules drawn at the risen glyph position missed
the strike/underline windows. Track rise in the text state (saved and
restored with q/Q) and shift the rendering position through the text
matrix's y column; advances stay on the unshifted matrix per spec.
- descriptor_style_flags re-decompressed and re-parsed the same embedded
font program on every page whenever the descriptor left a style flag
unset (the common case). Add a document-scoped FontStyleCache keyed by
the FontFile2/FontFile3 object id, threaded through page and form
extraction alongside the existing CMapDecisionCache.
The fourth finding (Form XObject rules never reach geometric detection)
is real but pre-existing for underline and needs the form walker to grow
path/paint tracking plus a new return type; deferred as a follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): ActualText items render at their glyphs' text rise (PR #125 review)
The EMC-built ActualText item used the captured text matrix without the
rise adjustment the ordinary Tj/TJ/' emission sites apply, so a tagged
run shown with Ts landed on the unshifted baseline — off the strikeout/
underline windows and inconsistent with untagged runs. The rise is
captured together with the first-glyph matrix (and at BDC for the
entry-position fallback): the item must render at the rise of its
GLYPHS, not whatever rise is set by EMC time.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): capture ActualText glyph position after the quote op's line move (PR #125 review)
The `'` handler skipped the entire suppressed-extraction block, so a
tagged span whose show op is `'` never captured its glyph matrix/rise —
the EMC item fell back to the BDC-entry matrix, which sits on the
PREVIOUS line (the `'` line move happens after BDC) with no rise. The
capture now happens right after the line move, matching the Tj/TJ
paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): style-boundary gate on subscript merge + strikeout suppression coverage (PR #125 review)
merge_subscript_items absorbed a script digit into its parent
regardless of underline/strikeout flags — dropping the digit's own mark
or widening the parent's over it. The merged item carries one flag, so
differing marks now break the merge, mirroring merge_text_items'
style-boundary rule (pre-existing for underline as well).
Also extends the table-suppression test to assert is_strikeout is
cleared alongside is_underline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(lib): detect substitution-cipher garbled text from broken ToUnicode CMaps
ParseBench text_simple__att10k.pdf (issue #118) ships Type0/Identity-H
fonts whose ToUnicode CMaps are authored garbled: every bfrange maps with
a wrong constant delta, so text extracts as pure-ASCII ciphertext
("Certificate" -> "8VceZWZTReV"). The embedded subset font has no cmap
table and no glyph names, so no decode source can recover the real text
(poppler and mupdf emit the same ciphertext). The only correct behavior
is to flag the page for OCR instead of serving the garbage silently --
but the text is 100% printable ASCII with word-like tokens, so it slipped
past is_garbage_text and detect_encoding_issues.
Add CipherGarbleStats, a letter-statistics discriminator that flags a
Latin-dominant sample (>=200 ASCII letters) when vowels are starved
(<=30% of letters) AND either:
- lowercase->uppercase transitions inside words exceed 10% of letter
bigrams (a shifted lowercase alphabet straddles the ASCII uppercase
block), or
- the letter histogram's cosine similarity against English letter
frequencies drops below 0.60 (catches shifts that stay within case
blocks).
Wired into analyze_text_quality (per-page, item-level) and
detect_encoding_issues (markdown-level), so extract_pages_markdown
reports needs_ocr + suspected_garbled_text and suppresses the garbage.
Thresholds validated against the 380-document pdf-evals snapshot corpus
(Swedish, Finnish, Turkish, German, romaji, schematics, all-caps and
camelCase-heavy docs): zero false positives, and byte-identical eval
output vs main. Garbled page measures vowel ratio 0.245 / case-shift
rate 0.225 / cosine 0.532; closest legitimate document on each axis is
0.264 / 0.021 / 0.801.
Fixes#118
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump pdf-inspector to 0.1.4, npm package to 1.9.11
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lib): exempt uniform-case structured content from cipher detection
Address PR review (cubic P2): the frequency branch (english_cosine < 0.60)
fired on any Latin-dominant, low-vowel letter distribution unlike English,
so non-linguistic ASCII — DNA/protein sequences, ticker symbols, hex dumps —
could be suppressed and routed to OCR despite not being garbled. Measured:
DNA cosine 0.428 / vowel ratio 0.260, protein 0.738, tickers 0.747, hex
0.549 — all would have flagged.
Add a mixed-case guard to looks_garbled: garbled English is a permutation of
natural language and carries sentence capitalization (block-straddling shifts
invert the ratio — att10k is 60% uppercase; in-case Caesar shifts preserve it
at ~3%), so both keep some of each case. The exempted structured content is
uniform case (all upper or all lower). Requiring the minority case to be >=1%
of ASCII letters exempts single-case sequences while preserving both garble
signals, including the in-case-shift scenario the frequency branch exists for.
Strictly tightens the detector: it can only remove flags, so the eval corpus
stays at zero false positives (verified byte-identical to a baseline main
binary across all 185 PDFs) and att10k remains flagged. Adds regression tests
for DNA, protein, tickers, and an in-case Caesar shift.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lib): make cipher detection case-agnostic via sorted-histogram shape
Address PR review follow-up: the mixed-case guard from the previous commit
returned before the vowel/frequency checks, creating a blind spot — a
uniform-case (all-lower or all-upper) substitution cipher is a plausible
broken-CMap output and would bypass OCR entirely.
Replace the case proxy with the actual invariant. A substitution cipher is
a bijection over a real language's alphabet, so it preserves the frequency
SHAPE (the sorted histogram) while scrambling letter POSITIONS (the unsorted
histogram). Signal 2 now flags when english_cosine < 0.60 (positions unlike
English) AND english_shape_cosine >= 0.90 (profile is still English-shaped).
This is independent of case, so it catches all-lower, all-upper, and
case-straddling shifts alike.
The exempted structured content fails one half: DNA/hex dumps have too steep
a profile (shape cosine 0.74 / 0.81 < 0.90), while protein sequences, ticker
symbols and base64 are not sufficiently unlike English in position (unsorted
cosine 0.74 / 0.75 / 0.77 >= 0.60). All stay out of OCR.
Still strictly corpus-safe: every real Latin document scores unsorted cosine
>= 0.70 (min 0.80), far above the 0.60 gate, so none can reach Signal 2.
Re-verified byte-identical to a baseline main binary across all 185 eval
PDFs; att10k remains flagged. Drops the now-unused case counters and adds
all-lowercase / all-uppercase shifted-prose regression tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: source Python package version from Cargo.toml via maturin
Address PR review (cubic P2): pyproject.toml pinned version = "0.1.0",
which overrides Cargo.toml, so a maturin build produced a 0.1.0 Python
artifact regardless of the crate version (it had drifted since the PyO3
bindings were added). Switch to dynamic = ["version"] so maturin sources
the version from Cargo.toml [package] version and the two can no longer
diverge. No workflow auto-publishes the Python package, so this is metadata
hygiene rather than a release-path fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(markdown): underline emission, Unicode scripts, style-preserving merges (ENG-5015 2b)
Three formatting losses in the direct-extraction markdown path:
1. text_with_formatting gains <u> run emission (detect_underline option,
default on) using the geometric is_underline flag from 1.9.9.
Underline runs stay free of nested bold/italic markers — consumers
match tag content literally. Heading lines keep plain text for
bold/italic but preserve <u>: the tag carries meaning `#` doesn't.
2. merge_subscript_items now maps absorbed digit scripts to Unicode
sub/superscript forms with direction from the baseline offset
("H"+"2" -> "H₂", "word"+raised "2" -> "word²", "m"+"3" -> "m³").
NFKC/NFKD folds these back to plain digits so text matching
downstream is unaffected; renderers keep the script semantics.
3. merge_text_items no longer merges across bold/italic boundaries —
absorbing a styled run into a plain neighbor erased the styling
before markdown emission ever saw it. On eval docs this recovers
20-82 italic runs per document that previously emitted as plain.
Snapshots regenerated (diffs are the features: CCl₂F₂, m³, underlined
legal section headings, finer bold runs). pdf-evals regression suite:
202/202 real PDFs pass. napi 1.9.9 -> 1.9.10.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): break merges at underline boundaries too (review)
OR-merging underline stretched the eventual <u> span over neighboring
plain fragments. Merge runs now break on any style-flag change, the
redundant accumulator is gone, and format_list_item learned to move
bullet markers outside <u> wrappers so fully-underlined bullet lines
still render as markdown lists. td9264 snapshot regenerated — spans are
tighter (trailing periods correctly outside the tag).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(markdown): strip stray spaces before sentence punctuation (review)
Style-boundary item splits can strand a trailing period in its own
fragment, and multiple assembly paths join fragments with spaces,
yielding "word ." artifacts. Rather than chasing every join site, a
postprocess pass removes a space before `.`/`,`/`;` when the mark ends
its token (whitespace, cell boundary `|`, or end of text follows).
Dot leaders/ellipses and mid-token periods are untouched.
Fixes the td9264 "companies ." artifacts and two pre-existing
"armoring ," artifacts in the 2013-app2 snapshot. pdf-evals: zero
markdown diffs across all 203 corpus PDFs vs committed baselines.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tables): trim spaces inside parenthetical cell fragments
* fix(tables): reject sparse prose row-stripe tables
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(extractor): geometric underline detection on TextItem (ENG-5015)
PDFs carry no underline font flag — underlines are stroked horizontal
lines or thin filled rects drawn under the baseline. Correlate those
graphics (already parsed from the content stream) with text items in a
post-pass: a rule within ~0.35em below the baseline covering >=60% of
an item's width marks is_underline.
Exposed through the napi and python bindings. Verified on real docs:
4/4 underlined sentences flagged on a Japanese report, links/headings
flagged on 8 of 10 underline-bearing eval docs, zero flags on docs
without underlines. Known FP source (table cell borders) documented —
downstream applies inline styling only to plain-text regions.
napi 1.9.8 -> 1.9.9.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): underline rules only from painted rects, normalized extents (review)
Two review fixes: (1) normalize rect extents before the thickness/width
checks — `re` operands pass through the CTM so width/height can be
negative, which missed negative-width rules and let negative-height
bands pass as thin; (2) only feed painted rects to underline detection —
`re` rects now wait in a pending list until a paint operator (S/s, f/F/
f*, B/B*/b/b*) confirms them, and `re W n` clip-only paths are discarded
at `n`, so invisible clip boundaries no longer underline nearby text.
Marking moved into content_stream where paint state lives (pre-rotation,
consistent device space).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): harden underline detection
* feat(cli): export positioned text item json
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* emit ItemType::Image bboxes for Image XObjects (was: silently dropped)
Background. ItemType::Image, MarkdownOptions::include_images, and the
markdown emitter's image-collection path have all been in the tree
for a while, but no producer ever populated them — content_stream.rs
explicitly `// Skip images — text extraction only` at the Do
operator, and the nested Form-XObject walker in xobjects.rs only
matched XObjectType::Form, silently dropping Image entries. The
declared types were dead code.
This PR lights them up. At every Do that resolves to an Image
XObject (both top-level and nested inside Form XObjects), we now
compute the page-space bbox from the current CTM via a new
`image_bbox_from_ctm` helper — handling both axis-aligned and
rotated/sheared placements via 4-corner AABB — and emit a TextItem
with `item_type: ItemType::Image` and the legacy `[Image: <name>]`
text payload that the markdown emitter already knows how to render.
Callers can now find raster figures via `extract_text_with_positions`
(and the `_mem` variant, newly re-exported at the crate root) without
needing to re-parse the PDF or run a vision/layout model. The intended
consumer is layout-aware text pipelines that want to crop figures and
caption them out-of-band.
Two backstops to avoid silent breakage for existing callers:
1. `MarkdownOptions::include_images` default flipped `true → false`.
If it stayed at `true`, every existing user of
`extract_pages_markdown` would suddenly see ``
placeholders inserted throughout their output the moment they
upgraded. Image data is still available structurally via
`extract_text_with_positions`; rendering it into markdown is now
an opt-in. New regression test asserts `extract_pages_markdown`
output is unchanged for the image-bearing fixture.
2. Image items now also skip the layout heuristics
(`detect_columns`, `detect_tables_from_rects`) via a new
`is_text_layout_item` predicate. Without this filter, an image's
left edge would land in the column-projection profile and skew
table column detection — surfaced by
`vector_grid_tests::upstage_key_functions_four_cols` going from 4
detected columns to 5 in CI before the filter was added.
Re-exporting `extract_text_with_positions_mem` at the crate root —
strictly additive; mirrors how `extract_pages_markdown_mem` is already
available there.
Tests:
- test_extract_text_with_positions_emits_image_bboxes — minimal PDF
with one 200×100 image at (50, 600); asserts one Image item with
correct bbox + page + text.
- test_image_xobject_bbox_handles_rotated_ctm — 90° rotated image
via shear-component CTM; asserts AABB is correct (handles non-
axis-aligned placements via 4-corner clamp).
- test_image_emission_does_not_change_default_markdown — asserts no
`Image:` token leaks into default markdown output, regression
guard for the include_images flip.
- test_markdown_options_default_has_include_images_false — explicit
sentinel so anyone flipping it back catches it in CI.
* Bump version from 1.8.15 to 1.9.0
* extract_tables: try vector-grid detectors before text heuristic
extract_tables_in_regions_mem previously ran only the text-only
heuristic detector (tables::detect_tables) on the items inside each
region, discarding the rects and lines that extract_page_text_items
returned. That left the rect-backed and line-backed detectors
(detect_tables_from_rects, detect_tables_from_lines) unused by the
public region-scoped extraction path — they only ran through
detect_vector_grid_in_region_mem, which most callers don't use.
Keep the rects and lines, filter them to each region, and try in
order: rect detector → line detector → heuristic. Each candidate's
markdown is quality-gated by the existing needs_ocr checks
(is_garbage_text, is_cid_garbage, detect_encoding_issues,
looks_like_partial_table_ex); only the first clean output wins.
If all three produce empty or noisy output we still return
needs_ocr=true, matching prior behavior.
Effect on real prod-shape inputs from shadow logs:
Full-page ruled ledger, 6 cols x ~15 rows:
before: heuristic emits a 355-char two-row fragment
after: line detector emits the full 6520-char table
Multi-row key/value layout with paragraph values:
before: heuristic emits a 188-char header-only fragment
after: rect detector emits the full 1733-char table including
the multi-bullet description cell
Existing fixtures that already passed via the heuristic continue to
pass: the quality gate rejects partial vector-grid output and falls
through, so the heuristic still wins where it produced the cleaner
result.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Bump version from 1.8.9 to 1.8.10
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a wire-bordered table has headers centered/right-aligned in their
cells but data left-aligned, cluster_x_positions can both merge adjacent
data columns (when the data-to-data gap is below the clamped threshold)
and drop the header-only x-positions in its singleton-filter pass. The
cell-rect fallback then used text-cluster column edges and lost a column
or fragmented neighbor cells.
Prefer rect-derived column edges when the rect grid has 3+ columns and
every rect column holds multiple text items. The all-cols-populated
check protects against decorative or background rects (prose laid out
in a frame, cell-fill rects with extra borders) that would otherwise
split a logical column into spurious sub-columns. The existing
prose-in-frame, well-distributed-columns, and wireless-prose guards
still fire for the cases they were built for.
Bump napi version 1.8.7 → 1.8.8.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: lift detection on shaded-header + alt-row tables (#wired-grids)
Production telemetry on `wired_high_confidence`-classified table regions
showed `detect_vector_grid_in_region_mem` returning a usable grid only
~27% of the time, with the rest falling through to GLM-OCR. Three
surgical fixes target the dominant production shapes:
* Path-fill cell backgrounds: when the page has no `re` rects but draws
cell backgrounds via `m`/`l`/`h`/`f*` sequences, prefer the fill-derived
rects over the few section-level `W*` clip paths that previously won
the priority gate. Activated when fill rects outnumber clip rects ≥3×.
* Dedup-induced cluster splits: page-background rects could pose as
containers in the sub-rect dedup and evict a slightly smaller
table-frame rect, breaking adjacency between column-cell groups so each
column became its own cluster. Origin-anchored containers are now
disqualified from sub-rect dedup. A separate exact-duplicate pass
collapses the cell-padding/text-bg/cell-border triple emissions some
PDFs produce, preserving original order to avoid reshuffling table
output on multi-table pages.
* Prose-words rejection: the `cell-rect` fallback's whole-grid prose
threshold also rejected real tables that include a description column.
Now relaxed when content is well-distributed (≥75% of cols filled),
while keeping the original strictness for prose-in-a-frame layouts.
Two regression fixtures from the opendataloader-bench corpus, covering
the dominant production failure categories:
* `greencomp_competence.pdf` — 2-col shaded-header + plain-body glossary.
Mirrors production crops #1 (Contractions glossary) and #6 (BIO 350
course header).
* `upstage_key_functions.pdf` — 4-col shaded-header + alt-row backgrounds
+ merged left column. Mirrors production crops #2 (Parameter/Value
alt-row), #7 (Spanish XML schema), and #8 (Córdoba multi-row header).
Existing fixtures stay green (doc 51 wrapped-label, doc 128 forecast
six-cols, td9264 snapshot). 133 unit + integration tests pass; clippy
clean.
Bumps napi/package.json 1.8.4 → 1.8.5.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: tighten prose-in-frame rejection — fixes pdf-evals #30 regression
PR #76's shaded-header detection lift surfaced a regression on
accessory_building_permit_application_1 (TEDS 0.10 → 0.05): a
paragraph of legal text laid out in a 2-column justified block was
being admitted as a 10×2 fake table where every cell holds a
sentence fragment ("I agree to comply...", "I", "It is the property
owner's responsibility..."). Per pdf-evals PR #30 review, this is
the kind of regression production users will notice — the markdown
is structurally and semantically misleading.
Root cause: PR #76's prose-rejection only fires for `num_cols >= 4`,
so the 2-col prose-in-a-frame case slipped past it entirely. The new
fill-priority + dedup changes started producing rects for this layout
that 1.8.4 correctly ignored.
Fix: tighten the prose-in-frame check.
- Lower the column-count guard from `>= 4` to `>= 2`.
- Add a content-length signal as the primary discriminator: when the
prose-words trigger fires AND mean non-empty cell length exceeds
65 chars, reject regardless of column distribution.
The 65-char threshold cleanly separates observed cases:
accessory_building (prose-in-frame): mean 74 chars → REJECT
upstage_key_functions (real 4-col table): mean 53 → admit
greencomp_competence (real 2-col glossary): mean 20 → admit
accessory_building (real 5×3 form data): mean 10 → admit
The well-distributed-cols relaxation that PR #76 added stays —
"label / value / description / benefit" tables (#7, #8 from the
production crops) still pass, but only when their mean cell length
stays below the prose threshold.
New regression test `accessory_building_rejects_prose_in_frame` asserts
both that the real 5×3 form data table survives AND the 10×2 prose
block is rejected. Snapshot test `test_snapshot_td9264` updated to
match new output — old snapshot captured the same prose-in-frame bug
on regulatory text (paragraphs emitted as 3-col `||text||` fake-table
rows). New snapshot emits clean prose paragraphs, which is correct.
Verification:
- cargo test --all: 424 lib + 133 integration + 2 doc tests pass
- cargo fmt --check clean
- cargo clippy -- -D warnings clean (lib-level; pre-existing
test-level clippy issues on the wired-grids branch unaffected)
- Existing fixtures stay green: forecast_table_chart_six_cols (PR
#72), bits_pilani_* (PR #73), greencomp_competence_two_cols and
upstage_key_functions_four_cols (PR #76).
This branch is based on abimaelmartell/wired-grids so it includes
PR #76's commits plus this fix on top. Suggest merging this and
closing #76, OR rebasing #76 to incorporate this fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: drop early-dedup atom — caused broad TOC + matrix corruption
Bisected PR #76's 4 atoms against the SEC 10-K
0001104659-25-093871_183e44ac.pdf which appeared as a TEDS regression
in pdf-evals PR #30. Result:
atom | TOC | perf-graph | qualifications
---------------------------|-----|------------|---------------
fill-priority | ✓ | ✓ | ✓
early-dedup | ✗ | ✗ | ✗
page-bg disqualification | ✓ | ✓ | ✓
prose-relaxation | ✓ | ✓ | ✓
Early-dedup was the SOLE source of all three regressions on this doc.
Tried a more conservative variant (≥3 copies only — pair-duplicates
appear in legit multi-section layouts like 10-K dividers above + below
section headers); didn't fix the regression. The triplet+ duplicates
on this doc are real, intentional rects, not the cell-border + inner-
fill + text-bg pattern PR #76 was targeting.
Drop early-dedup. Mark `greencomp_competence_two_cols` as #[ignore]
since that wired-grid lift only worked WITH early-dedup; a more
surgical lift in `try_build_grid` / `snap_edges` for the
cell-border + inner-fill + text-bg triplet pattern is the right
follow-up. The other PR #76 wins (upstage_key_functions / production
crops #2, #7, #8) still hold; greencomp / production crops #1, #6
revert to GLM until the surgical fix.
Validation on the regression doc:
0001104659 TOC PART II markers: 4 (matches main, was 2 with PR#76)
0001104659 perf-graph data row: 2 (matches main, was 1)
0001104659 qualifications rows: 9 (matches main, was 6)
Validation on the prose-frame doc:
accessory_building fake-table: 0 (matches main, was 1 with PR#76)
accessory_building prose intact: 1 (matches main)
cargo test --all clean, cargo fmt --check clean, cargo clippy --lib
-- -D warnings clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* extractor: drop Latin-1 mojibake on Type0/CID fonts; tokenize wide TSR items
Two text-extraction failure modes surfaced by table-candidate shadow
data; both also affect the existing TableFormer / vector-grid paths
since they share `extract_tables_with_structure_*_mem`'s downstream
cell-fill.
1. CJK / multi-byte mojibake. The bottom Latin-1 fallback in
`extract_text_from_operand` ran unconditionally. For a Type0/CID
(Identity-H) font whose ToUnicode CMap fails to parse, the bytes
are CIDs (font-internal indices), not character codes — per-byte
Latin-1 produces mojibake (e.g. 2-byte CID 0xCDD9 surfaces as "ÍÙ").
Gate that fallback on `FontWidthInfo.is_cid` (set by
`parse_type0_widths` for `/Subtype /Type0`). For Type0 fonts with
any non-ASCII byte, emit one U+FFFD per CID instead so
`detect_encoding_issues` still trips and the page is flagged for
OCR — preserving the existing OCR-routing path that the
high-Latin-1 garbage used to satisfy by accident. Type1 / TrueType
simple fonts retain the per-byte Latin-1 round-trip (it IS the
canonical interpretation for them; verified against an existing
pdf-evals fixture where bytes like 0xB6 are legitimate Latin-1).
Threaded `font_widths: &PageFontWidths` through
`extract_text_from_operand` and its 5 call sites in
`content_stream.rs` / `xobjects.rs`.
2. Dense-cell text collapse in `extract_tables_with_structure_cells_mem`.
Stage-1 routing did per-item assignment — each TextItem went into the
single cell whose bbox contained its center. When a row's text is
rendered as one wide Tj (e.g. "Marshall Islands 0.9 0.9 0.9"), the
whole row parks in one cell and the rest of the row stays empty.
New `split_item_into_token_subitems` helper splits each item into
per-token virtual sub-items with x positions estimated from
`effective_width / char_count` and the token's character offset.
Stage 1 then routes per-token. Single-token items collapse to a
one-element vector (no behavior change). Multi-token items spanning
multiple cells distribute correctly. Stage-2 orphan recovery now
operates on token-grain orphans rather than re-trying whole items.
Tests:
- `cid_font_with_unparseable_cmap_does_not_emit_latin1_mojibake` (unit)
exercises the Type0/CID + unparseable-CMap fallback path.
- `simple_font_latin1_fallback_passes_high_bytes_through` (unit)
guards the false-positive case where a Type1 font's `/ToUnicode`
reference is set but bytes are legitimate Latin-1 character codes.
- `test_extract_tables_with_structure_distributes_wide_item_across_cells`
(integration) builds a synthetic PDF with one wide Tj and asserts
each token lands in its own cell.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tests: pin CID mojibake fix mechanism with FFFD assertions
Two complementary tests for the Type0/CID Latin-1-fallback guard:
1. Tighten `test_identity_h_no_tounicode_suppresses_garbage` on the
existing real-PDF fixture `shinagawa_identity_h.pdf` to also assert
the pre-suppression text contains U+FFFD and contains no high-Latin-1
chars. Pins down WHICH mechanism is suppressing the garbage so a
future regression that re-enables Latin-1 mojibake fails loudly here
instead of silently switching the suppression chain back to
`is_cid_garbage` + high-Latin-1 detection.
2. Add `test_synthetic_type0_broken_tounicode_emits_fffd_not_latin1_mojibake`
with a fully-synthetic Type0 / Identity-H PDF built in process. We
control the malformed ToUnicode contents, the descendant CIDFontType2
shape (just enough for `parse_type0_widths` to set `is_cid=true`,
which is what the new guard keys off of), and the Tj byte stream.
No fixture file or external license needed. Reproduces the exact
"Type0 + non-ASCII bytes + unparseable ToUnicode" code path that
produced the production mojibake samples.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tables: expand multi-row TSR cells in place
Recover row-under-counted TSR tables by splitting overstuffed cells with native PDF text bands before falling back to heuristic extraction.
Made-with: Cursor
* docs: note multi-row expansion scope
Clarify that the row-band cap intentionally keeps v1 focused on common small row-loss cases while larger compressions continue to use heuristic fallback.
Made-with: Cursor
Three fixes to extract_tables_with_structure_auto_mem (added in
1.7.1) caught by external review:
1. multi_row_in_cell over-triggered on legitimate multi-line cells.
The previous threshold (item span > 1.3× either smallest cell or
tallest item height) fires on any cell with 2+ y-separated text
items — including rowspan>1 cells, wrapped descriptions, and
superscript/subscript runs. Replaced with two gates:
- skip cells whose declared rowspan > 1 (intentional multi-line)
- require an actual whitespace gap (>~half a line height)
between the bottom of one item and the top of the next, in
PDF-native y-coordinates. Same-line items with tall glyphs or
superscripts have negative or near-zero gap; truly separate
visual rows have gap ≈ leading − line-height.
FNBO regression test still passes; new test covers a rowspan=2
cell with two visible text lines and verifies no fallback fires.
2. Heuristic returning empty silently replaced TSR markdown with
"". The auto wrapper now keeps the TSR markdown when the
heuristic markdown is empty/whitespace and tags fallback_reason
with `_heuristic_empty` suffix (e.g.
`multi_row_in_cell_heuristic_empty`). Worst case we ship the
same wrong-but-non-empty TSR output we'd have shipped before
1.7.1; we never replace useful output with literally nothing.
3. One bad input blanked the whole batch. Errors from
detect_tsr_quality_issue or extract_tables_in_regions_mem now
stay scoped to the single input — that input falls through to
raw TSR markdown with a `_error` reason label so callers can
metric on it. Other inputs in the batch are unaffected.
3 new integration tests:
- test_auto_does_not_fire_on_legit_rowspan_cell
- test_auto_keeps_tsr_markdown_when_heuristic_returns_empty
- test_auto_isolates_per_input_failures
All 6 auto tests + full 123-test suite pass. FNBO local replay
still triggers fallback (phantom_empty_row signal in this run) and
emits correct Shawnee/BVP/Sonoma rows with correct census tracts.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds extract_tables_with_structure_auto_mem (Rust) /
extractTablesWithStructureAuto (napi). Returns
TableExtractionResult { markdown, fallback_reason } per input.
The wrapper runs the existing TSR-hybrid path then checks the
resulting cells for two known SLANet detection pathologies:
* phantom_empty_row: empty row sandwiched between non-empty rows
(cheap, cell-metadata only).
* multi_row_in_cell: re-reads PDF text items, flags any cell whose
contained items span >1.3× either the smallest cell height or
the tallest contained item height. Catches the FNBO failure mode
where a tall TSR cell absorbs two adjacent PDF rows.
When either fires, extract_tables_in_regions_mem runs over the same
crop bbox and its markdown replaces the TSR markdown.
fallback_reason carries the diagnostic label so callers can emit
metrics and watch each pathology independently.
Validated on FNBO branches PDF page 1 (Kansas region):
- TSR-only: merges Shawnee into BVP, wrong census tract on Sonoma.
- Auto fallback (phantom_empty_row): each row separate, correct tracts.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: TSR-aware table extraction (extract_tables_with_structure_mem)
New public function that consumes raw structure-recovery output (HTML
structure tokens + per-cell bboxes from a model like SLANet) and assembles
markdown tables by pulling cell text from the native PDF — no OCR, no
geometry inference.
Why: the existing extract_tables_in_regions_mem infers grid geometry from
text positions only and can't distinguish merged cells from multiple narrow
columns. Pairing structure recovery from a layout/TSR model with native
PDF text gets perfect text quality with proper row/col/span structure.
- New module src/tables/structured.rs: token state machine, polygon→AABB,
crop-px→page-pt, rowspan/colspan-aware cell layout, markdown emitter.
Accepts both 4-element rects and 8-element 4-corner polygons.
- New public extract_tables_with_structure_mem in src/lib.rs that reuses
extract_page_text_items, region_overlaps_item, and the shared region
text-collection helper. No existing public function modified.
- napi binding extractTablesWithStructure mirroring the existing
extractTablesInRegions shape (f64 in JS → f32 internally).
- 14 unit tests + 5 integration tests, including a real-PDF gold-standard
match against bits_pilani_feedback.pdf.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TSR follow-ups: header-aware separator, cells API, v1.6.0
- cells_to_markdown emits the separator after the LAST row that contains
is_header=true cells, falling back to "after row 0" when no header is
flagged. Multi-row theads now render correctly. Three new unit tests
cover: multi-row header, header not on row 0, no headers (fallback).
- New public extract_tables_with_structure_cells_mem returning
Vec<Vec<StructuredCell>> so callers can drive their own rendering or
debug overlays without re-doing the parse + extraction. The markdown
variant now wraps it. The previously-unused page_pt_bbox field is
surfaced through this API.
- New napi binding extractTablesWithStructureCells + StructuredCellJs.
- Bump @firecrawl/pdf-inspector to 1.6.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix TSR cell text assignment for overlapping bboxes
Made-with: Cursor
* bump npm package version to 1.6.1
Made-with: Cursor
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: TSR-aware table extraction (extract_tables_with_structure_mem)
New public function that consumes raw structure-recovery output (HTML
structure tokens + per-cell bboxes from a model like SLANet) and assembles
markdown tables by pulling cell text from the native PDF — no OCR, no
geometry inference.
Why: the existing extract_tables_in_regions_mem infers grid geometry from
text positions only and can't distinguish merged cells from multiple narrow
columns. Pairing structure recovery from a layout/TSR model with native
PDF text gets perfect text quality with proper row/col/span structure.
- New module src/tables/structured.rs: token state machine, polygon→AABB,
crop-px→page-pt, rowspan/colspan-aware cell layout, markdown emitter.
Accepts both 4-element rects and 8-element 4-corner polygons.
- New public extract_tables_with_structure_mem in src/lib.rs that reuses
extract_page_text_items, region_overlaps_item, and the shared region
text-collection helper. No existing public function modified.
- napi binding extractTablesWithStructure mirroring the existing
extractTablesInRegions shape (f64 in JS → f32 internally).
- 14 unit tests + 5 integration tests, including a real-PDF gold-standard
match against bits_pilani_feedback.pdf.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TSR follow-ups: header-aware separator, cells API, v1.6.0
- cells_to_markdown emits the separator after the LAST row that contains
is_header=true cells, falling back to "after row 0" when no header is
flagged. Multi-row theads now render correctly. Three new unit tests
cover: multi-row header, header not on row 0, no headers (fallback).
- New public extract_tables_with_structure_cells_mem returning
Vec<Vec<StructuredCell>> so callers can drive their own rendering or
debug overlays without re-doing the parse + extraction. The markdown
variant now wraps it. The previously-unused page_pt_bbox field is
surfaced through this API.
- New napi binding extractTablesWithStructureCells + StructuredCellJs.
- Bump @firecrawl/pdf-inspector to 1.6.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The rect-based cell fallback in detect_row_stripe_table_from_cell_rects
derives columns purely from text X-position clustering. When prose
wraps inside a bounding-box rect (chat transcripts, stylized figures),
the word-boundary gaps cluster into many spurious columns, producing
a multi-column "table" that is just fragmented prose.
Count cells containing common English function words (articles,
prepositions, pronouns, common verbs) and reject the fallback when
20%+ of non-empty cells contain any such word. Real tabular data —
labels, units, numbers, short identifiers — rarely contains these.
Update the td9264 snapshot: the government document section that
previously rendered as a malformed table now renders as cleaner
prose + a proper CFR list.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: don't reclassify wrapped bold list leads as headings
When a numbered/bulleted list item's bold lead phrase wraps onto a
second visual line, that line is all_bold + standalone, which scored
above the rarity heading threshold and was emitted as #### in the
middle of the item. That reset in_list, so the body continuation
below picked up a stray `- ` bullet via the struct-tree LI path,
shattering a single item into heading + stray bullets.
Guard the font heuristic: when already inside a list, skip heading
classification for lines at the list continuation indent with a Y
gap within para_threshold. Structure-tree headings still win.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: tighten rect-row span check in propagate_merged_cells
propagate_merged_cells used an overlap-based predicate with ±tol
slop that returned true at shared row boundaries — a rect whose
top exactly equals row N's bottom lies entirely below the row, yet
the predicate considered it to span row N. When multiple background
rects aligned on a shared Y edge (e.g. consecutive row-stripe
shading), each adjacent rect would over-reach by one row, cascading
labels and data from unrelated rows into a single merged cell.
Replace the overlap predicate with a containment check: rect bottom
at or below row bottom, rect top at or above row top (each within
tol). Genuine merged-cell rects fully contain the rows they span;
tangent rects do not.
Update two snapshots that were encoding the old buggy output.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: expose per-page markdown extraction to Python and Node (#49)
Implements the feature requested in issue #49: a list-of-pages markdown output
from the Python API. Matching the existing project pattern, the feature lives
in the Rust core and is surfaced through every binding.
- Rust core: `extract_pages_markdown` (path) and `extract_pages_markdown_mem`
(bytes) now take `Option<&[u32]>` — `None` returns every page in document
order; a slice restricts and preserves caller order.
- Python: new `extract_pages_markdown(path, pages=None)` and
`extract_pages_markdown_bytes(data, pages=None)` functions plus
`PageMarkdown` / `PagesExtractionResult` classes; stub file updated.
- Node: `extractPagesMarkdown(buffer, pages?)` — `pages` is now optional.
- Tests: 2 new Rust integration tests, 9 new Python tests, 2 new Node
assertions. All 372 unit + 107 integration + 53 Python tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version from 1.3.0 to 1.4.0
Minor bump for the new per-page markdown extraction API exposed through
the Python and Node bindings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: improve heuristic table detection for numeric columns and multi-line headers
Two fixes for tables that have clean extractable text but fail heuristic
structure detection:
1. Numeric column merge pass (grid.rs): After initial X-position
clustering, adjacent clusters are merged when one is sparse (header
text) and the other is dense with >50% numeric items (data column).
Multi-line wrapped headers often land slightly offset from their
data column — the merge closes gaps within 1.5× the clustering
threshold. New is_numeric_text() helper matches decimals, percentages,
negative numbers, and comma-separated thousands.
2. Duplicate-header skip (detect_heuristic.rs): Spanning super-headers
like "First Degree | First Degree | Higher Degree" contain duplicate
cells that trigger looks_like_partial_table_ex rejection. Now skips
rows with duplicate cells when a better header candidate exists
within the next 3 rows (higher fill ratio or numeric cells).
Tested on BITS Pilani university report (430 pages, 314 table pages).
Page 4 (multi-line header + numeric data) previously returned
needs_ocr=true; now correctly detects the table structure.
Eval: 197 PDFs, zero regressions, all 104+ tests pass, zero clippy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* bump version to 0.7.1
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Combine per-page markdown extraction with layout classification into a
single parse. extractPagesMarkdown now returns PagesExtractionResult with
pages_with_tables, pages_with_columns, pages_needing_ocr, and is_complex
alongside the per-page markdown — eliminating redundant PDF parses for
callers that need both.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* add extract_pages_markdown_mem for per-page markdown extraction
Enables hybrid OCR pipelines to skip GPU render+layout for simple text
pages by providing per-page markdown with needs_ocr flags. Font stats
are computed document-wide for consistent header detection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* bump napi package version to 0.6.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a new function that takes a PDF buffer and page+bbox regions (same interface
as extractTextInRegions), runs heuristic table detection on items within each region,
and returns markdown pipe-tables. Falls back to needs_ocr=true when no table
structure is found or text quality is suspect.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pre-scan lines to identify "isolated" ones — short lines (1-6 words)
with paragraph breaks both before AND after. These are heading
candidates even at body font size, common in academic papers
("Acknowledgements", "Limitations", "B.3 Prompt Engineering").
Inspired by opendataloader's HeadingProcessor which passes prevNode
and nextNode context to the heading probability scorer.
The isolated signal (+0.3) combines with rarity/bold/standalone
signals. A per-page density guard prevents false positives on
multi-column pages where many lines appear isolated. Continuation
word detection (ending in "the", "and", etc.) filters wrapped
paragraph lines.
MHS=0 docs: 18→13. MHS-S +0.004. No regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reverts commits 937311c, 1dcb0c6, 0300e96, 999f9a2. The thin-rect-to-line
synthesis and stacked table splitting improved extraction for specific
government PDFs but caused -0.05 TEDS regression on the benchmark by
preempting the heuristic detector with worse line-based grids.
These features need more targeted guards before re-enabling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rows with 3+ short-valued cells (avg ≤10 chars) and an empty first cell
are column headers (e.g. "UR | SC | ST | OBC | EWS"), not text overflow
from the previous row. Prevents them from being merged into the
preceding section title row.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three fixes for better table extraction from spreadsheet-exported PDFs:
1. Convert thin filled rects (< 2pt) to PdfLine objects before line-based
table detection. Many PDFs draw table borders as narrow filled rectangles
instead of stroked paths — these were invisible to the line detector.
2. Relax uniform row spacing rejection (CV 0.05 → 0.02). Spreadsheet
exports have very even row heights that were being rejected as "chart
grids".
3. Fix continuation row merging: don't merge rows where the only non-first
cell content is a long label (section headers like "Category No. 03").
Don't merge first-cell-only rows with long text ("Note: ...").
Also adds multi-Y row splitting in line-based detection and column-aware
table detection skipping for multi-column pages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace ad-hoc bold/ratio heading checks with a unified scoring system
based on font size rarity. For each line, compute:
score = font_rarity * 0.5 + bold * 0.3 + standalone * 0.2
Font rarity measures how infrequently a font size appears across the
document — heading fonts are rare while body text is common. This
approach (from opendataloader's ModeWeightStatistics) naturally adapts
to each document's font distribution instead of relying on fixed
thresholds.
Guards: require font_size >= 0.95 * base_size (no small-font headings),
word_count >= 3, and standalone (paragraph break before).
Benchmark improvement: MHS 0.56→0.58, MHS-S 0.66→0.70, overall +0.003.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lines with font size 1.10-1.20x body text that are standalone and short
(1-8 words) are promoted to headings. This catches academic paper
headings where the font is only ~10% larger than body text, below the
previous 1.2x threshold.
Also syncs the simpler to_markdown_from_lines path to match the
table-aware path (removes stale colon exclusion).
Benchmark improvement: MHS 0.54→0.56, overall 0.761→0.766.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Caption detection was incorrectly classifying "Table of Contents" as a
caption because it starts with "Table ". Now "Table" and "Figure"
prefixes require a digit, parenthesis, or hash after them — matching
actual captions like "Table 1", "Figure 3.2" but not titles.
Also removes debug logging left from previous iteration.
Benchmark improvement: MHS 0.52→0.54, overall 0.757→0.761.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The colon exclusion was preventing legitimate headings like "Steps for
Using the Microscope:" and "Changing objectives:" from being detected.
The single edge case it was protecting (chart sub-headers) is less
impactful than the many headings it was blocking.
Benchmark improvement: MHS 0.51→0.52, MHS-S 0.61→0.62.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Lower minimum item count for body-font table candidates from 9 to 6,
allowing small 2-3 row tables to be detected.
- Allow 2-column body-font tables with short cells (avg ≤25 chars) to
bypass the "table-like content" validation. This catches text-only
definition/category tables (e.g., species lists) without false-positiving
on 2-column paragraph text (which has longer cells).
Benchmark improvement: TEDS 0.498→0.519, overall 0.750→0.754.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bold lines at body font size that are standalone (preceded by a paragraph
break) and have ≥3 words are promoted to headings. This catches the
common pattern in academic/technical PDFs where section headings use
bold text at the same size as body text.
Guards against false positives: minimum word count, colon-ending
exclusion (labels like "Table I:").
Benchmark improvement: MHS 0.37→0.50, overall 0.71→0.75.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align region filtering with rotated-page coordinate rewrites, switch region text assembly to the shared line-grouping pipeline, and retain edge-overlap text to avoid false empty regions that incorrectly trigger OCR fallback. Also make Python region inputs fail fast with clear ValueError messages for malformed boxes.
Made-with: Cursor
Two bugs in collect_text_in_region / extract_text_in_regions_mem:
1. The threshold-based sort comparator in collect_text_in_region was not
transitive, causing Rust's sort to panic on certain PDFs. Replaced with
strict total_cmp ordering — the line-grouping phase already handles
fuzzy Y matching via threshold.
2. The needs_ocr check was missing is_cid_garbage, so Identity-H fonts
with CID garbage (C1 control chars, high Latin mojibake) could pass
all quality checks and be served as real text with needs_ocr=false.
Also adds 7 integration tests for extract_text_in_regions_mem (previously
had zero coverage): basic extraction, Identity-H needs_ocr, multiple
regions, nonexistent page, empty region, invalid input, and a fast-vs-normal
comparison test across all text-based fixtures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both bindings now expose the same 6 function families: process, detect,
classify, extractText, extractTextWithPositions, and extractTextInRegions.
Bumps PyO3 from 0.22 to 0.25 for Python 3.14 support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(tables): improve heuristic detection for borderless wrapped-cell tables
Three changes to the body-font heuristic detector:
1. Adaptive Y-gap in find_table_regions_strict: use median qualifying-row
spacing × 3 instead of fixed 25pt. Tables with wrapped cells have
larger gaps between qualifying rows (those with 3+ X-clusters).
2. Y-only region filtering: use full X range when collecting region items.
The strict X bounds from qualifying rows excluded continuation lines
in wrapped cells, starving find_column_boundaries of items.
3. Merged-band retry: when split_side_by_side splits a page into bands
but no band produces a table, retry heuristic detection on all items
merged as a single band.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): gap-histogram column detection for small tables + lower avg_cells
Two changes to fix PDF 045 (borderless table with narrow "No." column):
1. Extend gap-histogram column threshold to small tables: when the gap
between within-column jitter and between-column spacing is >10pt
(unambiguous bimodal signal), use the detected threshold even with
fewer than 500 items. Previously only triggered for dense tables.
2. Lower BodyFont avg_cells_per_row minimum from 2.5 to 2.0 to handle
tables with wrapped multi-line cells where continuation lines have
only 1 filled cell.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): trim empty outer columns + relax partial H-line validation
- Rect detection: trim empty first/last columns instead of rejecting
the whole table. Rect edges often extend beyond text boundaries.
- Line detection: accept tables with 6+ partial horizontal lines
(>15% width) when <3 full-spanning lines exist. Handles tables
with column-level separators instead of full-width rules.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): cell-rect fallback for tables with variable-width backgrounds
When rect clustering produces a grid that fails validation (empty
interior columns from variable-width cell backgrounds), fall through
to a new strategy: use rect Y-edges for row boundaries and text
X-position clustering for columns. This handles tables like the
opendataloader-bench 088-090 comparison tables where each cell has
its own background rect at different widths.
Also widen failed-cluster hint width cap for large clusters (≥30 rects)
to allow page-spanning table regions.
TEDS score on opendataloader-bench: 0.300 → 0.353.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): relax vertical line spanning validation for partial borders
Accept tables with 4+ partial vertical lines (>10% table height) when
fewer than 2 span >30%. Handles tables like opendataloader-bench 053
with column-level vertical separators that don't extend the full height.
TEDS: 0.353 → 0.377 on opendataloader-bench.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): lower cell-rect density threshold + add validation logging
- Lower cell-rect density minimum from 25% to 15% to accept sparser
tables with decorative backgrounds (fixes 147).
- Add debug logging to all heuristic validation paths for diagnosability.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): relax body-font validations for text-only and 2-column tables
Four fixes closing 71% of the TEDS gap vs opendataloader:
1. Validation 7 (table-like content): bypass numeric content requirement
for tables with 3+ columns that passed all structural checks. Text-only
tables (category lists, program descriptions) are legitimate.
2. Qualifying row threshold: lower from 3+ to 2+ X-clusters per row.
Enables 2-column body-font table detection (fixes 166).
3. Row-stripe max cell length: raise from 500 to 2000 for 3+ column
tables. Tables with paragraph descriptions in one column are valid
(fixes 121).
4. Row-stripe empty-column trimming: apply the same outer-column trim
as grid detection (fixes 121 column-0 rejection).
TEDS: 0.377 → 0.438 on opendataloader-bench (gap: -0.056 vs odl).
TEDS=0 docs: 14 → 9.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): enable 2-column body-font tables + lower all minimums
- Lower BodyFont minimum columns from 3 to 2 in detect_table_in_region
- Lower BodyFont minimum rows from 3 to 2
- Lower avg_cells_per_row minimum from 2.0 to 1.5 (handles wrapped cells
in 2-column tables)
- Apply empty-outer-column trimming to row-stripe detection (not just grid)
TEDS: 0.438 → 0.468 on opendataloader-bench (gap: -0.027 vs odl).
TEDS=0 docs: 9 → 8. 86% of original gap closed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): text-based row fallback + fix 120 flow-chart and 188 leaderboard
Three changes that push TEDS past opendataloader:
1. Cell-rect Y-edge fallback: when rects have too few Y-edges for row
structure, derive rows from text Y-position clustering within the
rect bounding box. Fixes flow-chart tables (120) and column-header-
only rects (188).
2. Lower cell-rect minimum from 20 to 6 rects to catch smaller tables.
3. Relax validation 1 (first-column presence) from 50% to 25% of rows.
Tables with wrapped model names have continuation lines without first
column content.
TEDS: 0.468 → 0.508 on opendataloader-bench.
Now BEATS opendataloader (0.508 vs 0.494, gap=+0.014).
TEDS=0 docs: 8 → 5.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): wrapped-cell continuation row merging
Merge rows that have fewer filled cells than the header row into the
previous row. Handles wrapped multi-line cells where text overflow
creates extra rows (e.g., "Direct" + "communications" → "Direct
communications").
Conditions: fewer filled cells than header, more than previous row had,
not a data row (numeric), not a short subheader label.
TEDS: 0.508 → 0.522 on opendataloader-bench (now +0.028 vs odl).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): tighten cell-rect validation + fix continuation-row merging
Cell-rect false positives:
- Raise density threshold back to 25% (from 15%)
- Add max cell length check (500 chars) to reject paragraph content
- Reject disproportionate grids (>20 rows, <4 cols)
Continuation-row merging:
- Wide tables (5+ cols): only merge rows with ≤50% header cells
- Narrow tables (2-4 cols): merge rows with fewer cells than header
- Prevents merging normal data rows in large tables (6_KE_Chart)
while keeping wrapped-cell merging for narrow tables (178)
TEDS: 0.498 on opendataloader-bench (still +0.004 vs odl).
pdf-evals: 191/192 passed, 0 regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Struct-tree tables with incomplete page tagging (e.g., only 22 of 50+
rows tagged on a page) would claim items and block rect detection,
leaving unclaimed items as loose text. Now require struct-tree tables
to capture ≥50% of band items before using them; incomplete trees
fall through to geometry-based detection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skip rect-detected tables that overlap with items already claimed by
struct-tree detection. Previously both strategies emitted separate
tables for the same content, doubling the output on tagged PDFs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a PDF has a well-formed structure tree with /Table > /TR > /TD|TH
elements linked to MCIDs, build tables directly from the semantic
hierarchy. Runs as highest-priority detection (step 0) before rect-based,
line-based, and heuristic strategies.
- Add StructTree::extract_tables() to walk the tree and collect table
descriptors with row/cell/MCID info
- Add detect_tables_from_struct_tree() to match MCIDs to TextItems
- Reject tables with <30% MCID cell coverage (stale structure trees)
- Update 2013-app2 snapshot (struct-tree gives valid but different
column ordering)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(text): handle Tc/Tw character and word spacing in text width computation
PDFs using Tc (character spacing) and Tw (word spacing) operators for text
justification had words incorrectly split across TextItems. The computed
advance width didn't account for these spacing parameters, causing spurious
spaces mid-word (e.g. "deve lopers" instead of "developers").
- Add Tc/Tw operator handling and graphics state save/restore
- Incorporate char_spacing and word_spacing into compute_string_width_ts
- Add adaptive merge threshold: tighter for lowercase→lowercase junctions,
wider before joining punctuation
- Add unit tests for Tc/Tw width computation and merge behavior
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(fonts): add large Tc width computation test
Verifies that large character spacing values are applied in full
without any artificial cap, matching PDF spec behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(text): guard against Tc/Tw-inflated widths in merge and join paths
Two targeted fixes to prevent character-spacing (Tc) and word-spacing
(Tw) inflation from causing data quality regressions:
1. should_join_items: reject large negative gaps (< -font_size) that
arise when Tc/Tw inflate item widths past adjacent items. Fixes
FY_2015 merged numbers (e.g. "239.696.0" → "239.69 6.0").
2. merge_text_items: cap effective width for gap computation when Tw
inflates space-containing items beyond 0.85× font_size per char.
Prevents column-level gaps from collapsing into merge range,
recovering table detection for Baldwin-Edwards and similar PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
PDFs that embed landscape content in portrait pages via a rotated text
matrix (e.g. [0, b, -b, 0, tx, ty] for 90° CCW) produced garbled output
because the layout engine assumed x=horizontal, y=vertical.
Track the dominant text direction from combined matrices during extraction.
When ≥67% of text operators are rotated, swap x↔y coordinates (with
y-negation for correct reading order) for all text items, rects, and lines.
Also estimate text widths from char count × font size since scale_x ≈ 0
for rotated text.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
