Compare commits

...
Author SHA1 Message Date
Abimael Martell 0a8fe61824 chore: mirror legacy npm package version 2026-06-23 13:51:30 -07:00
Abimael Martell 1fd7a93db9 Merge origin/main into legacy npm publish 2026-06-23 13:34:09 -07:00
Abimael Martell c490e5efcf chore: publish legacy npm wrapper 2026-06-23 13:26:26 -07:00
Abimael Martell 57b98c6a5d fix(extractor): flag garbled text spans for OCR (#108)
* fix(extractor): flag garbled text spans for OCR

* fix(extractor): apply text quality checks to regions

* chore(napi): bump npm package version
2026-06-23 13:12:10 -07:00
Abimael Martell f25808e0a7 fix(extractor): restore CID font state for Chinese text (#106)
* fix Chinese CID text decoding

* bump package versions
2026-06-20 19:43:27 -06:00
Abimael Martell 9360c8464d ci: add crates trusted publishing (#103) 2026-06-05 11:21:55 -07:00
Abimael Martell 252d87ac58 docs(readme): add package badges and install docs (#102)
* docs: add crates.io install instructions

* docs: add npm badge
2026-06-05 11:01:56 -07:00
Abimael Martell 85890648c9 chore: use crates.io lopdf (#101) 2026-06-05 10:50:05 -07:00
Abimael Martell 6e55e38b55 fix(markdown): handle wrapped bold abstracts (#100)
* fix(markdown): handle wrapped bold abstracts

* chore: bump napi package version
2026-06-01 14:59:18 -07:00
Abimael Martell 42befcea57 fix(pdf-inspector): recover wrapped key-value tables (#99)
* fix(pdf-inspector): recover wrapped key-value tables

* fix(pdf-inspector): satisfy clippy
2026-06-01 10:10:55 -07:00
Abimael Martell e547f616f9 fix(pdf-inspector): recover key-value region tables (#98) 2026-05-29 18:55:38 -07:00
Abimael Martell 455dfe5a74 fix(pdf-inspector): recover borderless region tables (#97) 2026-05-28 10:20:06 -07:00
Abimael Martell 839317525b fix(pdf-inspector): relax vector table confidence gates (#96) 2026-05-27 11:45:00 -07:00
Abimael Martell 8b63ceb084 emit ItemType::Image bboxes for Image XObjects (was: silently dropped) (#94)
* emit ItemType::Image bboxes for Image XObjects (was: silently dropped)

Background. ItemType::Image, MarkdownOptions::include_images, and the
markdown emitter's image-collection path have all been in the tree
for a while, but no producer ever populated them — content_stream.rs
explicitly `// Skip images — text extraction only` at the Do
operator, and the nested Form-XObject walker in xobjects.rs only
matched XObjectType::Form, silently dropping Image entries. The
declared types were dead code.

This PR lights them up. At every Do that resolves to an Image
XObject (both top-level and nested inside Form XObjects), we now
compute the page-space bbox from the current CTM via a new
`image_bbox_from_ctm` helper — handling both axis-aligned and
rotated/sheared placements via 4-corner AABB — and emit a TextItem
with `item_type: ItemType::Image` and the legacy `[Image: <name>]`
text payload that the markdown emitter already knows how to render.

Callers can now find raster figures via `extract_text_with_positions`
(and the `_mem` variant, newly re-exported at the crate root) without
needing to re-parse the PDF or run a vision/layout model. The intended
consumer is layout-aware text pipelines that want to crop figures and
caption them out-of-band.

Two backstops to avoid silent breakage for existing callers:

  1. `MarkdownOptions::include_images` default flipped `true → false`.
     If it stayed at `true`, every existing user of
     `extract_pages_markdown` would suddenly see `![Image: Im0](image)`
     placeholders inserted throughout their output the moment they
     upgraded. Image data is still available structurally via
     `extract_text_with_positions`; rendering it into markdown is now
     an opt-in. New regression test asserts `extract_pages_markdown`
     output is unchanged for the image-bearing fixture.

  2. Image items now also skip the layout heuristics
     (`detect_columns`, `detect_tables_from_rects`) via a new
     `is_text_layout_item` predicate. Without this filter, an image's
     left edge would land in the column-projection profile and skew
     table column detection — surfaced by
     `vector_grid_tests::upstage_key_functions_four_cols` going from 4
     detected columns to 5 in CI before the filter was added.

Re-exporting `extract_text_with_positions_mem` at the crate root —
strictly additive; mirrors how `extract_pages_markdown_mem` is already
available there.

Tests:

  - test_extract_text_with_positions_emits_image_bboxes — minimal PDF
    with one 200×100 image at (50, 600); asserts one Image item with
    correct bbox + page + text.
  - test_image_xobject_bbox_handles_rotated_ctm — 90° rotated image
    via shear-component CTM; asserts AABB is correct (handles non-
    axis-aligned placements via 4-corner clamp).
  - test_image_emission_does_not_change_default_markdown — asserts no
    `Image:` token leaks into default markdown output, regression
    guard for the include_images flip.
  - test_markdown_options_default_has_include_images_false — explicit
    sentinel so anyone flipping it back catches it in CI.

* Bump version from 1.8.15 to 1.9.0
2026-05-20 13:43:05 -07:00
Abimael MartellandClaude Opus 4.7 73cffed1da detect_tables: catch narrow-column undercount via text-cluster topology (#93)
text_cluster_column_undercount previously fired only on tables with
6+ markdown columns. That missed a common production failure shape:
4-column page geometry where the heuristic detector's x-position
clustering collapses 2 narrow numeric columns (dates, IDs, amounts)
into adjacent wider columns, producing 2-column markdown.

The original 6-column floor existed because raw x-cluster count is
noisy on small tables — wrapped continuations, bullet indents, and
within-cell text variation produce many small x-clusters that don't
correspond to real columns. Examples:

  - Pcmso-style 2-column key/value layout with multi-line values
    shows 10 raw x-clusters but only 1 has more than a single item.
  - Yale-style archival catalog with 3 columns shows 6 raw
    x-clusters but most clusters are single-item continuations.

Replace the raw cluster count with a "significant cluster" count:
clusters whose item count is at least 1/4 of the dominant cluster
(and ≥2 items). That filters the within-cell-variation noise while
preserving signal from real columns, which consistently have one
item per row.

With significant-cluster counting:
  - Narrow-undercount fires when significant_clusters >= 3 AND
    significant_clusters >= table_cols * 2 (catches 4-col-collapsed-
    to-2 cases without misfiring on pcmso 10-clusters or yale 6
    where the significant subset matches markdown).
  - Legacy wide-undercount path (table_cols >= 6) keeps the same
    +2 / 1.2x thresholds, now applied to significant clusters
    instead of raw clusters.

Verified against representative regression and must-not-regress
cases from prior PR cycles: narrow-undercount catches genuine
column-drop cases (significant=4, markdown=2 → routes to OCR);
must-not-regress fixtures (italian-gov 6520 chars, pcmso 1733
chars, yale 1754 chars, doc200/182/189 still routing to OCR via
PR #91 guards) unchanged.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 17:26:35 -07:00
Abimael MartellandClaude Opus 4.7 647ea5c7cd extract_tables: reject font-decode failures via text-density floor (#92)
Some PDFs have fonts whose ToUnicode CMap is missing or broken —
Identity-H fonts without unicode metadata, Type-3 fonts where every
glyph maps to garbage. The page extractor returns punctuation-only
fragments or single-glyph repeats; the rendered image still carries
the visible text, so the region should fall back to OCR rather than
serve a partial table.

The existing captured_only_a_fragment guard can't catch this case
because region_text_chars itself collapses under font-decode failure —
captured vs extracted is symmetrically low, and the ratio still looks
acceptable.

Add a complementary area-based density guard: when a region has lots
of pixel real estate but very few text chars, the page extractor
hit a font failure. Bbox area is independent of extraction success,
so the symmetry breaks.

Threshold 0.003 chars/sq pt sits between observed clean extractions
(≥0.005 on full-page A4 ledgers, key/value layouts, archival
catalogs) and observed font-decode failures (≤0.0014 on prod-traffic
samples). Three guards keep it from misfiring:
  - text_chars < 20 skipped: synthetic / fragmentary fixtures
  - area < 30,000 sq pt skipped: tiny stat blocks
  - area > 400,000 sq pt skipped: near-whole-A4 bboxes where
    density is unreliable (large white-space margins)

Verified against three reproducible cases from prod shadow logs
that previously served partial output:
  - Cyrillic page with punctuation-only decode (46 chars, density
    0.00045) → flagged, routes to OCR
  - Cyrillic page where every glyph collapsed to one letter (89
    chars, density 0.00025) → flagged, routes to OCR
  - Materials-test region where text extracted fine but the table
    body extends beyond the bbox (96 chars, density 0.00131)
    → flagged, routes to OCR

Existing fixtures (full-page A4 ledger, multi-row key/value with
paragraph values, archival catalog, bits_pilani whole-page tests,
synthetic line-grid test) all retain identical behavior.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 11:51:23 -04:00
Abimael Martell 36445595d0 Bump version from 1.8.12 to 1.8.13 2026-05-14 22:30:38 -04:00
Abimael Martell 0932ce331b [codex] harden native table selection against undercounts (#91)
* fix hierarchical table row merging

* fix table undercount prose fragments
2026-05-14 22:09:09 -04:00
Abimael MartellandClaude Opus 4.7 a6e59de8f0 ci: switch to Swatinem/rust-cache to fix macOS build (#90)
Caching ~/.cargo/bin/ via actions/cache@v4 was poisoning the cargo
shim on macOS runners — restored cargo resolved to rustup-init and
failed with "unexpected argument 'build' found". Swatinem/rust-cache
skips that directory and handles target/ pruning + key derivation.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 15:08:28 -04:00
Abimael MartellandClaude Opus 4.7 b22bd759ab Add SECURITY.md with private vulnerability reporting policy (#89)
Documents how to report security issues privately (help@firecrawl.dev
or GitHub's private advisory flow) and what is in/out of scope, so
researchers don't disclose publicly via GitHub issues.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 15:01:39 -04:00
Abimael MartellandClaude Opus 4.7 7539868bf8 extract_tables: reject partial extractions in needs_ocr gate (#87)
After enabling the vector-grid detectors on the extract path (#85)
and broadening detection to full-page grids (#83), long-cell tables
(#84), and segment-only layouts (#86), one residual failure shape
remained: detectors finding a valid grid but only capturing a small
fraction of the region's actual text. Two recurring sub-shapes:

  - "header-only": detector captured the column-header band (often a
    multi-line year/units block) but missed every data row below.
    Common in financial statements, securities tables, budget
    appendices.
  - "sparse": detector returned a handful of fragmentary cells from a
    content-rich region, missing the bulk of the page.

Both pass the existing needs_ocr quality gates — the captured cells
are well-formed markdown — but the customer would receive a 5-row
fragment of a 50-row table. Today these regions fell back to GLM-OCR
by default; flipping `__nativeTableExtraction=true` would start
serving the partials.

Add `captured_only_a_fragment(md, region_text_chars)`: rejects when
the captured non-delimiter character count is less than 25% of the
text the page extractor saw inside the region. The 200-char region
floor keeps short legitimate tables (units, axis labels) from being
mis-flagged. Wired into the existing `evaluate` quality gate
alongside is_garbage_text / is_cid_garbage / detect_encoding_issues
/ looks_like_partial_table_ex.

Verified against three representative residual cases from shadow
logs (financial-statement header band, securities-table fragment,
ESIA sparse region): all flip from `needs_ocr=false` with partial
output to `needs_ocr=true` so GLM takes over. Existing full-table
fixtures (governmental ledger, PPRA-style key/value, archival
catalog) still pass through unchanged.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 14:20:02 -04:00
Abimael MartellandClaude Opus 4.7 79d75dbdca detect_lines: derive column edges from horizontal-segment endpoints (#86)
Some catalog and archival-finding-aid tables draw each row's
horizontal rule as N segments (one segment per cell) with no vertical
lines at all. The previous detector rejected these outright at
`verticals.len() < 2`, even though the segment break points encoded
the column boundaries unambiguously.

When the vertical-line count is below the existing threshold, walk
the horizontal-segment x-endpoints and cluster them with the same
snap_edges path used for vertical-line columns. Accept the derived
edges only when ≥3 distinct x-positions each appear on ≥50% of the
unique horizontal-line rows — that consistency guard distinguishes
real per-cell segments from decorative rules with varying widths
(which never share endpoints across many rows).

When columns come from segment endpoints, skip the downstream
"spanning_v / partial_v" gate (there are no vertical lines to
validate against). All other gates — horizontal-span coverage,
content density, capture ratio, multi-column distribution, the
uniform-spacing chart-grid rejector — still apply.

Verified on a 7-row × 3-col archival catalog page that previously
extracted 98 chars (a 2-row fragment via the heuristic fallback); now
extracts 1754 chars with all rows + multi-line cells.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 10:21:09 -04:00
Abimael MartellandClaude Opus 4.7 b5b91470db extract_tables: try vector-grid detectors before text heuristic (#85)
* extract_tables: try vector-grid detectors before text heuristic

extract_tables_in_regions_mem previously ran only the text-only
heuristic detector (tables::detect_tables) on the items inside each
region, discarding the rects and lines that extract_page_text_items
returned. That left the rect-backed and line-backed detectors
(detect_tables_from_rects, detect_tables_from_lines) unused by the
public region-scoped extraction path — they only ran through
detect_vector_grid_in_region_mem, which most callers don't use.

Keep the rects and lines, filter them to each region, and try in
order: rect detector → line detector → heuristic. Each candidate's
markdown is quality-gated by the existing needs_ocr checks
(is_garbage_text, is_cid_garbage, detect_encoding_issues,
looks_like_partial_table_ex); only the first clean output wins.
If all three produce empty or noisy output we still return
needs_ocr=true, matching prior behavior.

Effect on real prod-shape inputs from shadow logs:

  Full-page ruled ledger, 6 cols x ~15 rows:
    before: heuristic emits a 355-char two-row fragment
    after:  line detector emits the full 6520-char table

  Multi-row key/value layout with paragraph values:
    before: heuristic emits a 188-char header-only fragment
    after:  rect detector emits the full 1733-char table including
            the multi-bullet description cell

Existing fixtures that already passed via the heuristic continue to
pass: the quality gate rejects partial vector-grid output and falls
through, so the heuristic still wins where it produced the cleaner
result.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Bump version from 1.8.9 to 1.8.10

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 19:25:08 -04:00
Abimael Martell f26efe6673 Bump version from 1.8.8 to 1.8.9 2026-05-12 18:40:12 -04:00
Abimael MartellandClaude Opus 4.7 06ccaf5732 detect_rects: accept multi-row tables with long-content cells (#84)
Three rejection sites in detect_rects.rs killed any candidate grid
where a single cell exceeded 500 chars:

  - detect_row_stripe_table (line 1399)
  - detect_row_stripe_table_from_cell_rects (line 1729)
  - detect_merged_cluster_table (line 2154)

The intent was to skip layout-background rects — sidebars, banners,
section bands — where one big rectangle wraps a paragraph of prose.
Those almost always present as ≤3 row stripes (header / body / footer
or single big block).

Multi-row key/value tables with paragraph-length values in one column
present the same cell-length signal but are legitimate tables.
Gating the rejection on `non_empty_rows < 4` preserves the
layout-background guard for narrow stripe layouts while letting
through multi-row tables with descriptive content.

Verified against a 9-row × 2-column key/value layout where the value
column has multi-line content (~1.4KB in the longest cell). Before:
rejected with `max cell length 1384 > 500`. After: accepted with 89%
density. The existing `test_row_stripe_rejects_layout_background_long_cells`
regression test for narrow stripe layouts still passes.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 18:35:56 -04:00
Abimael MartellandClaude Opus 4.7 5a84c29ea4 detect_lines: accept full-page tables with internal grid (#83)
The page-spanning-frame guard rejected any line set whose bounding box
exceeded ~90% of a standard A4/Letter page in both axes. The intent was
to skip decorative outer borders, but it also threw away every real
full-page table — common in governmental ledgers, financial filings,
and dense report layouts.

Decorative borders have just 4 edges (top/bottom/left/right). Real
full-page tables have many internal row and column rules. Gate the
rejection on `horizontals.len() <= 4 && verticals.len() <= 4` so the
guard still catches bare frames but lets through line sets with real
internal grid structure.

Tested against a regione.lazio.it Estrazione-provvedimenti page (full
A4-width table, 14 rows × 6 cols): now ACCEPTED with 211/211 items
captured. Bare-frame regression test added.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 18:28:08 -04:00
Abimael MartellandClaude Opus 4.7 96c0f2102a tables/detect_rects: prefer rect-grid edges over text-cluster on N≥3 column tables (#82)
When a wire-bordered table has headers centered/right-aligned in their
cells but data left-aligned, cluster_x_positions can both merge adjacent
data columns (when the data-to-data gap is below the clamped threshold)
and drop the header-only x-positions in its singleton-filter pass. The
cell-rect fallback then used text-cluster column edges and lost a column
or fragmented neighbor cells.

Prefer rect-derived column edges when the rect grid has 3+ columns and
every rect column holds multiple text items. The all-cols-populated
check protects against decorative or background rects (prose laid out
in a frame, cell-fill rects with extra borders) that would otherwise
split a logical column into spurious sub-columns. The existing
prose-in-frame, well-distributed-columns, and wireless-prose guards
still fire for the cases they were built for.

Bump napi version 1.8.7 → 1.8.8.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:44:26 -04:00
Abimael Martell 6086577d11 tables/detect_rects: emit grid for multiline indented cells (#79) 2026-05-07 11:44:53 -07:00
Abimael MartellandCursor f2186ec1aa tables/detect_rects: don't accept relaxed grid on wireless prose (#78)
Require rect-derived column evidence before relaxing prose checks for two-column cell-rect fallbacks, so text-position alignment alone cannot synthesize a vector grid on wireless content.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-07 09:30:29 -07:00
Abimael MartellandClaude Opus 4.7 59b17f372a tables: tighten prose-in-frame rejection (#77)
* tables: lift detection on shaded-header + alt-row tables (#wired-grids)

Production telemetry on `wired_high_confidence`-classified table regions
showed `detect_vector_grid_in_region_mem` returning a usable grid only
~27% of the time, with the rest falling through to GLM-OCR. Three
surgical fixes target the dominant production shapes:

* Path-fill cell backgrounds: when the page has no `re` rects but draws
  cell backgrounds via `m`/`l`/`h`/`f*` sequences, prefer the fill-derived
  rects over the few section-level `W*` clip paths that previously won
  the priority gate. Activated when fill rects outnumber clip rects ≥3×.

* Dedup-induced cluster splits: page-background rects could pose as
  containers in the sub-rect dedup and evict a slightly smaller
  table-frame rect, breaking adjacency between column-cell groups so each
  column became its own cluster. Origin-anchored containers are now
  disqualified from sub-rect dedup. A separate exact-duplicate pass
  collapses the cell-padding/text-bg/cell-border triple emissions some
  PDFs produce, preserving original order to avoid reshuffling table
  output on multi-table pages.

* Prose-words rejection: the `cell-rect` fallback's whole-grid prose
  threshold also rejected real tables that include a description column.
  Now relaxed when content is well-distributed (≥75% of cols filled),
  while keeping the original strictness for prose-in-a-frame layouts.

Two regression fixtures from the opendataloader-bench corpus, covering
the dominant production failure categories:

* `greencomp_competence.pdf` — 2-col shaded-header + plain-body glossary.
  Mirrors production crops #1 (Contractions glossary) and #6 (BIO 350
  course header).
* `upstage_key_functions.pdf` — 4-col shaded-header + alt-row backgrounds
  + merged left column. Mirrors production crops #2 (Parameter/Value
  alt-row), #7 (Spanish XML schema), and #8 (Córdoba multi-row header).

Existing fixtures stay green (doc 51 wrapped-label, doc 128 forecast
six-cols, td9264 snapshot). 133 unit + integration tests pass; clippy
clean.

Bumps napi/package.json 1.8.4 → 1.8.5.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* tables: tighten prose-in-frame rejection — fixes pdf-evals #30 regression

PR #76's shaded-header detection lift surfaced a regression on
accessory_building_permit_application_1 (TEDS 0.10 → 0.05): a
paragraph of legal text laid out in a 2-column justified block was
being admitted as a 10×2 fake table where every cell holds a
sentence fragment ("I agree to comply...", "I", "It is the property
owner's responsibility..."). Per pdf-evals PR #30 review, this is
the kind of regression production users will notice — the markdown
is structurally and semantically misleading.

Root cause: PR #76's prose-rejection only fires for `num_cols >= 4`,
so the 2-col prose-in-a-frame case slipped past it entirely. The new
fill-priority + dedup changes started producing rects for this layout
that 1.8.4 correctly ignored.

Fix: tighten the prose-in-frame check.
- Lower the column-count guard from `>= 4` to `>= 2`.
- Add a content-length signal as the primary discriminator: when the
  prose-words trigger fires AND mean non-empty cell length exceeds
  65 chars, reject regardless of column distribution.

The 65-char threshold cleanly separates observed cases:
  accessory_building (prose-in-frame): mean 74 chars  → REJECT
  upstage_key_functions (real 4-col table): mean 53   → admit
  greencomp_competence (real 2-col glossary): mean 20 → admit
  accessory_building (real 5×3 form data): mean 10    → admit

The well-distributed-cols relaxation that PR #76 added stays —
"label / value / description / benefit" tables (#7, #8 from the
production crops) still pass, but only when their mean cell length
stays below the prose threshold.

New regression test `accessory_building_rejects_prose_in_frame` asserts
both that the real 5×3 form data table survives AND the 10×2 prose
block is rejected. Snapshot test `test_snapshot_td9264` updated to
match new output — old snapshot captured the same prose-in-frame bug
on regulatory text (paragraphs emitted as 3-col `||text||` fake-table
rows). New snapshot emits clean prose paragraphs, which is correct.

Verification:
- cargo test --all: 424 lib + 133 integration + 2 doc tests pass
- cargo fmt --check clean
- cargo clippy -- -D warnings clean (lib-level; pre-existing
  test-level clippy issues on the wired-grids branch unaffected)
- Existing fixtures stay green: forecast_table_chart_six_cols (PR
  #72), bits_pilani_* (PR #73), greencomp_competence_two_cols and
  upstage_key_functions_four_cols (PR #76).

This branch is based on abimaelmartell/wired-grids so it includes
PR #76's commits plus this fix on top. Suggest merging this and
closing #76, OR rebasing #76 to incorporate this fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* tables: drop early-dedup atom — caused broad TOC + matrix corruption

Bisected PR #76's 4 atoms against the SEC 10-K
0001104659-25-093871_183e44ac.pdf which appeared as a TEDS regression
in pdf-evals PR #30. Result:

  atom                       | TOC | perf-graph | qualifications
  ---------------------------|-----|------------|---------------
  fill-priority              |  ✓  |     ✓      |       ✓
  early-dedup                |  ✗  |     ✗      |       ✗
  page-bg disqualification   |  ✓  |     ✓      |       ✓
  prose-relaxation           |  ✓  |     ✓      |       ✓

Early-dedup was the SOLE source of all three regressions on this doc.
Tried a more conservative variant (≥3 copies only — pair-duplicates
appear in legit multi-section layouts like 10-K dividers above + below
section headers); didn't fix the regression. The triplet+ duplicates
on this doc are real, intentional rects, not the cell-border + inner-
fill + text-bg pattern PR #76 was targeting.

Drop early-dedup. Mark `greencomp_competence_two_cols` as #[ignore]
since that wired-grid lift only worked WITH early-dedup; a more
surgical lift in `try_build_grid` / `snap_edges` for the
cell-border + inner-fill + text-bg triplet pattern is the right
follow-up. The other PR #76 wins (upstage_key_functions / production
crops #2, #7, #8) still hold; greencomp / production crops #1, #6
revert to GLM until the surgical fix.

Validation on the regression doc:
  0001104659 TOC PART II markers:    4 (matches main, was 2 with PR#76)
  0001104659 perf-graph data row:    2 (matches main, was 1)
  0001104659 qualifications rows:    9 (matches main, was 6)
Validation on the prose-frame doc:
  accessory_building fake-table:     0 (matches main, was 1 with PR#76)
  accessory_building prose intact:   1 (matches main)

cargo test --all clean, cargo fmt --check clean, cargo clippy --lib
-- -D warnings clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 13:37:08 -07:00
Abimael Martell 88844d18be Bump version from 1.8.3 to 1.8.4 2026-05-02 21:29:51 -07:00
Abimael MartellandClaude Opus 4.7 cfc080f79a extractor: drop Latin-1 mojibake on Type0/CID fonts; tokenize wide TSR items (#75)
* extractor: drop Latin-1 mojibake on Type0/CID fonts; tokenize wide TSR items

Two text-extraction failure modes surfaced by table-candidate shadow
data; both also affect the existing TableFormer / vector-grid paths
since they share `extract_tables_with_structure_*_mem`'s downstream
cell-fill.

1. CJK / multi-byte mojibake. The bottom Latin-1 fallback in
   `extract_text_from_operand` ran unconditionally. For a Type0/CID
   (Identity-H) font whose ToUnicode CMap fails to parse, the bytes
   are CIDs (font-internal indices), not character codes — per-byte
   Latin-1 produces mojibake (e.g. 2-byte CID 0xCDD9 surfaces as "ÍÙ").
   Gate that fallback on `FontWidthInfo.is_cid` (set by
   `parse_type0_widths` for `/Subtype /Type0`). For Type0 fonts with
   any non-ASCII byte, emit one U+FFFD per CID instead so
   `detect_encoding_issues` still trips and the page is flagged for
   OCR — preserving the existing OCR-routing path that the
   high-Latin-1 garbage used to satisfy by accident. Type1 / TrueType
   simple fonts retain the per-byte Latin-1 round-trip (it IS the
   canonical interpretation for them; verified against an existing
   pdf-evals fixture where bytes like 0xB6 are legitimate Latin-1).

   Threaded `font_widths: &PageFontWidths` through
   `extract_text_from_operand` and its 5 call sites in
   `content_stream.rs` / `xobjects.rs`.

2. Dense-cell text collapse in `extract_tables_with_structure_cells_mem`.
   Stage-1 routing did per-item assignment — each TextItem went into the
   single cell whose bbox contained its center. When a row's text is
   rendered as one wide Tj (e.g. "Marshall Islands 0.9 0.9 0.9"), the
   whole row parks in one cell and the rest of the row stays empty.

   New `split_item_into_token_subitems` helper splits each item into
   per-token virtual sub-items with x positions estimated from
   `effective_width / char_count` and the token's character offset.
   Stage 1 then routes per-token. Single-token items collapse to a
   one-element vector (no behavior change). Multi-token items spanning
   multiple cells distribute correctly. Stage-2 orphan recovery now
   operates on token-grain orphans rather than re-trying whole items.

Tests:
 - `cid_font_with_unparseable_cmap_does_not_emit_latin1_mojibake` (unit)
   exercises the Type0/CID + unparseable-CMap fallback path.
 - `simple_font_latin1_fallback_passes_high_bytes_through` (unit)
   guards the false-positive case where a Type1 font's `/ToUnicode`
   reference is set but bytes are legitimate Latin-1 character codes.
 - `test_extract_tables_with_structure_distributes_wide_item_across_cells`
   (integration) builds a synthetic PDF with one wide Tj and asserts
   each token lands in its own cell.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* tests: pin CID mojibake fix mechanism with FFFD assertions

Two complementary tests for the Type0/CID Latin-1-fallback guard:

1. Tighten `test_identity_h_no_tounicode_suppresses_garbage` on the
   existing real-PDF fixture `shinagawa_identity_h.pdf` to also assert
   the pre-suppression text contains U+FFFD and contains no high-Latin-1
   chars. Pins down WHICH mechanism is suppressing the garbage so a
   future regression that re-enables Latin-1 mojibake fails loudly here
   instead of silently switching the suppression chain back to
   `is_cid_garbage` + high-Latin-1 detection.

2. Add `test_synthetic_type0_broken_tounicode_emits_fffd_not_latin1_mojibake`
   with a fully-synthetic Type0 / Identity-H PDF built in process. We
   control the malformed ToUnicode contents, the descendant CIDFontType2
   shape (just enough for `parse_type0_widths` to set `is_cid=true`,
   which is what the new guard keys off of), and the Tj byte stream.
   No fixture file or external license needed. Reproduces the exact
   "Type0 + non-ASCII bytes + unparseable ToUnicode" code path that
   produced the production mojibake samples.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 21:28:39 -07:00
Abimael Martell 1f28e523fd tables: keep wrapped labels in TSR output (#73) 2026-04-30 10:56:05 -07:00
Abimael Martell 97fc32ac70 tables: prefer rect edges for cell-grid fallback (#72)
* tables: prefer rect edges in cell-grid fallback

* Bump version from 1.8.1 to 1.8.2
2026-04-29 21:27:27 -07:00
40 changed files with 6165 additions and 301 deletions
+5 -31
View File
@@ -20,17 +20,7 @@ jobs:
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo
uses: actions/cache@v4
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
target/
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-
uses: Swatinem/rust-cache@v2
- name: Run tests
run: cargo test --verbose
@@ -61,17 +51,9 @@ jobs:
components: clippy
- name: Cache cargo
uses: actions/cache@v4
uses: Swatinem/rust-cache@v2
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
target/
key: ${{ runner.os }}-cargo-clippy-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-clippy-
key: clippy
- name: Run clippy
run: cargo clippy -- -D warnings
@@ -89,17 +71,9 @@ jobs:
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo
uses: actions/cache@v4
uses: Swatinem/rust-cache@v2
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
target/
key: ${{ runner.os }}-cargo-build-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-build-
key: build
- name: Build
run: cargo build --release --verbose
+87
View File
@@ -0,0 +1,87 @@
name: Publish Rust crate
on:
push:
branches: [main]
paths: ['Cargo.toml']
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
jobs:
check-version:
name: Check version change
runs-on: ubuntu-latest
outputs:
changed: ${{ steps.check.outputs.changed }}
published: ${{ steps.check.outputs.published }}
version: ${{ steps.check.outputs.version }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 2
- name: Check if version changed
id: check
run: |
NEW_VERSION=$(python3 -c 'import pathlib, tomllib; print(tomllib.loads(pathlib.Path("Cargo.toml").read_text())["package"]["version"])')
OLD_VERSION=$(git show HEAD~1:Cargo.toml | python3 -c 'import sys, tomllib; print(tomllib.loads(sys.stdin.read())["package"]["version"])')
echo "old=$OLD_VERSION new=$NEW_VERSION"
echo "version=$NEW_VERSION" >> "$GITHUB_OUTPUT"
if [ "$NEW_VERSION" = "$OLD_VERSION" ]; then
echo "changed=false" >> "$GITHUB_OUTPUT"
echo "published=false" >> "$GITHUB_OUTPUT"
exit 0
fi
echo "changed=true" >> "$GITHUB_OUTPUT"
HTTP_STATUS=$(curl --silent --show-error --output /tmp/crate-version.json --write-out "%{http_code}" \
-H "User-Agent: firecrawl/pdf-inspector publish workflow (https://github.com/firecrawl/pdf-inspector)" \
"https://crates.io/api/v1/crates/pdf-inspector/$NEW_VERSION")
case "$HTTP_STATUS" in
200)
echo "published=true" >> "$GITHUB_OUTPUT"
echo "pdf-inspector v$NEW_VERSION is already published"
;;
404)
echo "published=false" >> "$GITHUB_OUTPUT"
;;
*)
cat /tmp/crate-version.json
echo "Unexpected crates.io response: $HTTP_STATUS" >&2
exit 1
;;
esac
publish:
name: Publish to crates.io
needs: check-version
if: needs.check-version.outputs.changed == 'true' && needs.check-version.outputs.published == 'false'
runs-on: ubuntu-latest
environment: crates-io
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
- name: Install Rust
uses: dtolnay/rust-toolchain@stable
- name: Verify package
run: cargo publish --dry-run
- name: Authenticate with crates.io
id: auth
uses: rust-lang/crates-io-auth-action@v1
- name: Publish crate
run: cargo publish
env:
CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }}
+67 -13
View File
@@ -3,7 +3,10 @@ name: Publish npm package
on:
push:
branches: [main]
paths: ['napi/package.json']
paths:
- '.github/workflows/publish.yml'
- 'napi/package.json'
- 'napi/legacy/**'
permissions:
contents: read
@@ -14,29 +17,60 @@ jobs:
name: Check version change
runs-on: ubuntu-latest
outputs:
changed: ${{ steps.check.outputs.changed }}
version: ${{ steps.check.outputs.version }}
scoped_changed: ${{ steps.check.outputs.scoped_changed }}
scoped_version: ${{ steps.check.outputs.scoped_version }}
legacy_changed: ${{ steps.check.outputs.legacy_changed }}
legacy_version: ${{ steps.check.outputs.legacy_version }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 2
- name: Check if version changed
- name: Check if versions changed
id: check
run: |
NEW_VERSION=$(node -p "require('./napi/package.json').version")
OLD_VERSION=$(git show HEAD~1:napi/package.json | node -p "JSON.parse(require('fs').readFileSync('/dev/stdin','utf8')).version")
echo "old=$OLD_VERSION new=$NEW_VERSION"
if [ "$NEW_VERSION" != "$OLD_VERSION" ]; then
echo "changed=true" >> "$GITHUB_OUTPUT"
echo "version=$NEW_VERSION" >> "$GITHUB_OUTPUT"
set -euo pipefail
SCOPED_NEW=$(node -p "require('./napi/package.json').version")
if git show HEAD~1:napi/package.json >/tmp/scoped-package-old.json 2>/dev/null; then
SCOPED_OLD=$(node -p "JSON.parse(require('fs').readFileSync('/tmp/scoped-package-old.json','utf8')).version")
else
echo "changed=false" >> "$GITHUB_OUTPUT"
SCOPED_OLD=""
fi
LEGACY_NEW=$(node -p "require('./napi/legacy/package.json').version")
if git show HEAD~1:napi/legacy/package.json >/tmp/legacy-package-old.json 2>/dev/null; then
LEGACY_OLD=$(node -p "JSON.parse(require('fs').readFileSync('/tmp/legacy-package-old.json','utf8')).version")
else
LEGACY_OLD=""
fi
echo "scoped old=$SCOPED_OLD new=$SCOPED_NEW"
echo "legacy old=$LEGACY_OLD new=$LEGACY_NEW"
if [ "$LEGACY_NEW" != "$SCOPED_NEW" ]; then
echo "Legacy package version ($LEGACY_NEW) must match scoped package version ($SCOPED_NEW)." >&2
exit 1
fi
echo "scoped_version=$SCOPED_NEW" >> "$GITHUB_OUTPUT"
echo "legacy_version=$LEGACY_NEW" >> "$GITHUB_OUTPUT"
if [ "$SCOPED_NEW" != "$SCOPED_OLD" ]; then
echo "scoped_changed=true" >> "$GITHUB_OUTPUT"
else
echo "scoped_changed=false" >> "$GITHUB_OUTPUT"
fi
if [ "$LEGACY_NEW" != "$LEGACY_OLD" ]; then
echo "legacy_changed=true" >> "$GITHUB_OUTPUT"
else
echo "legacy_changed=false" >> "$GITHUB_OUTPUT"
fi
build:
needs: check-version
if: needs.check-version.outputs.changed == 'true'
if: needs.check-version.outputs.scoped_changed == 'true'
name: Build ${{ matrix.target }}
runs-on: ${{ matrix.os }}
strategy:
@@ -99,6 +133,7 @@ jobs:
name: Publish to npm
needs: [check-version, build]
runs-on: ubuntu-latest
if: ${{ always() && (needs.check-version.outputs.scoped_changed == 'true' || needs.check-version.outputs.legacy_changed == 'true') && (needs.check-version.outputs.scoped_changed != 'true' || needs.build.result == 'success') }}
permissions:
contents: read
id-token: write
@@ -111,11 +146,13 @@ jobs:
registry-url: 'https://registry.npmjs.org'
- name: Download all artifacts
if: needs.check-version.outputs.scoped_changed == 'true'
uses: actions/download-artifact@v4
with:
path: napi/artifacts
- name: Collect binaries and publish
- name: Collect binaries and publish scoped package
if: needs.check-version.outputs.scoped_changed == 'true'
working-directory: napi
run: |
cp artifacts/bindings-*/*.node .
@@ -126,3 +163,20 @@ jobs:
ls -la *.node index.js index.d.ts
npm publish --provenance --access public
- name: Publish legacy package wrapper
if: needs.check-version.outputs.legacy_changed == 'true'
working-directory: napi/legacy
run: npm publish --provenance
- name: Deprecate legacy package
if: needs.check-version.outputs.legacy_changed == 'true'
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
run: |
if [ -z "${NODE_AUTH_TOKEN:-}" ]; then
echo "NPM_TOKEN is not configured; skipping npm deprecate."
exit 0
fi
npm deprecate firecrawl-pdf-inspector "Deprecated: this package has moved to @firecrawl/pdf-inspector. Please install @firecrawl/pdf-inspector instead."
+2 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "pdf-inspector"
version = "0.1.0"
version = "0.1.2"
edition = "2021"
autobins = false
authors = ["Firecrawl Team"]
@@ -17,7 +17,7 @@ crate-type = ["lib", "cdylib"]
pyo3 = { version = "0.25", features = ["extension-module"], optional = true }
# PDF parsing
lopdf = { git = "https://github.com/J-F-Liu/lopdf", rev = "7a05512d831415b1f2b1ce522391d6beab8a1284", features = ["rayon"] }
lopdf = { version = "0.41.0", features = ["rayon"] }
# Error handling
thiserror = "2.0"
+25 -9
View File
@@ -1,5 +1,8 @@
# pdf-inspector
[![Crates.io](https://img.shields.io/crates/v/pdf-inspector.svg)](https://crates.io/crates/pdf-inspector)
[![npm](https://img.shields.io/npm/v/@firecrawl/pdf-inspector.svg)](https://www.npmjs.com/package/@firecrawl/pdf-inspector)
Fast Rust library for PDF classification and text extraction. Detects whether a PDF is text-based or scanned, extracts text with position awareness, and converts to clean Markdown — all without OCR. Includes bindings for [Python](docs/python.md) and [Node.js](napi/README.md).
Built by [Firecrawl](https://firecrawl.dev) to handle text-based PDFs locally in under 200ms, skipping expensive OCR services for the ~54% of PDFs that don't need them.
@@ -71,9 +74,17 @@ console.log(result.markdown); // Markdown string or null
### Rust
Install from [crates.io](https://crates.io/crates/pdf-inspector):
```bash
cargo add pdf-inspector
```
Or add it manually:
```toml
[dependencies]
pdf-inspector = { git = "https://github.com/firecrawl/pdf-inspector" }
pdf-inspector = "0.1"
```
```rust
@@ -91,29 +102,34 @@ if let Some(markdown) = &result.markdown {
### CLI
```bash
# Install the CLI tools
cargo install pdf-inspector
# Convert PDF to Markdown
cargo run --bin pdf2md -- document.pdf
pdf2md document.pdf
# JSON output (for piping)
cargo run --bin pdf2md -- document.pdf --json
pdf2md document.pdf --json
# Raw markdown only (no headers)
cargo run --bin pdf2md -- document.pdf --raw
pdf2md document.pdf --raw
# Insert page break markers (<!-- Page N -->)
cargo run --bin pdf2md -- document.pdf --pages
pdf2md document.pdf --pages
# Process only specific pages
cargo run --bin pdf2md -- document.pdf --select-pages 1,3,5-10
pdf2md document.pdf --select-pages 1,3,5-10
# Detection only (no extraction)
cargo run --bin detect-pdf -- document.pdf
cargo run --bin detect-pdf -- document.pdf --json
detect-pdf document.pdf
detect-pdf document.pdf --json
# Detection + layout analysis (tables, columns)
cargo run --bin detect-pdf -- document.pdf --analyze --json
detect-pdf document.pdf --analyze --json
```
From a source checkout, use `cargo run --bin pdf2md -- document.pdf` or `cargo run --bin detect-pdf -- document.pdf` instead.
## Architecture
```
+33
View File
@@ -0,0 +1,33 @@
# Security Policy
## Reporting a Vulnerability
If you believe you've found a security vulnerability in pdf-inspector, please
report it privately so we can fix it before public disclosure.
**Preferred:** Email **help@firecrawl.dev** with:
- A description of the issue and its impact
- Steps to reproduce (a minimal PDF or input that triggers the bug is ideal)
- The version or commit hash of pdf-inspector you tested against
**Alternative:** Use GitHub's private vulnerability reporting under the
[Security tab](https://github.com/firecrawl/pdf-inspector/security/advisories/new).
We'll acknowledge your report in a timely manner and keep you updated on
remediation progress. Please do not open a public GitHub issue for security
bugs.
## Scope
In scope:
- Memory-safety issues (panics, OOB reads, UB) reachable from a crafted PDF
- Denial-of-service vectors (unbounded allocation, infinite loops) on
reasonably-sized inputs
- Bugs in the `pdf2md` / `detect-pdf` binaries or the `pdf-inspector` crate
that affect downstream consumers
Out of scope:
- Bugs in upstream dependencies (`lopdf`, etc.) — please report those upstream
- Extraction quality issues (wrong text, missing tables) — open a regular
GitHub issue instead
+44
View File
@@ -0,0 +1,44 @@
# Publishing
The Rust crate is published to [crates.io](https://crates.io/crates/pdf-inspector) with trusted publishing from GitHub Actions. The first release was published manually; future releases publish from `.github/workflows/publish-crate.yml` when a `Cargo.toml` version change lands on `main`.
## crates.io Trusted Publisher
Configure the trusted publisher for the `pdf-inspector` crate with:
- Repository: `firecrawl/pdf-inspector`
- Workflow: `publish-crate.yml`
- Environment: `crates-io`
The workflow uses `rust-lang/crates-io-auth-action@v1` to exchange GitHub's OIDC token for a short-lived crates.io token, then passes it to `cargo publish`.
## Release Steps
1. Update `version` in `Cargo.toml`.
2. Merge the version bump to `main`.
3. The publish workflow compares the new `Cargo.toml` version with `HEAD~1`, runs `cargo publish --dry-run`, then publishes if that version is not already on crates.io.
If `Cargo.toml` changes without a package version bump, the workflow exits without publishing.
## Legacy npm Package
The unscoped `firecrawl-pdf-inspector` package is deprecated in favor of `@firecrawl/pdf-inspector`.
The npm publish workflow publishes the compatibility wrapper in `napi/legacy` whenever `napi/legacy/package.json` has a version bump on `main`. The wrapper keeps older installs working while its README points users to the scoped package.
Keep `napi/legacy/package.json` on the same version as `napi/package.json`, and pin its `@firecrawl/pdf-inspector` dependency to that same version. The publish workflow fails fast if the package versions drift.
Configure npm trusted publishing for both packages against `.github/workflows/publish.yml`:
- `@firecrawl/pdf-inspector`
- `firecrawl-pdf-inspector`
To mark the legacy package as deprecated from CI, configure a GitHub Actions `NPM_TOKEN` secret with permission to manage `firecrawl-pdf-inspector`. Without that secret, the workflow still publishes the wrapper but skips `npm deprecate`.
You can also deprecate the legacy package manually from an npm account that owns it:
```bash
npm deprecate firecrawl-pdf-inspector "Deprecated: this package has moved to @firecrawl/pdf-inspector. Please install @firecrawl/pdf-inspector instead."
```
If the account has two-factor auth enabled, append `--otp=<code>`.
+5 -4
View File
@@ -672,8 +672,9 @@ checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897"
[[package]]
name = "lopdf"
version = "0.40.0"
source = "git+https://github.com/J-F-Liu/lopdf?rev=7a05512d831415b1f2b1ce522391d6beab8a1284#7a05512d831415b1f2b1ce522391d6beab8a1284"
version = "0.41.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67513274c50a2b51e5f75d9e682fcf4ab064a8a9c9ae2c3c59309084882bb24d"
dependencies = [
"aes",
"bitflags",
@@ -829,7 +830,7 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
[[package]]
name = "pdf-inspector"
version = "0.1.0"
version = "0.1.1"
dependencies = [
"env_logger",
"log",
@@ -844,7 +845,7 @@ dependencies = [
[[package]]
name = "pdf-inspector-napi"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"napi",
"napi-build",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "pdf-inspector-napi"
version = "0.2.0"
version = "0.2.1"
edition = "2021"
[lib]
+30
View File
@@ -0,0 +1,30 @@
# firecrawl-pdf-inspector
> Deprecated: this package has moved to [`@firecrawl/pdf-inspector`](https://www.npmjs.com/package/@firecrawl/pdf-inspector).
> Please install and import the scoped Firecrawl package instead.
```bash
npm install @firecrawl/pdf-inspector
# or
bun add @firecrawl/pdf-inspector
```
```typescript
import { processPdf, classifyPdf } from '@firecrawl/pdf-inspector'
import { readFileSync } from 'fs'
const pdf = readFileSync('document.pdf')
const result = processPdf(pdf)
console.log(result.pdfType)
console.log(result.markdown)
```
This package remains only as a compatibility wrapper for older installs:
```typescript
import { processPdf } from 'firecrawl-pdf-inspector'
```
New projects should use `@firecrawl/pdf-inspector` directly.
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env node
import '@firecrawl/pdf-inspector/bin/pdf-inspector.mjs'
+2
View File
@@ -0,0 +1,2 @@
export * from '@firecrawl/pdf-inspector'
+20
View File
@@ -0,0 +1,20 @@
'use strict'
const scoped = require('@firecrawl/pdf-inspector')
module.exports = scoped
module.exports.classifyPdf = scoped.classifyPdf
module.exports.detectPdf = scoped.detectPdf
module.exports.detectVectorGridInRegion = scoped.detectVectorGridInRegion
module.exports.extractPagesMarkdown = scoped.extractPagesMarkdown
module.exports.extractTablesInRegions = scoped.extractTablesInRegions
module.exports.extractTablesWithStructure = scoped.extractTablesWithStructure
module.exports.extractTablesWithStructureAuto = scoped.extractTablesWithStructureAuto
module.exports.extractTablesWithStructureCells = scoped.extractTablesWithStructureCells
module.exports.extractText = scoped.extractText
module.exports.extractTextInRegions = scoped.extractTextInRegions
module.exports.extractTextWithPositions = scoped.extractTextWithPositions
module.exports.ItemType = scoped.ItemType
module.exports.PdfType = scoped.PdfType
module.exports.processPdf = scoped.processPdf
+35
View File
@@ -0,0 +1,35 @@
{
"name": "firecrawl-pdf-inspector",
"version": "1.9.8",
"description": "Deprecated compatibility wrapper for @firecrawl/pdf-inspector.",
"main": "index.js",
"types": "index.d.ts",
"bin": {
"pdf-inspector": "bin/pdf-inspector.mjs"
},
"license": "MIT",
"keywords": [
"pdf",
"pdf-extraction",
"pdf-parser",
"text-extraction",
"ocr",
"pdf-classification",
"firecrawl",
"deprecated"
],
"files": [
"index.js",
"index.d.ts",
"bin/",
"README.md"
],
"repository": {
"type": "git",
"url": "https://github.com/firecrawl/pdf-inspector"
},
"homepage": "https://github.com/firecrawl/pdf-inspector",
"dependencies": {
"@firecrawl/pdf-inspector": "1.9.8"
}
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@firecrawl/pdf-inspector",
"version": "1.8.1",
"version": "1.9.8",
"description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
"main": "index.js",
"types": "index.d.ts",
+29
View File
@@ -0,0 +1,29 @@
import { readFileSync } from "node:fs";
import { createRequire } from "node:module";
const require = createRequire(import.meta.url);
const { detectVectorGridInRegion } = require("./index.js");
const pdfPath =
process.argv[2] ?? "/tmp/pdf_inspector_indent_fixtures/cis_edge_benchmark.pdf";
const pdf = readFileSync(pdfPath);
const dpi = Number(process.argv[3] ?? 200);
const crops = [
{ pageIdx: 29, box: [0, 0, 612, 792], label: "page30-full" },
{ pageIdx: 16, box: [0, 0, 612, 792], label: "page17-full" },
{ pageIdx: 23, box: [0, 0, 612, 792], label: "page24-full" },
];
for (const { pageIdx, box, label } of crops) {
const result = detectVectorGridInRegion(pdf, pageIdx, box, dpi);
if (!result) {
console.log(`${label}: null`);
continue;
}
const rows = result.structureTokens.filter((token) => token === "<tr>").length;
const cols = rows > 0 ? result.cellBboxes.length / rows : 0;
console.log(
`${label}: cells=${result.cellBboxes.length} rows=${rows} cols=${cols}`,
);
}
+149 -10
View File
@@ -18,7 +18,7 @@ use super::fonts::{
get_font_file2_obj_num, get_operand_bytes, CMapDecisionCache,
};
use super::xobjects::{extract_form_xobject_text, get_page_xobjects, XObjectType};
use super::{get_number, multiply_matrices};
use super::{get_number, image_bbox_from_ctm, multiply_matrices};
/// Strip PDF comments (% to end of line) from content stream bytes.
///
@@ -188,7 +188,17 @@ pub(crate) fn extract_page_text_items(
// Graphics state tracking
let mut ctm = [1.0f32, 0.0, 0.0, 1.0, 0.0, 0.0]; // Current Transformation Matrix
let mut text_rendering_mode: i32 = 0; // 0=fill, 1=stroke, 2=fill+stroke, 3=invisible
let mut gstate_stack: Vec<([f32; 6], i32, f32, f32)> = Vec::new();
#[derive(Clone)]
struct SavedGraphicsState {
ctm: [f32; 6],
text_rendering_mode: i32,
char_spacing: f32,
word_spacing: f32,
text_leading: f32,
current_font: String,
current_font_size: f32,
}
let mut gstate_stack: Vec<SavedGraphicsState> = Vec::new();
// Text state tracking
let mut current_font = String::new();
@@ -227,15 +237,26 @@ pub(crate) fn extract_page_text_items(
match op.operator.as_str() {
"q" => {
// Save graphics state
gstate_stack.push((ctm, text_rendering_mode, char_spacing, word_spacing));
gstate_stack.push(SavedGraphicsState {
ctm,
text_rendering_mode,
char_spacing,
word_spacing,
text_leading,
current_font: current_font.clone(),
current_font_size,
});
}
"Q" => {
// Restore graphics state
if let Some((saved_ctm, saved_tr, saved_tc, saved_tw)) = gstate_stack.pop() {
ctm = saved_ctm;
text_rendering_mode = saved_tr;
char_spacing = saved_tc;
word_spacing = saved_tw;
if let Some(saved) = gstate_stack.pop() {
ctm = saved.ctm;
text_rendering_mode = saved.text_rendering_mode;
char_spacing = saved.char_spacing;
word_spacing = saved.word_spacing;
text_leading = saved.text_leading;
current_font = saved.current_font;
current_font_size = saved.current_font_size;
}
}
"cm" => {
@@ -385,6 +406,7 @@ pub(crate) fn extract_page_text_items(
&font_encodings,
&encoding_cache,
&mut cmap_decisions,
&font_widths,
) {
let combined = multiply_matrices(&text_matrix, &ctm);
let rendered_size = effective_font_size(current_font_size, &combined);
@@ -533,6 +555,7 @@ pub(crate) fn extract_page_text_items(
&font_encodings,
&encoding_cache,
&mut cmap_decisions,
&font_widths,
) {
current_text.push_str(&text);
}
@@ -620,6 +643,7 @@ pub(crate) fn extract_page_text_items(
&font_encodings,
&encoding_cache,
&mut cmap_decisions,
&font_widths,
) {
if !text.trim().is_empty() {
let combined = multiply_matrices(&text_matrix, &ctm);
@@ -661,7 +685,29 @@ pub(crate) fn extract_page_text_items(
if let Some(xobj_type) = xobjects.get(&xobj_name) {
match xobj_type {
XObjectType::Image => {
// Skip images — text extraction only
// Emit a positional placeholder for the image
// so downstream consumers (layout-aware
// pipelines, figure-OCR routers) can locate
// raster figures without parsing the PDF
// again. The text field carries the
// XObject resource name in the legacy
// `[Image: Im0]` format that the markdown
// emitter already recognizes.
let (x, y, width, height) = image_bbox_from_ctm(&ctm);
items.push(TextItem {
text: format!("[Image: {}]", xobj_name),
x,
y,
width,
height,
font: String::new(),
font_size: 0.0,
page: page_num,
is_bold: false,
is_italic: false,
item_type: ItemType::Image,
mcid: current_mcid(&marked_content_stack),
});
}
XObjectType::Form(form_id) => {
// Extract text from Form XObject
@@ -1012,9 +1058,17 @@ pub(crate) fn extract_page_text_items(
// producing thousands of identical rects that yield a degenerate grid.
// After dedup, if too few unique clip rects remain we fall through to
// fill rects (explicitly drawn visible rectangles).
//
// When fill rects substantially outnumber clip rects, the clips are
// typically section-level wrappers and the fills are the actual table
// cell backgrounds (e.g. shaded-header tables drawn with `m`/`l`/`h`/`f*`
// sequences). In that case, prefer fills.
if rects.is_empty() {
dedup_rects(&mut clip_rects);
if clip_rects.len() >= 4 {
let prefer_fills = !fill_rects.is_empty() && fill_rects.len() >= clip_rects.len() * 3;
if prefer_fills {
rects = fill_rects;
} else if clip_rects.len() >= 4 {
rects = clip_rects;
} else if !fill_rects.is_empty() {
rects = fill_rects;
@@ -1253,6 +1307,91 @@ mod tests {
assert!(lines.is_empty());
}
#[test]
fn test_q_restores_current_font_for_text_decoding() {
use crate::tounicode::FontCMaps;
use lopdf::{dictionary, Object, Stream};
fn cmap_stream(dst_hex: &str) -> Stream {
let cmap = format!(
r#"/CIDInit /ProcSet findresource begin
12 dict begin
begincmap
/CIDSystemInfo << /Registry (Adobe) /Ordering (UCS) /Supplement 0 >> def
/CMapName /Test-UCS def
/CMapType 2 def
1 begincodespacerange
<00> <FF>
endcodespacerange
1 beginbfchar
<41> <{dst_hex}>
endbfchar
endcmap
CMapName currentdict /CMap defineresource pop
end
end"#
);
Stream::new(dictionary! {}, cmap.into_bytes())
}
let mut doc = lopdf::Document::new();
let f1_cmap = doc.add_object(Object::Stream(cmap_stream("0058"))); // X
let f2_cmap = doc.add_object(Object::Stream(cmap_stream("0059"))); // Y
let f1 = doc.add_object(dictionary! {
"Type" => "Font",
"Subtype" => "Type1",
"BaseFont" => "Helvetica",
"ToUnicode" => Object::Reference(f1_cmap),
});
let f2 = doc.add_object(dictionary! {
"Type" => "Font",
"Subtype" => "Type1",
"BaseFont" => "Helvetica",
"ToUnicode" => Object::Reference(f2_cmap),
});
let content = b"BT /F1 12 Tf 10 700 Tm <41> Tj ET
q
BT /F2 12 Tf 20 700 Tm <41> Tj ET
Q
BT 30 700 Tm <41> Tj ET";
let content_id = doc.add_object(Object::Stream(Stream::new(
dictionary! {},
content.to_vec(),
)));
let page_id = doc.add_object(dictionary! {
"Type" => "Page",
"Contents" => Object::Reference(content_id),
"Resources" => dictionary! {
"Font" => dictionary! {
"F1" => Object::Reference(f1),
"F2" => Object::Reference(f2),
},
},
"MediaBox" => vec![0.into(), 0.into(), 612.into(), 792.into()],
});
let pages_id = doc.add_object(dictionary! {
"Type" => "Pages",
"Count" => Object::Integer(1),
"Kids" => vec![Object::Reference(page_id)],
});
let catalog_id = doc.add_object(dictionary! {
"Type" => "Catalog",
"Pages" => Object::Reference(pages_id),
});
doc.trailer.set("Root", Object::Reference(catalog_id));
let font_cmaps = FontCMaps::from_doc(&doc);
let ((items, _, _), _, _) =
extract_page_text_items(&doc, page_id, 1, &font_cmaps, false).unwrap();
let text = items
.iter()
.map(|item| item.text.as_str())
.collect::<String>();
assert_eq!(text, "XYX");
}
#[test]
fn test_strip_pdf_comments() {
// Basic comment stripping
+108 -1
View File
@@ -720,7 +720,11 @@ pub(crate) fn extract_text_from_operand(
font_encodings: &PageFontEncodings,
encoding_cache: &HashMap<String, Encoding<'_>>,
cmap_decisions: &mut CMapDecisionCache,
font_widths: &PageFontWidths,
) -> Option<String> {
let is_type0_cid_font = font_widths
.get(current_font)
.is_some_and(|info| info.is_cid);
let result = (|| -> Option<String> {
if let Object::String(bytes, _) = obj {
let mut decode_with_entry = |entry: &crate::tounicode::CMapEntry| -> Option<String> {
@@ -854,6 +858,13 @@ pub(crate) fn extract_text_from_operand(
// unmapped. Don't fall through to text-interpretation fallbacks
// (Latin-1, UTF-16, etc.) which would misinterpret CID bytes as
// character codes (e.g. CID 0x01A9 → Latin-1 "©").
if is_type0_cid_font && bytes.iter().any(|&b| b > 0x7F) {
// 2-byte CIDs (Identity-H) are by far the common case; for
// an odd byte count we still emit at least one marker so
// detection downstream fires.
let cid_count = (bytes.len() / 2).max(1);
return Some("\u{FFFD}".repeat(cid_count));
}
// Try our custom encoding map from Differences arrays.
// The Differences array overrides specific codes in a base encoding (typically
@@ -962,7 +973,10 @@ pub(crate) fn extract_text_from_operand(
return Some(symbol_text);
}
// Latin-1 fallback
// Pure ASCII bytes round-trip safely (Latin-1 == ASCII for
// 0x00..=0x7F), and non-CID (Type1 / TrueType / Type3) fonts
// use single-byte encodings where Latin-1 fallback is the
// canonical interpretation.
Some(bytes.iter().map(|&b| b as char).collect())
} else {
None
@@ -1213,4 +1227,97 @@ mod tests {
let bad = "###!!!@@@$$$";
assert!(score_text(good) > score_text(bad));
}
#[test]
fn cid_font_with_unparseable_cmap_does_not_emit_latin1_mojibake() {
// Type0/CID font (font_widths reports `is_cid=true`) where the
// ToUnicode CMap couldn't be parsed (FontCMaps doesn't have the
// obj_num). Bytes are a 2-byte CID stream containing high bytes
// that aren't valid UTF-8 — exactly the case in the production
// samples (Identity-H text where the ToUnicode CMap was missing
// or malformed, scrape_id 019de78c-..., e.g. "Í Ù Z)¿").
//
// Without the guard, the function falls through to the byte-by-byte
// Latin-1 fallback and produces "ÍÙ" (U+00CD U+00D9). The correct
// behavior is to emit U+FFFD per CID so downstream
// `detect_encoding_issues` flags the page for OCR.
let bytes = vec![0xCD_u8, 0xD9, 0xCD, 0xD9];
let obj = Object::String(bytes, lopdf::StringFormat::Hexadecimal);
let font_cmaps = FontCMaps::default();
let mut font_tounicode_refs: HashMap<String, u32> = HashMap::new();
font_tounicode_refs.insert("F0".to_string(), 999);
let inline_cmaps = HashMap::new();
let font_encodings: PageFontEncodings = HashMap::new();
let encoding_cache: HashMap<String, Encoding<'_>> = HashMap::new();
let mut decisions = CMapDecisionCache::new();
let mut font_widths: PageFontWidths = HashMap::new();
font_widths.insert("F0".to_string(), make_font_info(&[], 1000, true));
let result = extract_text_from_operand(
&obj,
"F0",
None,
&font_cmaps,
&font_tounicode_refs,
&inline_cmaps,
&font_encodings,
&encoding_cache,
&mut decisions,
&font_widths,
);
let text = result.expect("CID font fallback should still emit a marker");
assert!(
!text.contains('\u{00CD}') && !text.contains('\u{00D9}'),
"CID font with unparseable CMap leaked Latin-1 mojibake: {text:?}"
);
assert!(
text.contains('\u{FFFD}'),
"CID font with unparseable CMap should emit U+FFFD so detect_encoding_issues fires: {text:?}"
);
}
#[test]
fn simple_font_latin1_fallback_passes_high_bytes_through() {
// A Type1/TrueType simple font (is_cid=false) with a `/ToUnicode`
// reference but no usable CMap and no `/Differences` map.
// Per-byte Latin-1 IS the canonical interpretation here — these
// bytes are character codes, not CIDs. The CID guard must NOT
// strip them. Reproduces the false positive that an earlier
// version of the guard introduced for fonts in PDFs like
// pdf-evals/Navigating-Artificial-Intelligence-..., where bytes
// like 0xB6 are legitimate Latin-1 character codes.
let bytes = vec![0x24_u8, 0x47, 0xB6, 0x56]; // "$G¶V"
let obj = Object::String(bytes, lopdf::StringFormat::Hexadecimal);
let font_cmaps = FontCMaps::default();
let mut font_tounicode_refs: HashMap<String, u32> = HashMap::new();
font_tounicode_refs.insert("F1".to_string(), 999);
let inline_cmaps = HashMap::new();
let font_encodings: PageFontEncodings = HashMap::new();
let encoding_cache: HashMap<String, Encoding<'_>> = HashMap::new();
let mut decisions = CMapDecisionCache::new();
let mut font_widths: PageFontWidths = HashMap::new();
font_widths.insert("F1".to_string(), make_font_info(&[], 1000, false));
let text = extract_text_from_operand(
&obj,
"F1",
None,
&font_cmaps,
&font_tounicode_refs,
&inline_cmaps,
&font_encodings,
&encoding_cache,
&mut decisions,
&font_widths,
)
.expect("simple font should round-trip Latin-1 bytes");
assert_eq!(text, "$G\u{00B6}V");
assert!(
!text.contains('\u{FFFD}'),
"simple font fallback must not stamp FFFD over legitimate bytes: {text:?}"
);
}
}
+6 -2
View File
@@ -29,8 +29,12 @@ pub(crate) fn detect_columns(
const MIN_ITEMS_PER_COLUMN: usize = 10;
const NOISE_FRACTION: f32 = 0.15;
// Get items for this page
let page_items: Vec<&TextItem> = items.iter().filter(|i| i.page == page).collect();
// Get items for this page. Strip Image placeholders — an image's left edge
// would otherwise count toward the column projection profile.
let page_items: Vec<&TextItem> = items
.iter()
.filter(|i| i.page == page && crate::extractor::is_text_layout_item(i))
.collect();
if page_items.is_empty() {
return vec![];
+63
View File
@@ -227,6 +227,69 @@ fn extract_positioned_text_impl(
// Shared helpers (used by submodules via `super::`)
// ---------------------------------------------------------------------------
/// Return true when this item should participate in text-layout
/// heuristics (column detection, table grid detection, line grouping).
///
/// Image XObjects emit a positional placeholder via
/// `extract_text_with_positions` (so layout-aware callers can crop +
/// caption figures), but their bboxes don't carry text glyphs and would
/// skew column/row clustering if they reached the heuristics. Hyperlinks
/// and form fields *do* participate — the existing logic treats them as
/// text-like and we keep that.
pub(crate) fn is_text_layout_item(item: &crate::types::TextItem) -> bool {
!matches!(item.item_type, crate::types::ItemType::Image)
}
/// Map a (u, v) point in unit-square coordinates through the 6-element CTM
/// to page-space. CTM format is `[a, b, c, d, e, f]` per
/// [`multiply_matrices`].
fn apply_ctm_point(ctm: &[f32; 6], u: f32, v: f32) -> (f32, f32) {
(
u * ctm[0] + v * ctm[2] + ctm[4],
u * ctm[1] + v * ctm[3] + ctm[5],
)
}
/// Compute the page-space axis-aligned bounding box of an Image XObject
/// invoked under the given CTM.
///
/// Per the PDF spec, an image XObject is always rendered into a unit
/// square `(0,0)(1,1)` in its local coordinate system, and the `Do`
/// operator applies the current CTM to position/scale/rotate that square
/// onto the page. For the common axis-aligned case (no rotation/shear),
/// the CTM reduces to `[w, 0, 0, h, x, y]` and the bbox is just
/// `(x, y, w, h)`. For rotated/sheared images we transform all four
/// corners and return their axis-aligned bbox so the caller always gets
/// an upright rectangle.
///
/// Coordinates are PDF user space (origin at bottom-left, y-up). Width
/// and height are non-negative.
pub(crate) fn image_bbox_from_ctm(ctm: &[f32; 6]) -> (f32, f32, f32, f32) {
let corners = [
apply_ctm_point(ctm, 0.0, 0.0),
apply_ctm_point(ctm, 1.0, 0.0),
apply_ctm_point(ctm, 1.0, 1.0),
apply_ctm_point(ctm, 0.0, 1.0),
];
let (mut x_min, mut x_max) = (corners[0].0, corners[0].0);
let (mut y_min, mut y_max) = (corners[0].1, corners[0].1);
for (cx, cy) in corners.iter().skip(1) {
if *cx < x_min {
x_min = *cx;
}
if *cx > x_max {
x_max = *cx;
}
if *cy < y_min {
y_min = *cy;
}
if *cy > y_max {
y_max = *cy;
}
}
(x_min, y_min, x_max - x_min, y_max - y_min)
}
/// Multiply two 2D transformation matrices
/// Matrix format: [a, b, c, d, e, f] representing:
/// | a b 0 |
+39 -13
View File
@@ -10,7 +10,7 @@ use super::fonts::{
build_font_encodings, build_font_widths, compute_string_width_ts, extract_text_from_operand,
get_font_file2_obj_num, get_operand_bytes, CMapDecisionCache,
};
use super::{get_number, multiply_matrices};
use super::{get_number, image_bbox_from_ctm, multiply_matrices};
const MAX_FORM_XOBJECT_DEPTH: u8 = 5;
@@ -262,19 +262,43 @@ fn extract_form_xobject_text_inner(
if !op.operands.is_empty() {
if let Ok(name) = op.operands[0].as_name() {
let xobj_name = String::from_utf8_lossy(name).to_string();
if let Some(XObjectType::Form(nested_id)) = form_xobjects.get(&xobj_name) {
if depth < MAX_FORM_XOBJECT_DEPTH {
let nested_items = extract_form_xobject_text_inner(
doc,
*nested_id,
page_num,
font_cmaps,
&ctm,
cmap_decisions,
depth + 1,
);
items.extend(nested_items);
match form_xobjects.get(&xobj_name) {
Some(XObjectType::Form(nested_id)) => {
if depth < MAX_FORM_XOBJECT_DEPTH {
let nested_items = extract_form_xobject_text_inner(
doc,
*nested_id,
page_num,
font_cmaps,
&ctm,
cmap_decisions,
depth + 1,
);
items.extend(nested_items);
}
}
Some(XObjectType::Image) => {
// Mirror the top-level Image-XObject emission
// in content_stream.rs so figures embedded
// inside Form XObjects (common in print-to-PDF
// workflows) aren't silently dropped.
let (x, y, width, height) = image_bbox_from_ctm(&ctm);
items.push(TextItem {
text: format!("[Image: {}]", xobj_name),
x,
y,
width,
height,
font: String::new(),
font_size: 0.0,
page: page_num,
is_bold: false,
is_italic: false,
item_type: ItemType::Image,
mcid: None,
});
}
None => {}
}
}
}
@@ -373,6 +397,7 @@ fn extract_form_xobject_text_inner(
&font_encodings,
&encoding_cache,
cmap_decisions,
&font_widths,
) {
let combined = multiply_matrices(&text_matrix, &ctm);
let rendered_size = effective_font_size(current_font_size, &combined);
@@ -517,6 +542,7 @@ fn extract_form_xobject_text_inner(
&font_encodings,
&encoding_cache,
cmap_decisions,
&font_widths,
) {
current_text.push_str(&text);
}
+2099 -126
View File
File diff suppressed because it is too large Load Diff
+232 -2
View File
@@ -149,6 +149,79 @@ fn find_isolated_lines(lines: &[TextLine], base_size: f32, para_threshold: f32)
set
}
/// Pre-scan body-size all-bold runs that are too long to be headings.
///
/// Some academic PDFs use an all-bold abstract/summary paragraph immediately
/// after the author block. A line-local bold heading heuristic sees each
/// wrapped visual line as "standalone" once the first line is misclassified,
/// producing a stack of `##` headings. Multi-line body-size bold runs with a
/// paragraph-sized word count should stay paragraph text.
fn find_wrapped_bold_paragraph_lines(
lines: &[TextLine],
base_size: f32,
para_threshold: f32,
) -> HashSet<usize> {
let mut set = HashSet::new();
let mut i = 0usize;
while i < lines.len() {
if !is_body_size_all_bold_line(&lines[i], base_size) {
i += 1;
continue;
}
let start = i;
let mut end = i;
let mut word_count = lines[i].text().split_whitespace().count();
while end + 1 < lines.len()
&& is_body_size_all_bold_line(&lines[end + 1], base_size)
&& is_wrapped_same_style_line(&lines[end], &lines[end + 1], para_threshold)
{
end += 1;
word_count += lines[end].text().split_whitespace().count();
}
let line_count = end - start + 1;
if line_count >= 3 && word_count > 20 {
for idx in start..=end {
set.insert(idx);
}
}
i = end + 1;
}
set
}
fn is_body_size_all_bold_line(line: &TextLine, base_size: f32) -> bool {
let Some(first) = line.items.first() else {
return false;
};
first.font_size >= base_size * 0.95
&& first.font_size < base_size * 1.2
&& line
.items
.iter()
.all(|item| item.is_bold && (item.font_size - first.font_size).abs() < 0.5)
}
fn is_wrapped_same_style_line(prev: &TextLine, next: &TextLine, para_threshold: f32) -> bool {
if prev.page != next.page {
return false;
}
let y_gap = prev.y - next.y;
if !(y_gap > 0.0 && y_gap <= para_threshold) {
return false;
}
let prev_x = prev.items.first().map(|item| item.x).unwrap_or(0.0);
let next_x = next.items.first().map(|item| item.x).unwrap_or(0.0);
(prev_x - next_x).abs() <= 40.0
}
/// Resolve the dominant structure role for a text line by looking up its items' MCIDs.
///
/// Returns the first non-container role found (skipping Document/Part/Sect/Div/NonStruct/Span).
@@ -397,6 +470,8 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
// between paragraphs at body font size. Inspired by opendataloader's
// lookahead in HeadingProcessor (prevNode/nextNode context).
let isolated_lines = find_isolated_lines(&lines, base_size, para_threshold);
let wrapped_bold_paragraph_lines =
find_wrapped_bold_paragraph_lines(&lines, base_size, para_threshold);
// Detect struct heading levels that are overused (body text mistagged as headings)
let overused_heading_levels = detect_overused_struct_heading_levels(&lines, struct_roles);
@@ -410,6 +485,7 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
let mut last_list_x: Option<f32> = None;
let mut in_code_block = false;
let mut prev_had_dot_leaders = false;
let mut paragraph_in_wrapped_bold_run = false;
let mut inserted_tables: HashSet<(u32, usize)> = HashSet::new();
let mut inserted_images: HashSet<(u32, usize)> = HashSet::new();
@@ -475,6 +551,7 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
current_page = line.page;
prev_y = f32::MAX;
prev_x = 0.0;
paragraph_in_wrapped_bold_run = false;
if options.include_page_numbers {
output.push_str(&format!("<!-- Page {} -->\n\n", current_page));
@@ -489,6 +566,7 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
output.push('\n');
output.push_str(table_md);
@@ -506,6 +584,7 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
output.push('\n');
output.push_str(image_md);
@@ -527,9 +606,18 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
&& y_gap.abs() <= para_threshold
&& (prev_x - line_x).abs() > 50.0
&& prev_y < f32::MAX;
if (is_para_break || is_band_switch) && in_paragraph {
let line_all_bold = !line.items.is_empty() && line.items.iter().all(|item| item.is_bold);
let line_in_wrapped_bold_run = wrapped_bold_paragraph_lines.contains(&line_idx);
let is_bold_to_regular_break = in_paragraph
&& paragraph_in_wrapped_bold_run
&& !line_in_wrapped_bold_run
&& !line_all_bold
&& y_gap > base_size * 1.2
&& y_gap <= para_threshold;
if (is_para_break || is_band_switch || is_bold_to_regular_break) && in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
// Don't immediately end list on paragraph break
// Let the continuation check below decide if we're still in a list
@@ -572,6 +660,7 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
output.push_str(trimmed);
output.push_str("\n\n");
@@ -625,6 +714,9 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
if !(1..=15).contains(&word_count) {
return None;
}
if wrapped_bold_paragraph_lines.contains(&line_idx) {
return None;
}
let rarity = font_size_rarity(line_font_size, &font_stats);
let all_bold = !line.items.is_empty() && line.items.iter().all(|i| i.is_bold);
let standalone = !in_paragraph;
@@ -656,6 +748,7 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
let prefix = "#".repeat(level);
// Use plain text for headers to avoid redundant formatting
@@ -678,6 +771,7 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
output.push_str(&format!("- {}", trimmed));
output.push('\n');
@@ -691,6 +785,7 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
let formatted = format_list_item(trimmed);
output.push_str(&formatted);
@@ -737,6 +832,7 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
output.push_str(&format!("> {}\n", trimmed));
continue;
@@ -747,6 +843,7 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
if !in_code_block {
output.push_str("```\n");
@@ -767,6 +864,11 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
}
}
output.push_str(trimmed);
paragraph_in_wrapped_bold_run = if in_paragraph {
paragraph_in_wrapped_bold_run || line_in_wrapped_bold_run
} else {
line_in_wrapped_bold_run
};
in_paragraph = true;
prev_had_dot_leaders = cur_dot_leaders;
}
@@ -836,6 +938,8 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
let para_threshold = compute_paragraph_threshold(&lines, base_size);
let isolated_lines = find_isolated_lines(&lines, base_size, para_threshold);
let wrapped_bold_paragraph_lines =
find_wrapped_bold_paragraph_lines(&lines, base_size, para_threshold);
let mut output = String::new();
let mut current_page = 0u32;
@@ -844,6 +948,7 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
let mut in_paragraph = false;
let mut last_list_x: Option<f32> = None;
let mut prev_had_dot_leaders = false;
let mut paragraph_in_wrapped_bold_run = false;
for (line_idx, line) in lines.iter().enumerate() {
// Page break
@@ -860,6 +965,7 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
in_list = false;
last_list_x = None;
prev_had_dot_leaders = false;
paragraph_in_wrapped_bold_run = false;
if options.include_page_numbers {
output.push_str(&format!("<!-- Page {} -->\n\n", current_page));
@@ -870,9 +976,18 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
// (newspaper columns emitted sequentially on the same page).
let y_gap = prev_y - line.y;
let is_para_break = y_gap.abs() > para_threshold;
if is_para_break && in_paragraph {
let line_all_bold = !line.items.is_empty() && line.items.iter().all(|item| item.is_bold);
let line_in_wrapped_bold_run = wrapped_bold_paragraph_lines.contains(&line_idx);
let is_bold_to_regular_break = in_paragraph
&& paragraph_in_wrapped_bold_run
&& !line_in_wrapped_bold_run
&& !line_all_bold
&& y_gap > base_size * 1.2
&& y_gap <= para_threshold;
if (is_para_break || is_bold_to_regular_break) && in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
// Don't immediately end list on paragraph break
// Let the continuation check below decide if we're still in a list
@@ -896,6 +1011,7 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
output.push_str(trimmed);
output.push_str("\n\n");
@@ -918,6 +1034,9 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
if !(1..=15).contains(&word_count) {
return None;
}
if wrapped_bold_paragraph_lines.contains(&line_idx) {
return None;
}
let rarity = font_size_rarity(line_font_size, &font_stats);
let all_bold = !line.items.is_empty() && line.items.iter().all(|i| i.is_bold);
let standalone = !in_paragraph;
@@ -935,6 +1054,7 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
let prefix = "#".repeat(header_level);
// Use plain text for headers to avoid redundant formatting
@@ -949,6 +1069,7 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
let formatted = format_list_item(trimmed);
output.push_str(&formatted);
@@ -993,6 +1114,7 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
paragraph_in_wrapped_bold_run = false;
}
// Use plain text for code blocks
output.push_str(&format!("```\n{}\n```\n", plain_trimmed));
@@ -1010,6 +1132,11 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
}
}
output.push_str(trimmed);
paragraph_in_wrapped_bold_run = if in_paragraph {
paragraph_in_wrapped_bold_run || line_in_wrapped_bold_run
} else {
line_in_wrapped_bold_run
};
in_paragraph = true;
prev_had_dot_leaders = cur_dot_leaders;
}
@@ -1356,6 +1483,109 @@ mod tests {
);
}
#[test]
fn test_wrapped_bold_abstract_is_not_split_into_headings() {
// Regression for arXiv 1107.1353: the opening abstract paragraph is
// entirely bold at body size. The first wrapped lines used to become
// separate H2 headings, and the following body paragraph was joined to
// the bold abstract because the paragraph gap is modest.
let make = |text: &str, y: f32, font_size: f32, bold: bool| {
let mut item = make_item(text, 1, None);
item.y = y;
item.font_size = font_size;
item.height = font_size;
item.is_bold = bold;
item
};
let lines = vec![
make_line(vec![make(
"Quantum Nature of Light Measured With a Single Detector",
747.7,
25.0,
true,
)]),
make_line(vec![make(
"Gesine A. Steudle1*, Stefan Schietinger1, David Höckel1",
651.1,
11.0,
false,
)]),
make_line(vec![make(
"Zwiller2, and Oliver Benson1",
638.5,
11.0,
false,
)]),
make_line(vec![make(
"The introduction of light quanta by Einstein in 1905 triggered strong efforts to",
607.5,
11.0,
true,
)]),
make_line(vec![make(
"demonstrate the quantum properties of light directly, without involving matter",
594.8,
11.0,
true,
)]),
make_line(vec![make(
"quantization. It however took more than seven decades for the quantum granularity",
582.2,
11.0,
true,
)]),
make_line(vec![make(
"of light to be observed in the fluorescence of single atoms. Single atoms emit",
569.5,
11.0,
true,
)]),
make_line(vec![make(
"photons one at a time, this is typically demonstrated with a Hanbury-Brown-Twiss",
556.9,
11.0,
true,
)]),
make_line(vec![make(
"Our work significantly simplifies a widely used photon-correlation technique.",
544.2,
11.0,
true,
)]),
make_line(vec![make(
"A photon is a single excitation of a mode of the electromagnetic field.",
528.7,
11.0,
false,
)]),
];
let md = to_markdown_from_lines_with_tables_and_images(
lines,
MarkdownOptions::default(),
HashMap::new(),
HashMap::new(),
&std::collections::HashSet::new(),
None,
);
assert!(
md.contains("# Quantum Nature of Light Measured With a Single Detector"),
"title should remain a heading: {md}"
);
assert!(
!md.contains("## The introduction")
&& !md.contains("## demonstrate")
&& !md.contains("## quantization"),
"bold abstract lines should not become headings: {md}"
);
assert!(
md.contains("technique.**\n\nA photon is a single excitation"),
"body paragraph should be separated from bold abstract: {md}"
);
}
#[test]
fn test_struct_role_code_multiline_accumulation() {
let mut line1 = make_item("fn main() {", 1, Some(0));
+10 -1
View File
@@ -422,7 +422,16 @@ impl Default for MarkdownOptions {
fix_hyphenation: true,
detect_bold: true,
detect_italic: true,
include_images: true,
// `include_images: false` is intentional. The content-stream walker
// now emits `ItemType::Image` `TextItem`s for every Image XObject
// it encounters (see `extractor/content_stream.rs`). If we rendered
// those into markdown by default, every existing caller would
// suddenly see `![Image: Im0](image)` placeholders inserted
// throughout their output — a silent regression for anyone who
// upgrades. Image bboxes are still available via
// `extract_text_with_positions` for callers (e.g. layout-aware
// pipelines) that want to crop + caption figures themselves.
include_images: false,
include_links: true,
include_page_numbers: false,
strip_headers_footers: true,
+27
View File
@@ -29,6 +29,7 @@ pub(crate) fn clean_markdown(mut text: String, options: &MarkdownOptions) -> Str
// text item, which combine with gap-based space insertion to produce
// double spaces ("Vice President" instead of "Vice President").
collapse_consecutive_spaces(&mut text);
remove_spaces_before_closing_brackets(&mut text);
// Remove excessive newlines (more than 2 in a row)
while text.contains("\n\n\n") {
@@ -71,6 +72,20 @@ fn collapse_consecutive_spaces(text: &mut String) {
*text = result;
}
/// Remove spaces before closing square brackets.
/// Unit markers and markdown links occasionally pick up a gap-inserted space
/// before `]` (e.g. `[kg/m3 ]`), which is cosmetic padding.
fn remove_spaces_before_closing_brackets(text: &mut String) {
let mut result = String::with_capacity(text.len());
for ch in text.chars() {
if ch == ']' && result.ends_with(' ') {
result.pop();
}
result.push(ch);
}
*text = result;
}
/// Collapse dot leaders (runs of 4+ dots) into " ... "
/// Common in tables of contents: "Introduction...............................1" -> "Introduction ... 1"
fn collapse_dot_leaders(text: &str) -> String {
@@ -342,6 +357,18 @@ mod tests {
assert!(result.contains("Chapter 2 ... 20"));
}
// --- remove_spaces_before_closing_brackets ---
#[test]
fn test_remove_spaces_before_closing_brackets() {
let mut input = "Density [kg/m3 ] and [linked text ](https://example.com)".to_string();
remove_spaces_before_closing_brackets(&mut input);
assert_eq!(
input,
"Density [kg/m3] and [linked text](https://example.com)"
);
}
// --- fix_hyphenation ---
#[test]
+261 -29
View File
@@ -4,11 +4,74 @@
//! gridlines. Many IRS forms and government PDFs use these instead of
//! `re` (rectangle) operators.
use std::collections::HashSet;
use crate::tables::Table;
use crate::types::{PdfLine, TextItem};
use super::detect_rects::{assign_items_to_grid, snap_edges};
/// Derive column edges from the x-endpoints of horizontal-rule
/// segments when no vertical lines were drawn.
///
/// Catalog and archival-finding-aid tables are commonly drawn with
/// per-row horizontal rules broken into N segments (one segment per
/// cell), with no vertical dividers at all. The segment break points
/// (e.g. `[50, 127], [127, 485], [485, 562]` per row) implicitly
/// encode the column boundaries.
///
/// Returns column edges if ≥3 distinct x-positions each show up as a
/// segment endpoint on ≥50% of the unique horizontal-line rows.
/// Returns `None` otherwise — decorative rules with varying widths
/// shouldn't be mistaken for a table.
fn derive_columns_from_horizontal_segments(horizontals: &[(f32, f32, f32)]) -> Option<Vec<f32>> {
if horizontals.len() < 3 {
return None;
}
let mut endpoints: Vec<f32> = Vec::with_capacity(horizontals.len() * 2);
for &(_, x_min, x_max) in horizontals {
endpoints.push(x_min);
endpoints.push(x_max);
}
let clusters = snap_edges(&endpoints, 5.0);
if clusters.len() < 3 {
return None;
}
// Bucket y-values to count unique rows. Tolerance ~0.1pt (×10
// rounding) tolerates the snap_edges 3pt clustering used later
// for row edges.
let unique_rows: HashSet<i32> = horizontals
.iter()
.map(|&(y, _, _)| (y * 10.0).round() as i32)
.collect();
if unique_rows.len() < 2 {
return None;
}
let min_rows = (unique_rows.len() as f32 * 0.5).ceil() as usize;
let qualifying: Vec<f32> = clusters
.iter()
.copied()
.filter(|&cluster_x| {
let rows_touched: HashSet<i32> = horizontals
.iter()
.filter(|&&(_, x_min, x_max)| {
(x_min - cluster_x).abs() < 5.0 || (x_max - cluster_x).abs() < 5.0
})
.map(|&(y, _, _)| (y * 10.0).round() as i32)
.collect();
rows_touched.len() >= min_rows
})
.collect();
if qualifying.len() < 3 {
return None;
}
Some(qualifying)
}
/// Detect tables from line segments on a given page.
///
/// Lines are classified as horizontal or vertical, snapped into grid edges,
@@ -52,25 +115,50 @@ pub fn detect_tables_from_lines(items: &[TextItem], lines: &[PdfLine], page: u32
// Diagonal lines are ignored
}
if horizontals.len() < 3 || verticals.len() < 2 {
if horizontals.len() < 3 {
return Vec::new();
}
// If no/very-few vertical lines are drawn, try to derive column edges
// from the x-endpoints of the horizontal-rule segments. Catalog and
// archival-finding-aid layouts commonly draw each row's horizontal
// rule as N segments (one per cell), with no vertical dividers at
// all — the segment break points encode the column boundaries.
let implicit_col_edges: Option<Vec<f32>> = if verticals.len() < 2 {
derive_columns_from_horizontal_segments(&horizontals)
} else {
None
};
if verticals.len() < 2 && implicit_col_edges.is_none() {
return Vec::new();
}
let cols_from_segments = implicit_col_edges.is_some();
log::debug!(
"detect_lines p{}: {} horiz, {} vert lines (of {} total on page)",
"detect_lines p{}: {} horiz, {} vert lines (of {} total on page){}",
page,
horizontals.len(),
verticals.len(),
page_lines.len()
page_lines.len(),
if cols_from_segments {
" — columns from horizontal segments"
} else {
""
}
);
// Snap Y-values of horizontal lines → row edges
let h_ys: Vec<f32> = horizontals.iter().map(|(y, _, _)| *y).collect();
let row_edges = snap_edges(&h_ys, 3.0);
// Snap X-values of vertical lines → column edges
let v_xs: Vec<f32> = verticals.iter().map(|(x, _, _)| *x).collect();
let col_edges = snap_edges(&v_xs, 3.0);
// Column edges from drawn verticals when present, else from the
// horizontal-segment endpoints derived above.
let col_edges = if let Some(c) = implicit_col_edges {
c
} else {
let v_xs: Vec<f32> = verticals.iter().map(|(x, _, _)| *x).collect();
snap_edges(&v_xs, 3.0)
};
log::debug!(
"detect_lines p{}: {} row edges, {} col edges after snap",
@@ -110,15 +198,21 @@ pub fn detect_tables_from_lines(items: &[TextItem], lines: &[PdfLine], page: u32
return Vec::new();
}
// Reject page-spanning frames: if the grid covers >90% of a standard page
// dimension in both axes, it's a border frame, not a table.
// Reject page-spanning frames: a decorative outer border has just 4
// edges (top/bottom/left/right). Real full-page tables — common in
// governmental ledgers, financial reports, etc. — span the same A4 /
// Letter dimensions but have many internal row/column rules. Only
// reject when the line set looks like a bare frame, not a grid.
// Standard pages are ~595×842 (A4) or ~612×792 (Letter).
if table_width > 500.0 && table_height > 700.0 {
if table_width > 500.0 && table_height > 700.0 && horizontals.len() <= 4 && verticals.len() <= 4
{
log::debug!(
"detect_lines p{}: rejected — page-spanning frame ({:.0}×{:.0})",
"detect_lines p{}: rejected — page-spanning frame ({:.0}×{:.0}, {} h + {} v)",
page,
table_width,
table_height
table_height,
horizontals.len(),
verticals.len()
);
return Vec::new();
}
@@ -146,24 +240,33 @@ pub fn detect_tables_from_lines(items: &[TextItem], lines: &[PdfLine], page: u32
// Validate vertical lines: at least 2 should span a meaningful height.
// Full spanning (>30%) is ideal, but accept many shorter lines (>10%)
// for tables with partial column separators.
let spanning_v = verticals
.iter()
.filter(|(_, y_min, y_max)| (y_max - y_min) > table_height * 0.3)
.count();
let partial_v = verticals
.iter()
.filter(|(_, y_min, y_max)| (y_max - y_min) > table_height * 0.10)
.count();
if spanning_v < 2 && partial_v < 4 {
log::debug!(
"detect_lines p{}: rejected — {} spanning + {} partial V lines",
page,
spanning_v,
partial_v
);
return Vec::new();
}
// for tables with partial column separators. Skipped entirely when
// columns came from horizontal-segment endpoints — there are no
// vertical lines to validate against, and the segment-endpoint
// consistency check in `derive_columns_from_horizontal_segments`
// is the equivalent guard.
let spanning_v = if cols_from_segments {
0
} else {
let s = verticals
.iter()
.filter(|(_, y_min, y_max)| (y_max - y_min) > table_height * 0.3)
.count();
let p = verticals
.iter()
.filter(|(_, y_min, y_max)| (y_max - y_min) > table_height * 0.10)
.count();
if s < 2 && p < 4 {
log::debug!(
"detect_lines p{}: rejected — {} spanning + {} partial V lines",
page,
s,
p
);
return Vec::new();
}
s
};
// Row edges need to be in descending order (top of page = higher Y first)
let mut row_edges_desc = row_edges;
@@ -410,6 +513,135 @@ mod tests {
assert!(tables.is_empty());
}
#[test]
fn test_horizontal_segments_only_implicit_columns_accepted() {
// Catalog/finding-aid pattern: each row's horizontal rule is
// drawn as 3 segments at consistent x-endpoints (50, 127, 485,
// 562), with no vertical lines anywhere. The segment break
// points must be inferred as column edges.
let mut lines = Vec::new();
// Slightly uneven row spacing so the chart-gridline rejector
// (CV < 0.02) doesn't fire.
let row_ys = [80.0_f32, 145.0, 215.0, 280.0, 350.0, 415.0, 485.0];
for &y in &row_ys {
lines.push(make_hline(y, 50.0, 127.0, 1));
lines.push(make_hline(y, 127.0, 485.0, 1));
lines.push(make_hline(y, 485.0, 562.0, 1));
}
// Populate every cell so capture / density checks pass.
let mut items = Vec::new();
for w in row_ys.windows(2) {
let row_y = (w[0] + w[1]) / 2.0;
items.push(make_item("id", 80.0, row_y, 1));
items.push(make_item("description here", 200.0, row_y, 1));
items.push(make_item("date", 510.0, row_y, 1));
}
let tables = detect_tables_from_lines(&items, &lines, 1);
assert_eq!(
tables.len(),
1,
"horizontal-segment-only grid should be accepted"
);
let t = &tables[0];
assert!(
t.cells.len() >= 4,
"expected ≥4 rows, got {}",
t.cells.len()
);
assert_eq!(t.cells[0].len(), 3, "expected 3 columns");
}
#[test]
fn test_horizontal_segments_with_inconsistent_endpoints_rejected() {
// Decorative rules of varying widths shouldn't be detected as a
// table — each line has its own x-endpoints, no consistent
// column boundary survives the 50%-of-rows threshold.
let lines = vec![
make_hline(100.0, 50.0, 150.0, 1),
make_hline(200.0, 50.0, 220.0, 1),
make_hline(300.0, 50.0, 310.0, 1),
make_hline(400.0, 50.0, 470.0, 1),
];
let items = vec![
make_item("decorative", 100.0, 150.0, 1),
make_item("text", 100.0, 250.0, 1),
];
let tables = detect_tables_from_lines(&items, &lines, 1);
assert!(
tables.is_empty(),
"varying-width decorative rules should not be detected"
);
}
#[test]
fn test_page_spanning_bare_frame_rejected() {
// Just an outer A4-sized rectangle: 2 horizontals + 2 verticals.
// No internal structure → decorative border, not a table.
let lines = vec![
make_hline(20.0, 20.0, 575.0, 1), // top
make_hline(820.0, 20.0, 575.0, 1), // bottom
make_vline(20.0, 20.0, 820.0, 1), // left
make_vline(575.0, 20.0, 820.0, 1), // right
];
let items = vec![
make_item("title", 100.0, 100.0, 1),
make_item("body", 100.0, 200.0, 1),
];
let tables = detect_tables_from_lines(&items, &lines, 1);
assert!(
tables.is_empty(),
"Page-sized 4-edge frame should be rejected as decoration"
);
}
#[test]
fn test_page_spanning_grid_with_internal_lines_accepted() {
// Full-page table (governmental-ledger pattern): A4-sized grid
// that previously hit the "page-spanning frame" early reject
// before downstream validation could even look at it.
// Verticals span the full table height so we isolate the
// frame-vs-grid decision under test.
let mut lines = Vec::new();
// 13 horizontal rules: header + 12 row separators
let h_ys = [
22.5, 37.9, 95.5, 144.5, 184.9, 233.9, 291.7, 340.7, 415.8, 499.6, 574.7, 623.7, 698.8,
];
for &y in &h_ys {
lines.push(make_hline(y, 22.6, 566.6, 1));
}
// 7 column dividers spanning full table height.
let v_xs = [22.6, 66.3, 116.3, 186.6, 263.1, 493.5, 566.5];
for &x in &v_xs {
lines.push(make_vline(x, 22.5, 698.8, 1));
}
// Populate every cell so the capture-ratio + density checks pass.
let mut items = Vec::new();
for r in 0..(h_ys.len() - 1) {
let row_y = (h_ys[r] + h_ys[r + 1]) / 2.0;
for c in 0..(v_xs.len() - 1) {
let col_x = (v_xs[c] + v_xs[c + 1]) / 2.0;
items.push(make_item("x", col_x, row_y, 1));
}
}
let tables = detect_tables_from_lines(&items, &lines, 1);
assert_eq!(
tables.len(),
1,
"Full-page table with internal grid should be accepted"
);
let t = &tables[0];
assert!(
t.cells.len() >= 6,
"expected ≥6 rows, got {}",
t.cells.len()
);
assert!(
t.cells[0].len() >= 3,
"expected ≥3 columns, got {}",
t.cells[0].len()
);
}
#[test]
fn test_single_column_rejected() {
// Only 2 col edges (1 column) — not a table even with verticals
+635 -41
View File
@@ -231,6 +231,15 @@ pub fn detect_tables_from_rects(
rects: &[PdfRect],
page: u32,
) -> (Vec<Table>, Vec<RectHintRegion>) {
// Strip Image placeholders before column/row clustering — an image's bbox
// would otherwise show up as a spurious column edge. See `is_text_layout_item`.
let items_owned: Vec<TextItem> = items
.iter()
.filter(|i| crate::extractor::is_text_layout_item(i))
.cloned()
.collect();
let items = items_owned.as_slice();
// Filter rects on this page; normalize negative widths/heights; skip tiny rects.
let mut page_rects: Vec<(f32, f32, f32, f32)> = Vec::new(); // (x, y, w, h) normalized
for r in rects {
@@ -285,7 +294,11 @@ pub fn detect_tables_from_rects(
//
// Only remove when the container is a similarly-sized cell (height
// ratio < 4×), NOT when the container is a table-wide background
// that dwarfs the sub-rect.
// that dwarfs the sub-rect. Origin-anchored page-background rects
// also disqualify as containers — they normally exceed the 4× ratio,
// but when the sub-rect is itself a tall table-frame the ratio can
// fall under the gate, and dropping the frame collapses cluster
// adjacency between adjacent column-cell groups.
//
// Skip this O(n²) dedup when there are too many rects — pages with
// thousands of vector-drawing rects won't benefit from cell dedup.
@@ -295,9 +308,11 @@ pub fn detect_tables_from_rects(
page_rects.retain(|&(ax, ay, aw, ah)| {
let tol = 2.0;
!snapshot.iter().any(|&(bx, by, bw, bh)| {
let container_is_page_bg = bx < 5.0 && by < 5.0;
// b must strictly contain a (b is larger in area)
bw * bh > aw * ah * 1.2
&& bh < ah * 4.0 // container must be similarly sized, not a table background
&& !container_is_page_bg
&& bx <= ax + tol
&& (bx + bw) >= (ax + aw) - tol
&& by <= ay + tol
@@ -1388,13 +1403,15 @@ fn detect_row_stripe_table(
.max()
.unwrap_or(0);
// Allow longer cells for multi-column tables (descriptions in one column
// are common). Single-column or 2-column "tables" with giant cells are
// almost always layout backgrounds.
// are common). Narrow grids with giant cells are usually layout
// backgrounds — but only when the row count is also small. A 4+-row
// key/value table with one descriptive column reads as a real table
// on every other gate, so don't reject it on cell length alone.
let max_allowed = if num_cols >= 3 { 2000 } else { 500 };
if max_cell_len > max_allowed {
if max_cell_len > max_allowed && non_empty_rows < 4 {
debug!(
" row-stripe rejected: max cell length {} > {} (layout background)",
max_cell_len, max_allowed
" row-stripe rejected: max cell length {} > {} (layout background, {} rows)",
max_cell_len, max_allowed, non_empty_rows
);
return None;
}
@@ -1587,25 +1604,111 @@ fn detect_row_stripe_table_from_cell_rects(
return None;
}
// Derive columns from text X-position clustering
// Derive columns from text X-position clustering, but prefer rect
// X-edges when they already provide a tighter scaffold. Some PDFs draw
// only the row-index cells in the body plus a full header row; that is
// not dense enough for `try_build_grid`, but the header rects still define
// the real columns. Text starts inside wide cells can otherwise split the
// table into spurious sub-columns.
let columns = cluster_x_positions(&page_items, 15.0);
if columns.len() < 2 {
let text_col_edges = if columns.len() >= 2 {
let mut edges: Vec<f32> = Vec::with_capacity(columns.len() + 1);
let min_x = page_items.iter().map(|(_, i)| i.x).reduce(f32::min)?;
edges.push(min_x - 5.0);
for pair in columns.windows(2) {
edges.push((pair[0] + pair[1]) / 2.0);
}
let max_x_right = page_items
.iter()
.map(|(_, i)| i.x + i.width)
.reduce(f32::max)?;
edges.push(max_x_right + 5.0);
Some(edges)
} else {
None
};
let rect_col_edges = {
let mut x_vals = Vec::with_capacity(content_rects.len() * 2);
for &&(x, _, w, _) in &content_rects {
x_vals.push(x);
x_vals.push(x + w);
}
let mut edges = snap_edges(&x_vals, 6.0);
edges.sort_by(|a, b| a.total_cmp(b));
if (3..=26).contains(&edges.len()) {
Some(edges)
} else {
None
}
};
// For wired-grid tables whose header text is centered/right-aligned but
// whose data is left-aligned, cluster_x_positions can drop the header-only
// x-cluster in its singleton-filter pass and merge adjacent data clusters
// when the gap is below threshold, losing a column. Rect borders are
// ground truth in that case — but only when each rect column actually
// holds text. Decorative or background rects (prose laid out in a frame,
// cell-fill rects with extra borders) can produce more rect-derived
// columns than the text supports; preferring rects there would split a
// logical column into spurious sub-columns.
let rect_cols_match_text = match (&rect_col_edges, &text_col_edges) {
(Some(rect_edges), _) if rect_edges.len() >= 4 => {
let num_rect_cols = rect_edges.len() - 1;
let mut col_item_counts = vec![0usize; num_rect_cols];
for (_, item) in &page_items {
let cx = item.x + item.width / 2.0;
for c in 0..num_rect_cols {
if cx >= rect_edges[c] - 2.0 && cx <= rect_edges[c + 1] + 2.0 {
col_item_counts[c] += 1;
break;
}
}
}
// Require every rect column to hold multiple text items. A rect
// column with no (or only one) item is decorative or the rect grid
// is detecting a spurious column the data does not need; in those
// cases the old text-cluster preference is the safer fallback.
col_item_counts.iter().all(|&n| n >= 2)
}
_ => false,
};
let (col_edges, columns_from_text) = match (rect_col_edges, text_col_edges) {
(Some(rect_edges), text_edges_opt) if rect_cols_match_text => {
debug!(
" cell-rect using {} rect-derived columns (text clusters: {}; rect cols well-distributed)",
rect_edges.len() - 1,
text_edges_opt
.as_ref()
.map(|e| (e.len() - 1) as i32)
.unwrap_or(-1)
);
(rect_edges, false)
}
(Some(rect_edges), Some(text_edges)) if rect_edges.len() <= text_edges.len() => {
debug!(
" cell-rect using {} rect-derived columns over {} text clusters",
rect_edges.len() - 1,
text_edges.len() - 1
);
(rect_edges, false)
}
(_, Some(text_edges)) => (text_edges, true),
(Some(rect_edges), None) => (rect_edges, false),
(None, None) => {
debug!(
" cell-rect rejected: only {} columns from text clustering",
columns.len()
);
return None;
}
};
if col_edges.len() < 3 {
return None;
}
// Build column edges
let mut col_edges: Vec<f32> = Vec::with_capacity(columns.len() + 1);
let min_x = page_items.iter().map(|(_, i)| i.x).reduce(f32::min)?;
col_edges.push(min_x - 5.0);
for pair in columns.windows(2) {
col_edges.push((pair[0] + pair[1]) / 2.0);
}
let max_x_right = page_items
.iter()
.map(|(_, i)| i.x + i.width)
.reduce(f32::max)?;
col_edges.push(max_x_right + 5.0);
let num_cols = col_edges.len() - 1;
let num_rows = row_edges.len() - 1;
@@ -1617,12 +1720,25 @@ fn detect_row_stripe_table_from_cell_rects(
page_items.len()
);
let (cells, item_indices) = assign_items_to_grid(items, &col_edges, &row_edges, page);
let (mut cells, item_indices) = assign_items_to_grid(items, &col_edges, &row_edges, page);
if item_indices.is_empty() {
return None;
}
let mut row_edges = row_edges;
let (collapsed_cells, collapsed_row_edges, collapsed_rows) =
collapse_multiline_description_rows(cells, row_edges, &col_edges);
let has_wrapped_description_rows = collapsed_rows > 0;
cells = collapsed_cells;
row_edges = collapsed_row_edges;
if collapsed_rows > 0 {
debug!(
" cell-rect collapsed {} wrapped description rows",
collapsed_rows
);
}
// Validate: >=2 non-empty rows, >=25% density
let non_empty_rows = cells
.iter()
@@ -1636,6 +1752,7 @@ fn detect_row_stripe_table_from_cell_rects(
return None;
}
let num_rows = cells.len();
let total_cells = (num_cols * num_rows) as f32;
let non_empty_cells = cells
.iter()
@@ -1655,17 +1772,21 @@ fn detect_row_stripe_table_from_cell_rects(
return None;
}
// Reject tables with paragraph-length cells (layout backgrounds, not tables)
// Reject tables with paragraph-length cells — typically layout
// backgrounds (sidebars, banners) where a single big rectangle
// contains a wall of prose. Spare multi-row key/value tables where
// the value column is a multi-bullet description: those pass every
// other gate and shouldn't get killed on cell length alone.
let max_cell_len = cells
.iter()
.flat_map(|row| row.iter())
.map(|c| c.len())
.max()
.unwrap_or(0);
if max_cell_len > 500 {
if max_cell_len > 500 && non_empty_rows < 4 {
debug!(
" cell-rect rejected: max cell length {} > 500",
max_cell_len
" cell-rect rejected: max cell length {} > 500 ({} rows, layout background)",
max_cell_len, non_empty_rows
);
return None;
}
@@ -1681,13 +1802,36 @@ fn detect_row_stripe_table_from_cell_rects(
// Reject "tables" that are actually prose in a framed region.
// Columns here come from text X-position clustering; when prose wraps
// inside a bounding-box rect (e.g. chat-transcript figures) the
// word-boundary gaps cluster into many spurious columns, and the
// resulting cells hold sentence fragments riddled with common English
// function words. Count cells with any such word and reject when
// 20%+ of non-empty cells match — real tabular data (labels, units,
// numbers) rarely contains these words.
if num_cols >= 4 {
// inside a bounding-box rect (e.g. chat-transcript figures, two-column
// legal-text blocks in forms) the word-boundary gaps cluster into
// spurious columns, and the resulting cells hold sentence fragments
// riddled with common English function words.
//
// Apply at any column count >= 2. The 2-col case is the bite — a
// paragraph wrapped into 2 justified columns produces the same
// surface signal as a real "label / value" table in the
// well-distributed-cols check (both cols populated), so we need a
// content-based signal to tell them apart.
//
// Layered checks combine after the 20%-of-cells prose-word
// trigger fires:
// (a) Long-cell content: prose-in-a-frame averages ~70-100 chars
// per non-empty cell (sentence fragments); real data tables
// are typically <30 chars, occasionally up to ~55 for
// descriptive 4-col tables. The 65-char threshold cleanly
// separates them on observed fixtures (accessory_building
// prose=74 chars, upstage data=53, greencomp=20). This
// overrides the well-distributed relaxation — long cells
// are the strongest prose signal even when both cols are
// populated.
// (b) Two-column text-only scaffold: when both columns were inferred
// from text starts rather than rect edges, prose fragments can look
// perfectly balanced. Require rect evidence for this relaxed shape.
// (c) Well-distributed columns: ≥75% of cols hold ≥2 non-empty
// cells. Catches the prose-paragraph-as-many-cols shape
// while admitting real "label / value / description /
// benefit"-style tables.
if num_cols >= 2 {
const PROSE_WORDS: &[&str] = &[
"a", "an", "the", "of", "to", "is", "was", "are", "were", "be", "been", "in", "on",
"at", "with", "for", "by", "as", "and", "or", "but", "this", "that", "these", "those",
@@ -1697,6 +1841,7 @@ fn detect_row_stripe_table_from_cell_rects(
];
let mut prose_cells = 0usize;
let mut counted = 0usize;
let mut total_chars = 0usize;
for row in &cells {
for cell in row {
let t = cell.trim();
@@ -1704,6 +1849,7 @@ fn detect_row_stripe_table_from_cell_rects(
continue;
}
counted += 1;
total_chars += t.chars().count();
let lower = t.to_ascii_lowercase();
let has_prose_word = lower
.split(|c: char| !c.is_ascii_alphabetic() && c != '\'')
@@ -1714,11 +1860,65 @@ fn detect_row_stripe_table_from_cell_rects(
}
}
if counted > 0 && prose_cells * 5 >= counted {
// (a) Long-cell content: overrides the well-distributed
// relaxation. The 2-col prose-in-a-frame case populates
// both cols (passes well-distributed) but every cell
// holds a sentence fragment, so mean cell length is the
// discriminator.
const PROSE_MEAN_CHAR_THRESHOLD: usize = 65;
let mean_chars = total_chars / counted;
if mean_chars > PROSE_MEAN_CHAR_THRESHOLD && !has_wrapped_description_rows {
debug!(
" cell-rect rejected: prose-in-frame, mean non-empty cell {} chars > {} (prose words {}/{})",
mean_chars, PROSE_MEAN_CHAR_THRESHOLD, prose_cells, counted
);
return None;
} else if mean_chars > PROSE_MEAN_CHAR_THRESHOLD {
debug!(
" cell-rect prose check relaxed: wrapped description rows, mean {} chars (prose words {}/{})",
mean_chars, prose_cells, counted
);
}
// (b) Two text-derived columns are not enough vector evidence once
// the content looks prose-like. Real 2-col rect tables still pass
// when the column scaffold comes from drawn cell geometry.
if columns_from_text && num_cols == 2 {
debug!(
" cell-rect rejected: prose-in-frame with text-derived 2-col scaffold (mean {} chars, prose words {}/{})",
mean_chars, prose_cells, counted
);
return None;
}
// (c) Well-distributed columns.
let filled_cols = (0..num_cols)
.filter(|&c| {
cells
.iter()
.filter(|row| {
!row.get(c)
.map(String::as_str)
.unwrap_or("")
.trim()
.is_empty()
})
.count()
>= 2
})
.count();
let well_distributed = filled_cols * 4 >= num_cols * 3;
if !well_distributed {
debug!(
" cell-rect rejected: {}/{} cells contain prose function words — likely prose ({}/{} cols filled, mean {} chars)",
prose_cells, counted, filled_cols, num_cols, mean_chars
);
return None;
}
debug!(
" cell-rect rejected: {}/{} cells contain prose function words — likely prose",
prose_cells, counted
" cell-rect prose check relaxed: {}/{} cols filled, mean {} chars — table-with-description-col",
filled_cols, num_cols, mean_chars
);
return None;
}
}
@@ -1739,6 +1939,132 @@ fn detect_row_stripe_table_from_cell_rects(
Some(Table::new(column_centers, row_centers, cells, item_indices))
}
/// Merge wrapped description-line bands back into their visual data rows.
///
/// Some Word/PDF exports draw enough rectangle geometry to prove a table exists
/// but expose Y bands per wrapped text line instead of per cell row. In the
/// common mapping-table shape, a narrow row-label column precedes one wide
/// description column, and wrapped continuation bands have content only in that
/// wide column. Merge only that high-confidence shape so framed prose still
/// falls through the existing prose guards.
fn collapse_multiline_description_rows(
cells: Vec<Vec<String>>,
row_edges: Vec<f32>,
col_edges: &[f32],
) -> (Vec<Vec<String>>, Vec<f32>, usize) {
let num_rows = cells.len();
let num_cols = col_edges.len().saturating_sub(1);
if num_rows < 3 || num_cols < 3 || row_edges.len() != num_rows + 1 {
return (cells, row_edges, 0);
}
let table_width = col_edges[num_cols] - col_edges[0];
if table_width <= 0.0 {
return (cells, row_edges, 0);
}
let Some((description_col, description_width)) = (0..num_cols)
.map(|c| (c, col_edges[c + 1] - col_edges[c]))
.max_by(|a, b| a.1.total_cmp(&b.1))
else {
return (cells, row_edges, 0);
};
// Require a preceding row-label column. Without it (e.g. a prose frame
// split into text-start columns), "one populated wide column" is not enough
// evidence to find visual row starts safely.
if description_col == 0 || description_width < table_width * 0.35 {
return (cells, row_edges, 0);
}
let row_has_left_label = |row: &[String]| {
row.iter()
.take(description_col)
.any(|cell| !cell.trim().is_empty())
};
let labeled_rows = cells.iter().filter(|row| row_has_left_label(row)).count();
if labeled_rows < 2 {
return (cells, row_edges, 0);
}
let mut merged_rows = 0usize;
let mut wrapped_description_rows = 0usize;
let mut new_cells: Vec<Vec<String>> = Vec::with_capacity(num_rows);
let mut new_edges = Vec::with_capacity(row_edges.len());
new_edges.push(row_edges[0]);
for (row_idx, row) in cells.into_iter().enumerate() {
let desc_text = row
.get(description_col)
.map(String::as_str)
.unwrap_or("")
.trim();
let left_label = row_has_left_label(&row);
let non_desc_non_empty = row
.iter()
.enumerate()
.filter(|(col, cell)| *col != description_col && !cell.trim().is_empty())
.count();
// Wrapped continuation bands contain only description-column text.
// The preceding label/marker column is empty because the visual row's
// label cell spans the whole wrapped block.
let is_description_continuation = row_idx > 0
&& !desc_text.is_empty()
&& !left_label
&& non_desc_non_empty == 0
&& !new_cells.is_empty();
// Header cells are often split as "Controls" / "Version" in the first
// column while the other header labels sit on the first band.
let only_first_col = row
.iter()
.enumerate()
.all(|(col, cell)| col == 0 || cell.trim().is_empty());
let is_header_continuation = row_idx > 0
&& only_first_col
&& row
.first()
.is_some_and(|cell| !cell.trim().is_empty() && cell.chars().count() <= 24)
&& !new_cells.is_empty()
&& new_cells
.last()
.is_some_and(|prev| prev.iter().filter(|c| !c.trim().is_empty()).count() >= 2);
if is_description_continuation || is_header_continuation {
if let Some(prev) = new_cells.last_mut() {
for (col, cell) in row.iter().enumerate() {
let text = cell.trim();
if text.is_empty() {
continue;
}
if !prev[col].trim().is_empty() {
prev[col].push(' ');
}
prev[col].push_str(text);
}
}
merged_rows += 1;
if is_description_continuation {
wrapped_description_rows += 1;
}
} else {
if !new_cells.is_empty() {
new_edges.push(row_edges[row_idx]);
}
new_cells.push(row);
}
}
new_edges.push(*row_edges.last().unwrap());
if merged_rows == 0 || new_cells.len() < 2 || new_edges.len() != new_cells.len() + 1 {
return (new_cells, row_edges, 0);
}
(new_cells, new_edges, wrapped_description_rows)
}
/// Detect a table by merging all cluster rects into one group.
///
/// This handles clip-path PDFs where each column's cell rects form a separate
@@ -1874,18 +2200,20 @@ fn detect_merged_cluster_table(
return None;
}
// Reject if any cell has excessive text — layout background rects produce
// "cells" containing paragraphs, not short data-table values.
// Reject if any cell has excessive text — layout background rects
// produce "cells" containing paragraphs, not short data-table values.
// Multi-row key/value tables can legitimately have one column of
// long descriptive text, so only reject narrow-row layouts here.
let max_cell_len = cells
.iter()
.flat_map(|row| row.iter())
.map(|c| c.len())
.max()
.unwrap_or(0);
if max_cell_len > 500 {
if max_cell_len > 500 && non_empty_rows < 4 {
debug!(
" merged-cluster rejected: max cell length {} > 500 (layout background)",
max_cell_len
" merged-cluster rejected: max cell length {} > 500 ({} rows, layout background)",
max_cell_len, non_empty_rows
);
return None;
}
@@ -2315,6 +2643,46 @@ mod tests {
);
}
#[test]
fn test_row_stripe_accepts_multi_row_key_value_long_cells() {
// Multi-row 2-column key/value table where one value cell holds
// a paragraph (>500 chars). The old `max_cell_len > 500` check
// rejected this shape as a "layout background"; with the
// multi-row guard, it should be accepted.
let mut rects = Vec::new();
let row_h = 25.0_f32;
let y_top = 700.0_f32;
for i in 0..8 {
let y = y_top - (i as f32) * row_h;
rects.push((40.0, y, 510.0, row_h));
}
let mut items = Vec::new();
for i in 0..8 {
let row_center_y = y_top - (i as f32) * row_h + row_h / 2.0;
// Left column: short label
items.push(make_item(&format!("Field {}", i), 45.0, row_center_y, 10.0));
// Right column: short value, except the last row which is a paragraph
let value = if i == 7 {
"X".repeat(800)
} else {
"value".to_string()
};
items.push(make_item(&value, 300.0, row_center_y, 10.0));
}
let result = detect_row_stripe_table(&items, &rects, 1);
assert!(
result.is_some(),
"multi-row key/value table with one long cell should be accepted"
);
let t = result.unwrap();
assert!(
t.cells.len() >= 4,
"expected ≥4 rows, got {}",
t.cells.len()
);
assert_eq!(t.cells[0].len(), 2, "expected 2 columns");
}
// --- propagate_merged_cells ---
#[test]
@@ -2929,6 +3297,232 @@ mod tests {
// If tables were detected, that's also acceptable
}
#[test]
fn text_derived_two_col_prose_is_not_cell_rect_table() {
let page = 1;
let mut rects = Vec::new();
for row in 0..8 {
rects.push(PdfRect {
x: 50.0,
y: 100.0 + row as f32 * 20.0,
width: 180.0,
height: 18.0,
page,
});
}
let mut items = Vec::new();
let left = [
"the annual plan was revised",
"and the team noted changes",
"this section explains limits",
"with additional notes below",
"the policy was reviewed",
"and results are summarized",
"this appendix describes scope",
"with examples for reference",
];
let right = [
"for each area in the review",
"as part of the assessment",
"that were applied in context",
"to support the conclusion",
"for use by the committee",
"as shown in the narrative",
"that remain under discussion",
"to clarify the method",
];
for row in 0..8 {
let y = 104.0 + row as f32 * 20.0;
let mut left_item = make_item(left[row], 60.0, y, 9.0);
left_item.width = 50.0;
items.push(left_item);
let mut right_item = make_item(right[row], 150.0, y, 9.0);
right_item.width = 50.0;
items.push(right_item);
}
let (tables, _hints) = detect_tables_from_rects(&items, &rects, page);
assert!(
tables.is_empty(),
"text-derived two-column prose must not be accepted as a rect table; got {:?}",
tables
.iter()
.map(|t| (t.rows.len(), t.columns.len()))
.collect::<Vec<_>>()
);
}
#[test]
fn multiline_indented_description_rows_collapse_to_visual_rows() {
let page = 1;
let col_edges = [0.0, 60.0, 420.0, 460.0, 500.0, 540.0];
let row_edges = [
340.0, 320.0, 300.0, 270.0, 250.0, 230.0, 200.0, 180.0, 160.0,
];
let mut rects = Vec::new();
for row in 0..row_edges.len() - 1 {
let y_top = row_edges[row];
let y_bot = row_edges[row + 1];
for col in 0..col_edges.len() - 1 {
rects.push((
col_edges[col],
y_bot,
col_edges[col + 1] - col_edges[col],
y_top - y_bot,
));
}
}
let mut items = vec![
make_item("Controls", 8.0, 330.0, 9.0),
make_item("Control", 70.0, 330.0, 9.0),
make_item("IG 1", 428.0, 330.0, 9.0),
make_item("IG 2", 468.0, 330.0, 9.0),
make_item("IG 3", 508.0, 330.0, 9.0),
make_item("Version", 8.0, 310.0, 9.0),
make_item("v8", 20.0, 285.0, 9.0),
make_item(
"4.5 Implement and Manage a Firewall on End-User Devices",
70.0,
285.0,
9.0,
),
make_item("*", 438.0, 285.0, 9.0),
make_item("*", 478.0, 285.0, 9.0),
make_item("*", 518.0, 285.0, 9.0),
make_item("v7", 20.0, 215.0, 9.0),
make_item(
"9.4 Apply Host-based Firewalls or Port-Filtering",
70.0,
215.0,
9.0,
),
make_item("*", 478.0, 215.0, 9.0),
make_item("*", 518.0, 215.0, 9.0),
];
items.push(make_item(
"Implement and manage a host-based firewall or port-filtering tool",
84.0,
260.0,
8.0,
));
items.push(make_item(
"on end-user devices with a default-deny rule",
84.0,
240.0,
8.0,
));
items.push(make_item(
"Apply host-based firewalls or port filtering tools on end systems",
84.0,
190.0,
8.0,
));
items.push(make_item(
"and deny unauthorized network communication",
84.0,
170.0,
8.0,
));
let table = detect_row_stripe_table_from_cell_rects(&items, &rects, page)
.expect("expected multiline description table");
assert_eq!(table.columns.len(), 5);
assert_eq!(
table.rows.len(),
3,
"wrapped lines should collapse to header plus two data rows"
);
assert_eq!(table.cells[0][0], "Controls Version");
assert!(table.cells[1][1].contains("host-based firewall"));
assert!(table.cells[1][1].contains("default-deny rule"));
assert!(table.cells[2][1].contains("deny unauthorized"));
}
/// Wire-bordered 4-column table whose header text is centered/right-aligned
/// inside each cell while the data is left-aligned: cluster_x_positions
/// merges adjacent columns (data Item→EAN gap is below threshold) and
/// drops the header-only x-clusters in the filter pass, leaving only 3
/// text-derived columns. Rect borders are 4 columns of ground truth.
/// Before the fix the cell-rect path preferred text edges when they were
/// the smaller set — losing a column. After the fix, 3+ rect columns
/// always win.
#[test]
fn wired_header_data_misaligned_keeps_all_columns_from_rects() {
let page = 1;
// 4 cols: Item | EAN | Nombre | Cant
let col_xs = [380.0_f32, 410.0, 470.0, 660.0, 700.0];
// Header + 9 data rows at 15pt tall each (y descending).
let row_ys: Vec<f32> = (0..=10).map(|r| 400.0 - 15.0 * r as f32).collect();
let mut rects: Vec<(f32, f32, f32, f32)> = Vec::new();
for r in 0..10 {
let y_top = row_ys[r];
let y_bot = row_ys[r + 1];
for c in 0..4 {
rects.push((col_xs[c], y_bot, col_xs[c + 1] - col_xs[c], y_top - y_bot));
}
}
let mut items: Vec<TextItem> = Vec::new();
// Header row (y ≈ 392.5): headers sit further to the right than data
// because they are centered/right-aligned in the cells.
items.push(make_item("Item", 389.0, 392.5, 9.0));
items.push(make_item("EAN", 432.0, 392.5, 9.0));
items.push(make_item("Nombre", 552.0, 392.5, 9.0));
items.push(make_item("Cant", 672.0, 392.5, 9.0));
let names = [
"Arnes Frontal",
"Arnes Motor",
"Arnes Piso",
"Arnes Techo",
"Arnes Puerta",
"Arnes Tablero",
"Arnes Trasero",
"Arnes Lateral",
"Arnes Sensor",
];
for r in 0..9 {
let y = 377.5 - 15.0 * r as f32;
items.push(make_item(&(r + 1).to_string(), 396.0, y, 9.0));
items.push(make_item("7701023403016", 410.0, y, 9.0));
items.push(make_item(names[r], 480.0, y, 9.0));
items.push(make_item("1", 680.0, y, 9.0));
}
let table = detect_row_stripe_table_from_cell_rects(&items, &rects, page)
.expect("wired 4-column table with header/data x-misalignment must detect");
assert_eq!(
table.columns.len(),
4,
"expected 4 columns from rect borders; cells: {:?}",
table.cells
);
for c in 0..4 {
let any_populated = table.cells.iter().any(|row| !row[c].trim().is_empty());
assert!(
any_populated,
"column {} empty across all rows; cells: {:?}",
c, table.cells
);
}
// Header row populated in all 4 cells.
let header = &table.cells[0];
assert_eq!(header[0].trim(), "Item");
assert_eq!(header[1].trim(), "EAN");
assert_eq!(header[2].trim(), "Nombre");
assert_eq!(header[3].trim(), "Cant");
// First data row: Item="1", EAN, name, count="1" — no Item↔EAN merge.
let data1 = &table.cells[1];
assert_eq!(data1[0].trim(), "1");
assert_eq!(data1[1].trim(), "7701023403016");
assert!(data1[2].trim().contains("Arnes"));
assert_eq!(data1[3].trim(), "1");
}
#[test]
fn failed_cluster_no_hint_without_items() {
// Rects with no text items inside → no failed-cluster hint generated.
+269 -7
View File
@@ -160,6 +160,81 @@ fn starts_with_uppercase_word(cell: &str) -> bool {
.is_some_and(|c| c.is_uppercase())
}
fn starts_with_uppercase_alpha(cell: &str) -> bool {
cell.chars()
.find(|c| c.is_alphabetic())
.is_some_and(|c| c.is_uppercase())
}
fn starts_with_lowercase_alpha(cell: &str) -> bool {
cell.chars()
.find(|c| c.is_alphabetic())
.is_some_and(|c| c.is_lowercase())
}
fn starts_with_numbered_label(cell: &str) -> bool {
let trimmed = cell.trim_start();
let digit_count = trimmed.chars().take_while(|c| c.is_ascii_digit()).count();
digit_count > 0
&& digit_count <= 3
&& trimmed
.chars()
.nth(digit_count)
.is_some_and(|c| matches!(c, '.' | ')' | '-' | ':'))
}
fn alpha_word_count(cell: &str) -> usize {
cell.split_whitespace()
.filter(|word| word.chars().any(|c| c.is_alphabetic()))
.count()
}
fn looks_like_compact_entry_label(cell: &str) -> bool {
let trimmed = cell.trim();
if trimmed.len() < 3 || trimmed.len() > 80 {
return false;
}
if !starts_with_uppercase_alpha(trimmed) && !starts_with_numbered_label(trimmed) {
return false;
}
if trimmed.ends_with(['.', ',', ';', ':']) {
return false;
}
let words = alpha_word_count(trimmed);
(1..=6).contains(&words)
}
fn looks_like_plain_section_label(cell: &str) -> bool {
let trimmed = cell.trim();
if trimmed.len() < 4 || trimmed.len() > 40 {
return false;
}
if trimmed.ends_with(['.', ',', ';', ':']) || trimmed.contains(|ch: char| ch.is_ascii_digit()) {
return false;
}
if trimmed.len() <= 4 && trimmed.chars().all(|ch| !ch.is_lowercase()) {
return false;
}
trimmed
.chars()
.all(|ch| ch.is_alphabetic() || ch.is_whitespace() || matches!(ch, '&' | '/' | '-'))
&& starts_with_uppercase_alpha(trimmed)
&& (1..=4).contains(&alpha_word_count(trimmed))
}
fn ends_like_incomplete_phrase(cell: &str) -> bool {
let lower = cell.trim_end().to_ascii_lowercase();
lower.ends_with(" and")
|| lower.ends_with(" or")
|| lower.ends_with(',')
|| lower.ends_with('-')
|| lower.ends_with('/')
}
/// Clean up table cells: merge continuation rows, extract footnotes, remove empty rows
fn clean_table_cells(cells: &[Vec<String>]) -> (Vec<Vec<String>>, Vec<String>) {
let mut cleaned: Vec<Vec<String>> = Vec::new();
@@ -185,6 +260,9 @@ fn clean_table_cells(cells: &[Vec<String>]) -> (Vec<Vec<String>>, Vec<String>) {
continue;
}
let num_cols = row.len();
let filled_cells = row.iter().filter(|c| !c.trim().is_empty()).count();
// Check if this is a continuation row (first column is empty but others have content).
// A row with only 1 short non-empty cell (besides the first) is more likely a
// section sub-header (e.g. "JAN", "FEB") than overflow text — don't merge it.
@@ -222,31 +300,76 @@ fn clean_table_cells(cells: &[Vec<String>]) -> (Vec<Vec<String>>, Vec<String>) {
.iter()
.filter(|cell| starts_with_uppercase_word(cell))
.count();
let first_non_empty_col = row.iter().position(|c| !c.trim().is_empty());
let first_non_empty_cell = first_non_empty_col
.and_then(|idx| row.get(idx))
.map(|c| c.trim())
.unwrap_or("");
let title_like_later_cells = first_non_empty_col
.map(|idx| {
row.iter()
.skip(idx + 1)
.map(|c| c.trim())
.filter(|c| !c.is_empty() && starts_with_uppercase_alpha(c))
.count()
})
.unwrap_or(0);
let prev_first_cell_empty = cleaned
.last()
.and_then(|r| r.first())
.is_some_and(|c| c.trim().is_empty());
let prev_first_cell = cleaned
.last()
.and_then(|r| r.first())
.map(|c| c.trim())
.unwrap_or("");
let header_filled = cleaned
.first()
.map(|r| r.iter().filter(|c| !c.trim().is_empty()).count())
.unwrap_or(num_cols);
let looks_like_spanning_first_column_row = first_cell.is_empty()
&& row.len() >= 4
&& non_first_cells.len() == row.len().saturating_sub(1)
&& uppercase_leading_cells >= non_first_cells.len().saturating_sub(1);
// Hierarchical tables often use a row-spanned first column: sub-rows
// leave column 0 blank, then start a compact title-like label in
// column 1. Wrapped continuations in the existing fixtures start
// mid-sentence/lowercase ("continued text here", "with 3.5%...") or
// carry lowercase fragments in the later cells, so keep those mergeable.
let looks_like_hierarchical_subrow = first_cell.is_empty()
&& row.len() >= 3
&& first_non_empty_col == Some(1)
&& looks_like_compact_entry_label(first_non_empty_cell)
&& ((non_first_cells.len() >= 2 && title_like_later_cells > 0)
|| (non_first_cells.len() == 1
&& prev_first_cell_empty
&& alpha_word_count(first_non_empty_cell) >= 2));
let looks_like_new_first_column_entry = !first_cell.is_empty()
&& (starts_with_numbered_label(first_cell) || starts_with_uppercase_alpha(first_cell))
&& filled_cells >= 2
&& non_first_cells
.iter()
.any(|cell| looks_like_compact_entry_label(cell));
let looks_like_section_label_row = !first_cell.is_empty()
&& filled_cells == 1
&& header_filled >= 3
&& looks_like_plain_section_label(first_cell);
// Classic continuation: first cell empty, content in other cells
let is_classic_continuation = first_cell.is_empty()
&& !non_first_cells.is_empty()
&& !is_short_subheader
&& !looks_like_data_row
&& !looks_like_spanning_first_column_row
&& !looks_like_hierarchical_subrow
&& cleaned.len() > 1;
// Wrapped-cell continuation: row has fewer filled cells than the header
// row, suggesting it's overflow text from the previous row's cells.
// Only trigger when the previous row has significantly more filled cells.
let num_cols = row.len();
let filled_cells = row.iter().filter(|c| !c.trim().is_empty()).count();
let prev_filled = cleaned
.last()
.map(|r| r.iter().filter(|c| !c.trim().is_empty()).count())
.unwrap_or(0);
let header_filled = cleaned
.first()
.map(|r| r.iter().filter(|c| !c.trim().is_empty()).count())
.unwrap_or(num_cols);
// Merge when the row has significantly fewer filled cells than header.
// For wide tables (5+ cols), require ≤50% of header cells.
// For narrow tables (2-4 cols), require fewer than header cells.
@@ -257,11 +380,18 @@ fn clean_table_cells(cells: &[Vec<String>]) -> (Vec<Vec<String>>, Vec<String>) {
} else {
header_filled.saturating_sub(1)
};
let continues_wrapped_first_column_label = !first_cell.is_empty()
&& starts_with_lowercase_alpha(first_cell)
&& ends_like_incomplete_phrase(prev_first_cell);
let is_wrapped_continuation = cleaned.len() > 1
&& filled_cells <= max_filled_for_merge
&& prev_filled > filled_cells
&& (prev_filled > filled_cells
|| (continues_wrapped_first_column_label && prev_filled >= filled_cells))
&& !looks_like_data_row
&& !looks_like_spanning_first_column_row
&& !looks_like_hierarchical_subrow
&& !looks_like_new_first_column_entry
&& !looks_like_section_label_row
&& !is_short_subheader;
let is_continuation = is_classic_continuation || is_wrapped_continuation;
@@ -407,6 +537,44 @@ mod tests {
assert!(cleaned[1][1].contains("continued text here"));
}
#[test]
fn test_clean_table_cells_first_column_section_label_not_merged() {
let cells = vec![
vec![
"Properties".into(),
"Conditions".into(),
"Method".into(),
"Typical values".into(),
"Units".into(),
],
vec![
"Melt Flow Rate".into(),
"230 C/2.16 kg".into(),
"ASTM D1238".into(),
"3.0".into(),
"g/10 min".into(),
],
vec![
"Mechanical".into(),
"".into(),
"".into(),
"".into(),
"".into(),
],
vec![
"Tensile Stress at Yield".into(),
"50 mm/min".into(),
"ASTM D638".into(),
"31".into(),
"MPa".into(),
],
];
let (cleaned, _) = clean_table_cells(&cells);
assert_eq!(cleaned.len(), 4);
assert_eq!(cleaned[2][0], "Mechanical");
}
#[test]
fn test_clean_table_cells_short_subheader_not_merged() {
let cells = vec![
@@ -459,6 +627,100 @@ mod tests {
assert_eq!(cleaned[2][1], "Uncertainty around other copies");
}
#[test]
fn test_clean_table_cells_numbered_hierarchy_rows_not_overmerged() {
let cells = vec![
vec![
"Group".into(),
"Task".into(),
"Detail".into(),
"Benefit".into(),
],
vec![
"1. Group alpha".into(),
"Task setup and".into(),
"Begin setup".into(),
"Faster start".into(),
],
vec![
"".into(),
"management".into(),
"recommended profile".into(),
"with saved defaults".into(),
],
vec![
"2. Group beta and".into(),
"Storage setup".into(),
"Provides upload tools".into(),
"".into(),
],
vec![
"fine-tuning".into(),
"".into(),
"for filtered inputs".into(),
"service".into(),
],
vec![
"".into(),
"Label workspace".into(),
"Creates review sets".into(),
"Lets teams review".into(),
],
vec![
"".into(),
"Model training".into(),
"".into(),
"Supports custom model".into(),
],
];
let (cleaned, _) = clean_table_cells(&cells);
assert_eq!(cleaned.len(), 5);
assert_eq!(cleaned[1][0], "1. Group alpha");
assert_eq!(cleaned[1][1], "Task setup and management");
assert_eq!(cleaned[2][0], "2. Group beta and fine-tuning");
assert_eq!(cleaned[2][1], "Storage setup");
assert_eq!(cleaned[3][1], "Label workspace");
assert_eq!(cleaned[4][1], "Model training");
}
#[test]
fn test_clean_table_cells_partial_hierarchical_subrow_not_merged() {
let cells = vec![
vec![
"Group".into(),
"Task".into(),
"Detail".into(),
"Benefit".into(),
],
vec![
"Group A".into(),
"Alpha task".into(),
"Initial detail".into(),
"Initial benefit".into(),
],
vec![
"".into(),
"Beta task".into(),
"Parallel detail".into(),
"".into(),
],
vec![
"".into(),
"second line".into(),
"additional detail".into(),
"".into(),
],
];
let (cleaned, _) = clean_table_cells(&cells);
assert_eq!(cleaned.len(), 3);
assert_eq!(cleaned[1][1], "Alpha task");
assert_eq!(cleaned[2][0], "");
assert_eq!(cleaned[2][1], "Beta task second line");
assert_eq!(cleaned[2][2], "Parallel detail additional detail");
}
#[test]
fn test_clean_table_cells_full_width_continuation_row_still_merges_when_lowercase() {
let cells = vec![
+1233
View File
File diff suppressed because it is too large Load Diff
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+635 -5
View File
@@ -2,13 +2,14 @@
use pdf_inspector::detector::{estimate_page_count_from_bytes, DetectionConfig, ScanStrategy};
use pdf_inspector::extractor::group_into_lines;
use pdf_inspector::types::ItemType;
use pdf_inspector::types::TextLine;
use pdf_inspector::{
detect_pdf_type, detect_vector_grid_in_region_mem, extract_pages_markdown,
extract_pages_markdown_mem, extract_tables_in_regions_mem, extract_text,
extract_text_in_regions_mem, extract_text_with_positions, process_pdf_mem,
process_pdf_with_options, to_markdown, MarkdownOptions, PdfError, PdfOptions, PdfType,
TextItem,
extract_text_in_regions_mem, extract_text_with_positions, extract_text_with_positions_mem,
process_pdf_mem, process_pdf_with_options, to_markdown, MarkdownOptions, PdfError, PdfOptions,
PdfType, TextItem,
};
use std::collections::HashSet;
@@ -1189,9 +1190,38 @@ fn test_firecrawl_tagged_pdf_struct_tree() {
#[test]
fn test_identity_h_no_tounicode_suppresses_garbage() {
// shinagawa_identity_h.pdf uses YuGothic with Identity-H encoding and no
// ToUnicode CMap. The raw CID values look like random Latin characters.
// We should suppress the garbage and flag the page for OCR.
// usable ToUnicode CMap. The raw CID bytes (e.g. 0x08 0x37, 0x0E 0x0F)
// contain non-ASCII high bytes and previously fell through to the
// per-byte Latin-1 fallback, producing high-Latin-1 mojibake that
// `is_cid_garbage` flagged. The Type0/CID guard in
// `extract_text_from_operand` now emits one U+FFFD per CID instead of
// mojibake; `detect_encoding_issues` trips on that and suppresses the
// markdown / flags the page for OCR — so we still pass this test, but
// via the deliberate marker path rather than by accident.
let buf = std::fs::read("tests/fixtures/shinagawa_identity_h.pdf").unwrap();
// Pre-suppression check: the raw text items must contain the U+FFFD
// markers that prove the Type0/CID fallback fired. This pins the
// mechanism so a future regression that re-enables Latin-1 mojibake
// would fail loudly here, not just silently change the suppression
// chain to one that depends on `is_cid_garbage` + high-Latin-1 chars.
let items = pdf_inspector::extractor::extract_text_with_positions_mem(&buf).unwrap();
let combined: String = items.iter().map(|i| i.text.as_str()).collect();
assert!(
combined.contains('\u{FFFD}'),
"Type0/CID font with unparseable ToUnicode CMap should emit U+FFFD per CID; \
got {} chars: {:?}",
combined.len(),
&combined[..combined.len().min(100)]
);
assert!(
!combined
.chars()
.any(|c| ('\u{0080}'..='\u{00FF}').contains(&c)),
"Latin-1 mojibake (high bytes) must not leak from Type0/CID fallback; got: {:?}",
&combined[..combined.len().min(100)]
);
let result = pdf_inspector::process_pdf_mem(&buf).unwrap();
// Page 1 should be flagged for OCR
@@ -1619,6 +1649,33 @@ fn test_bits_pilani_page8_table_detection() {
assert!(!region.needs_ocr, "Page 8 table should still be detected");
}
#[test]
fn test_extract_tables_in_regions_uses_line_grid() {
// Stroked-grid table (m/l/S path operators forming a 2x2 grid).
// The heuristic text-only detector handles the same cells already,
// so this guards that the line-backed path doesn't regress: the
// markdown still contains all four data cells.
let buf = synthetic_vector_grid_pdf(false);
let results =
extract_tables_in_regions_mem(&buf, &[(0, vec![[40.0, 50.0, 220.0, 760.0]])]).unwrap();
let region = &results[0].regions[0];
assert!(
!region.needs_ocr,
"stroked-grid table should be extracted, got needs_ocr=true"
);
for tok in ["A1", "B1", "A2", "B2"] {
assert!(
region.text.contains(tok),
"expected '{tok}' in output, got: {}",
region.text
);
}
assert!(
region.text.contains('|'),
"expected pipe-delimited markdown"
);
}
// =========================================================================
// extract_tables_with_structure_mem tests (TSR-aware path)
// =========================================================================
@@ -2532,6 +2589,61 @@ fn test_auto_expands_under_counted_vector_grid_rows() {
);
}
#[test]
fn test_auto_keeps_wrapped_header_vector_grid_doc51() {
use pdf_inspector::{extract_tables_with_structure_auto_mem, TsrTableInput};
let buf = std::fs::read("tests/fixtures/government_positions_women.pdf").unwrap();
let crop = [0.0, 0.0, 612.0, 792.0];
let grid = detect_vector_grid_in_region_mem(&buf, 0, crop, 200.0)
.unwrap()
.expect("expected doc 51 vector grid");
assert_eq!(
grid.cell_bboxes.len(),
36,
"doc 51 should have a 9x4 vector grid"
);
let results = extract_tables_with_structure_auto_mem(
&buf,
&[TsrTableInput {
page: 0,
crop_pdf_pt_bbox: crop,
render_dpi: 200.0,
structure_tokens: grid.structure_tokens,
cell_bboxes: grid.cell_bboxes,
}],
)
.unwrap();
assert_eq!(results.len(), 1);
let r = &results[0];
assert!(
r.fallback_reason.is_none(),
"wrapped header/label text should not trigger heuristic fallback: {:?}\n{}",
r.fallback_reason,
r.markdown
);
let md = &r.markdown;
assert!(md.contains("Government Position"), "missing header: {md}");
assert!(
md.contains("Aquino Administration"),
"missing Aquino header: {md}"
);
assert!(
md.contains("Ramos Administration"),
"missing Ramos header: {md}"
);
assert!(
md.contains("City Municipal Councilor"),
"row label was truncated: {md}"
);
assert!(
!md.contains("|Position||Administration"),
"heuristic fallback split the header row: {md}"
);
}
#[test]
fn test_auto_returns_empty_inputs() {
use pdf_inspector::extract_tables_with_structure_auto_mem;
@@ -2945,3 +3057,521 @@ fn test_extract_pages_markdown_path_none_returns_all_pages() {
let result = extract_pages_markdown(path, None).unwrap();
assert_eq!(result.pages.len() as u32, page_count);
}
// ============================================================================
// PROBE: investigate dense-cell text-assignment failure mode (failure mode 2)
// ============================================================================
fn synthetic_wide_row_pdf() -> Vec<u8> {
use lopdf::content::{Content, Operation};
use lopdf::{dictionary, Document, Object, Stream};
let mut doc = Document::with_version("1.5");
let pages_id = doc.new_object_id();
let page_id = doc.new_object_id();
let font_id = doc.new_object_id();
let content_id = doc.new_object_id();
doc.objects.insert(
font_id,
dictionary! {
"Type" => "Font",
"Subtype" => "Type1",
"BaseFont" => "Helvetica",
}
.into(),
);
let operations = vec![
Operation::new("BT", vec![]),
Operation::new("Tf", vec!["F1".into(), 10.into()]),
Operation::new("Td", vec![20.into(), 700.into()]),
// A single Tj that visually spans multiple cells. This mirrors PDFs
// where a row's address/role/email columns are emitted as one literal
// string with embedded spaces, producing one wide TextItem.
Operation::new(
"Tj",
vec![Object::string_literal("Name JobTitle Email Phone")],
),
Operation::new("ET", vec![]),
];
let content = Content { operations }.encode().unwrap();
doc.objects
.insert(content_id, Stream::new(dictionary! {}, content).into());
doc.objects.insert(
page_id,
dictionary! {
"Type" => "Page",
"Parent" => pages_id,
"MediaBox" => vec![0.into(), 0.into(), 200.into(), 800.into()],
"Resources" => dictionary! {
"Font" => dictionary! {
"F1" => font_id,
},
},
"Contents" => content_id,
}
.into(),
);
doc.objects.insert(
pages_id,
dictionary! {
"Type" => "Pages",
"Kids" => vec![page_id.into()],
"Count" => 1,
}
.into(),
);
let catalog_id = doc.add_object(dictionary! {
"Type" => "Catalog",
"Pages" => pages_id,
});
doc.trailer.set("Root", catalog_id);
let mut bytes = Vec::new();
doc.save_to(&mut bytes).unwrap();
bytes
}
#[test]
fn test_extract_tables_with_structure_distributes_wide_item_across_cells() {
use pdf_inspector::{extract_tables_with_structure_cells_mem, TsrTableInput};
// Reproduces failure mode 2: a row of multi-token text rendered as one
// Tj produces a single wide TextItem that visually spans multiple cells.
// The current first-match-by-center routing parks the entire item in
// whichever cell holds the item's center, leaving the other cells empty.
// See production samples in scrape_id 019de788-ff41-... where 10-column
// grids ended up with row text packed into one cell.
let buf = synthetic_wide_row_pdf();
// Helvetica 10pt with width=0 falls back to char_count*font_size*0.5.
// "Name JobTitle Email Phone" is 25 chars → effective_width 125pt,
// text starts at PDF (20, 700), top-down y=[90, 100], char_w≈5pt.
// Tokens land at:
// "Name" chars 0-3 center≈x=30
// "JobTitle" chars 5-12 center≈x=65
// "Email" chars 14-18 center≈x=100
// "Phone" chars 20-24 center≈x=130
let cell_bboxes = vec![
poly(15.0, 88.0, 50.0, 102.0),
poly(50.0, 88.0, 85.0, 102.0),
poly(85.0, 88.0, 120.0, 102.0),
poly(120.0, 88.0, 155.0, 102.0),
];
let tokens: Vec<String> = [
"<table>",
"<tbody>",
"<tr>",
"<td></td>",
"<td></td>",
"<td></td>",
"<td></td>",
"</tr>",
"</tbody>",
"</table>",
]
.into_iter()
.map(String::from)
.collect();
let cells_lists = extract_tables_with_structure_cells_mem(
&buf,
&[TsrTableInput {
page: 0,
crop_pdf_pt_bbox: [0.0, 0.0, 200.0, 800.0],
render_dpi: 72.0,
structure_tokens: tokens,
cell_bboxes,
}],
)
.unwrap();
let cells = &cells_lists[0];
assert_eq!(cells.len(), 4);
assert_eq!(
cells[0].text, "Name",
"cell 0 should hold 'Name', got {:?}",
cells[0].text
);
assert_eq!(
cells[1].text, "JobTitle",
"cell 1 should hold 'JobTitle', got {:?}",
cells[1].text
);
assert_eq!(
cells[2].text, "Email",
"cell 2 should hold 'Email', got {:?}",
cells[2].text
);
assert_eq!(
cells[3].text, "Phone",
"cell 3 should hold 'Phone', got {:?}",
cells[3].text
);
}
// ============================================================================
// PROPER TEST: synthetic Type0/Identity-H PDF with malformed ToUnicode CMap
// ============================================================================
//
// Complements the existing real-PDF fixture `shinagawa_identity_h.pdf` by
// building a minimal Type0 / Identity-H font in process. We control:
// * the byte stream emitted by Tj (a 2-byte CID containing one high byte),
// * the malformed ToUnicode contents (junk bytes that won't parse), and
// * the DescendantFonts shape (just enough for `parse_type0_widths` to set
// `is_cid=true`, which is what the new guard in `extract_text_from_operand`
// keys off of).
// No fixture file or external license to worry about.
fn synthetic_type0_broken_tounicode_pdf() -> Vec<u8> {
use lopdf::content::{Content, Operation};
use lopdf::{dictionary, Document, Object, Stream};
let mut doc = Document::with_version("1.5");
let pages_id = doc.new_object_id();
let page_id = doc.new_object_id();
let font_id = doc.new_object_id();
let cid_font_id = doc.new_object_id();
let descriptor_id = doc.new_object_id();
let tounicode_id = doc.new_object_id();
let cid_system_info_id = doc.new_object_id();
let content_id = doc.new_object_id();
// Type0 font with Identity-H encoding and a broken ToUnicode reference.
doc.objects.insert(
font_id,
dictionary! {
"Type" => "Font",
"Subtype" => "Type0",
"BaseFont" => "AAAAAA+SyntheticCID",
"Encoding" => "Identity-H",
"DescendantFonts" => vec![cid_font_id.into()],
"ToUnicode" => tounicode_id,
}
.into(),
);
// CIDSystemInfo and a minimal CIDFontType2 descendant. parse_type0_widths
// walks DescendantFonts → returns FontWidthInfo with is_cid=true. That's
// the only thing the new Latin-1 guard needs to see.
doc.objects.insert(
cid_system_info_id,
dictionary! {
"Registry" => Object::string_literal("Adobe"),
"Ordering" => Object::string_literal("Identity"),
"Supplement" => 0,
}
.into(),
);
doc.objects.insert(
cid_font_id,
dictionary! {
"Type" => "Font",
"Subtype" => "CIDFontType2",
"BaseFont" => "AAAAAA+SyntheticCID",
"CIDSystemInfo" => cid_system_info_id,
"FontDescriptor" => descriptor_id,
"DW" => 1000,
}
.into(),
);
doc.objects.insert(
descriptor_id,
dictionary! {
"Type" => "FontDescriptor",
"FontName" => "AAAAAA+SyntheticCID",
"Flags" => 4,
"FontBBox" => vec![Object::Integer(-100), Object::Integer(-100), 1000.into(), 1000.into()],
"ItalicAngle" => 0,
"Ascent" => 800,
"Descent" => Object::Integer(-200),
"CapHeight" => 700,
"StemV" => 80,
}
.into(),
);
// Intentionally malformed ToUnicode stream — just junk bytes. ToUnicode
// CMap parsing will fail, so `font_cmaps.get_by_obj` returns None and
// `has_cmap` stays false. The reference still exists in the font dict,
// so `font_tounicode_refs` contains the entry — but the new guard now
// routes off `is_cid` from font_widths instead, which is robust to a
// failed CMap parse.
doc.objects.insert(
tounicode_id,
Stream::new(dictionary! {}, b"this is not a valid CMap stream".to_vec()).into(),
);
// Tj with a 2-byte CID stream containing a non-ASCII high byte.
// Pre-fix this would have decoded as Latin-1 to "\u{00CD}\u{00D9}" ("ÍÙ").
// Post-fix it should produce U+FFFD per CID.
let cid_bytes = vec![0xCD_u8, 0xD9, 0xCD, 0xD9];
let operations = vec![
Operation::new("BT", vec![]),
Operation::new("Tf", vec!["F0".into(), 12.into()]),
Operation::new("Td", vec![50.into(), 100.into()]),
Operation::new(
"Tj",
vec![Object::String(cid_bytes, lopdf::StringFormat::Hexadecimal)],
),
Operation::new("ET", vec![]),
];
let content = Content { operations }.encode().unwrap();
doc.objects
.insert(content_id, Stream::new(dictionary! {}, content).into());
doc.objects.insert(
page_id,
dictionary! {
"Type" => "Page",
"Parent" => pages_id,
"MediaBox" => vec![0.into(), 0.into(), 200.into(), 200.into()],
"Resources" => dictionary! {
"Font" => dictionary! {
"F0" => font_id,
},
},
"Contents" => content_id,
}
.into(),
);
doc.objects.insert(
pages_id,
dictionary! {
"Type" => "Pages",
"Kids" => vec![page_id.into()],
"Count" => 1,
}
.into(),
);
let catalog_id = doc.add_object(dictionary! {
"Type" => "Catalog",
"Pages" => pages_id,
});
doc.trailer.set("Root", catalog_id);
let mut bytes = Vec::new();
doc.save_to(&mut bytes).unwrap();
bytes
}
#[test]
fn test_synthetic_type0_broken_tounicode_emits_fffd_not_latin1_mojibake() {
let buf = synthetic_type0_broken_tounicode_pdf();
let items = pdf_inspector::extractor::extract_text_with_positions_mem(&buf).unwrap();
let combined: String = items.iter().map(|i| i.text.as_str()).collect();
// Mojibake leak check: 2-byte CID 0xCDD9 must NOT come out as "ÍÙ"
// (U+00CD U+00D9). That was the production scrape symptom.
assert!(
!combined.contains('\u{00CD}'),
"Latin-1 mojibake leaked from Type0 font: {combined:?}"
);
assert!(
!combined.contains('\u{00D9}'),
"Latin-1 mojibake leaked from Type0 font: {combined:?}"
);
// Marker presence: Type0/CID + non-ASCII bytes must produce U+FFFD so
// `detect_encoding_issues` can flag the page for OCR downstream.
assert!(
combined.contains('\u{FFFD}'),
"Type0 font with malformed ToUnicode CMap should emit U+FFFD per CID; got: {combined:?}"
);
// End-to-end check: the page is correctly routed to OCR.
let result = pdf_inspector::process_pdf_mem(&buf).unwrap();
assert!(
result.pages_needing_ocr.contains(&1),
"Type0 page with broken ToUnicode + non-ASCII bytes must be flagged for OCR; \
pages_needing_ocr={:?}",
result.pages_needing_ocr
);
}
// ============================================================================
// Image XObject emission
// ============================================================================
/// Build a minimal PDF containing one Image XObject placed at a known CTM.
/// `image_ctm` is the 6-element matrix applied to the unit square by the
/// `Do` operator (per PDF spec section 8.9.5 "Image Coordinate System").
/// For an axis-aligned image at `(x, y)` with size `w × h`, that's
/// `[w, 0, 0, h, x, y]`.
fn make_pdf_with_image(image_ctm: [f32; 6]) -> Vec<u8> {
let mut pdf = b"%PDF-1.4\n".to_vec();
let mut offsets = vec![0usize];
fn add_object(pdf: &mut Vec<u8>, offsets: &mut Vec<usize>, id: usize, body: &str) {
offsets.push(pdf.len());
pdf.extend_from_slice(format!("{id} 0 obj\n").as_bytes());
pdf.extend_from_slice(body.as_bytes());
pdf.extend_from_slice(b"\nendobj\n");
}
fn add_stream_object(
pdf: &mut Vec<u8>,
offsets: &mut Vec<usize>,
id: usize,
dict: &str,
stream_bytes: &[u8],
) {
offsets.push(pdf.len());
pdf.extend_from_slice(format!("{id} 0 obj\n").as_bytes());
pdf.extend_from_slice(
format!("<< {} /Length {} >>\nstream\n", dict, stream_bytes.len()).as_bytes(),
);
pdf.extend_from_slice(stream_bytes);
pdf.extend_from_slice(b"\nendstream\nendobj\n");
}
// 1: catalog → 2: pages → 3: page with XObject /Im0 → 4: content stream
// 5: font → 6: image XObject (1×1 grayscale)
add_object(
&mut pdf,
&mut offsets,
1,
"<< /Type /Catalog /Pages 2 0 R >>",
);
add_object(
&mut pdf,
&mut offsets,
2,
"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
);
add_object(
&mut pdf,
&mut offsets,
3,
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] \
/Resources << /Font << /F1 5 0 R >> /XObject << /Im0 6 0 R >> >> \
/Contents 4 0 R >>",
);
let [a, b, c, d, e, f] = image_ctm;
// BT/ET around a small text item just so the page isn't classified as
// image-only (which would route to a different code path). Then save
// graphics state, apply the image CTM, invoke Im0, restore.
let content = format!(
"BT /F1 12 Tf 100 700 Td (Hi) Tj ET\nq {} {} {} {} {} {} cm /Im0 Do Q",
a, b, c, d, e, f
);
add_stream_object(&mut pdf, &mut offsets, 4, "", content.as_bytes());
add_object(
&mut pdf,
&mut offsets,
5,
"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
);
// 1×1 grayscale image; the single byte is mid-gray. Contents don't
// matter to the extractor — it only cares about the XObject's
// /Subtype and the CTM at the `Do` operator.
let image_pixel = [128u8];
add_stream_object(
&mut pdf,
&mut offsets,
6,
"/Type /XObject /Subtype /Image /Width 1 /Height 1 \
/ColorSpace /DeviceGray /BitsPerComponent 8",
&image_pixel,
);
let xref_start = pdf.len();
pdf.extend_from_slice(format!("xref\n0 {}\n", offsets.len()).as_bytes());
pdf.extend_from_slice(b"0000000000 65535 f \n");
for offset in offsets.iter().skip(1) {
pdf.extend_from_slice(format!("{offset:010} 00000 n \n").as_bytes());
}
pdf.extend_from_slice(
format!(
"trailer\n<< /Size {} /Root 1 0 R >>\nstartxref\n{}\n%%EOF",
offsets.len(),
xref_start
)
.as_bytes(),
);
pdf
}
#[test]
fn test_extract_text_with_positions_emits_image_bboxes() {
// Place a 200×100 image at (50, 600) in PDF user space (origin
// bottom-left). The Do operator applies the CTM to a unit square,
// so for an axis-aligned image, CTM = [w, 0, 0, h, x, y].
let pdf = make_pdf_with_image([200.0, 0.0, 0.0, 100.0, 50.0, 600.0]);
let items = extract_text_with_positions_mem(&pdf).expect("extract");
let images: Vec<&TextItem> = items
.iter()
.filter(|i| matches!(i.item_type, ItemType::Image))
.collect();
assert_eq!(
images.len(),
1,
"expected exactly one Image item, got items: {:?}",
items
.iter()
.map(|i| (&i.text, &i.item_type))
.collect::<Vec<_>>()
);
let img = images[0];
assert!((img.x - 50.0).abs() < 0.01, "x={}", img.x);
assert!((img.y - 600.0).abs() < 0.01, "y={}", img.y);
assert!((img.width - 200.0).abs() < 0.01, "width={}", img.width);
assert!((img.height - 100.0).abs() < 0.01, "height={}", img.height);
assert_eq!(img.page, 1);
// text field carries the legacy `[Image: <resource-name>]` form that
// the markdown emitter already knows how to parse.
assert_eq!(img.text, "[Image: Im0]");
}
#[test]
fn test_image_xobject_bbox_handles_rotated_ctm() {
// 90° rotation CTM: a unit square at the origin maps to a square
// rotated counter-clockwise about (0,0), then translated to (200, 300).
// For a 100×100 image, that's CTM = [0, 100, -100, 0, 200, 300]
// (apply the rotation: (1,0) → (0,100); (0,1) → (-100,0)).
// The page-space corners are:
// (0,0) → (200, 300)
// (1,0) → (200, 400)
// (1,1) → (100, 400)
// (0,1) → (100, 300)
// → AABB: x=100..200 (w=100), y=300..400 (h=100).
let pdf = make_pdf_with_image([0.0, 100.0, -100.0, 0.0, 200.0, 300.0]);
let items = extract_text_with_positions_mem(&pdf).expect("extract");
let img = items
.iter()
.find(|i| matches!(i.item_type, ItemType::Image))
.expect("image item");
assert!((img.x - 100.0).abs() < 0.01, "x={}", img.x);
assert!((img.y - 300.0).abs() < 0.01, "y={}", img.y);
assert!((img.width - 100.0).abs() < 0.01, "width={}", img.width);
assert!((img.height - 100.0).abs() < 0.01, "height={}", img.height);
}
#[test]
fn test_image_emission_does_not_change_default_markdown() {
// Default `MarkdownOptions::include_images = false` — adding image
// emission MUST NOT make `extract_pages_markdown` start producing
// `![Image: …]` placeholders for everyone. Existing callers that
// upgrade should see no diff in their markdown.
let pdf = make_pdf_with_image([200.0, 0.0, 0.0, 100.0, 50.0, 600.0]);
let result = extract_pages_markdown_mem(&pdf, None).expect("extract");
assert_eq!(result.pages.len(), 1);
assert!(
!result.pages[0].markdown.contains("Image:"),
"default markdown leaked an image placeholder: {:?}",
result.pages[0].markdown
);
}
#[test]
fn test_markdown_options_default_has_include_images_false() {
// Explicit assertion so anyone flipping this back catches it in CI.
// See `MarkdownOptions::default` in src/markdown/mod.rs for the
// long-form rationale.
let opts = MarkdownOptions::default();
assert!(!opts.include_images);
}
+9 -3
View File
@@ -54,9 +54,14 @@ company other than a life insurance company shall make a return on Form 1120PC.
annual statement (or a pro forma annual statement), including the underwriting and investment exhibit for the year covered by such return.
||(3) Foreign insurance companies. The provisions of paragraphs (c)(1) and|
|---|---|
||(c)(2) of this section concerning the returns and statements of insurance companies subject to tax under section 801 or section 831 also apply to foreign insurance companies subject to tax under those sections, except that the copy of the annual statement required to be submitted with the return shall, in the case of a foreign insurance company that is not required to file an annual statement, be a copy of the pro forma annual statement relating to the United States business of such company. (4) Exception for insurance companies filing their Federal income tax returns electronically. If an insurance company described in paragraph (c)(1), (c)(2), or (c)(3) of this section files its Federal income tax return electronically, it should not include on or with such return its annual statement (or pro forma annual statement), or any portion thereof. Such statement must be available at all times for inspection by authorized Internal Revenue Service officers or employees and retained for so long as such statements may be material in the administration of any internal revenue law. See §1.6001-1(e). (5) Definition. For purposes of this section, the term annual statement means the annual statement, the form of which is approved by the National Association of Insurance Commissioners (NAIC), which is filed by an insurance company for the year with the insurance departments of States, Territories, and the District of|
(3) Foreign insurance companies. The provisions of paragraphs (c)(1) and
(c)(2) of this section concerning the returns and statements of insurance companies subject to tax under section 801 or section 831 also apply to foreign insurance companies subject to tax under those sections, except that the copy of the annual statement required to be submitted with the return shall, in the case of a foreign insurance company that is not required to file an annual statement, be a copy of the pro forma annual statement relating to the United States business of such company.
(4) Exception for insurance companies filing their Federal income tax returns
electronically. If an insurance company described in paragraph (c)(1), (c)(2), or
(c)(3) of this section files its Federal income tax return electronically, it should not include on or with such return its annual statement (or pro forma annual statement), or any portion thereof. Such statement must be available at all times for inspection by authorized Internal Revenue Service officers or employees and retained for so long as such statements may be material in the administration of any internal revenue law. See §1.6001-1(e).
(5) Definition. For purposes of this section, the term annual statement means
the annual statement, the form of which is approved by the National Association of Insurance Commissioners (NAIC), which is filed by an insurance company for the year with the insurance departments of States, Territories, and the District of
Columbia. The term annual statement also includes a pro forma annual statement if the insurance company is not required to file the NAIC annual statement.
@@ -201,3 +206,4 @@ CFR part or section where Current OMB identified or described control No.
Deputy Commissioner for Services and Enforcement.
Approved: May 19, 2006 Eric Solomon Acting Deputy Assistant Secretary of the Treasury (Tax Policy).