Compare commits

...
Author SHA1 Message Date
Abimael MartellandCursor 4bee4f993b chore(release): bump package versions to 1.14.2 (#382)
Ship the extractor resource bounds and layout fixes that landed since 1.14.1.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-13 14:13:59 -07:00
Abimael MartellandCursor 1719d24871 fix(tables): bound disjoint-rect clustering so overlap tests stay subquadratic (#381)
* fix(tables): bound disjoint-rect clustering so overlap tests stay subquadratic

MAX_CLUSTER_RECTS only helped when a component actually merged. Pairwise-disjoint drawing rects never hit that cap, so the all-pairs loop stayed O(n²). Sweep by left edge and cap AABB tests at 1e6.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tables): cap clustering overlap tests per rect, not globally

A page-wide AABB budget could be spent on a dense stack of disjoint drawings and never reach an independent table at a later X. Limit each rect to 256 later candidates so other X-ranges still cluster.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tables): cluster overlapping rects with a spatial grid

A per-rect cap in X-sort order could skip a same-X neighbor after 256 junk candidates. Hash rects into 64-pt cells and pair only inside each cell so independent regions still cluster and disjoint drawings stay subquadratic.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tables): cluster oversized rects via a bounded fallback

A span cap of 64 grid cells could omit the far end of a huge rect. Those rects now compare against every other rect (up to 32 oversized). Grid buckets are visited in sorted key order so union-find is deterministic.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tables): visit every oversized rect under a per-rect overlap budget

Dropping .take(32) on the oversized-span list so later page-wide rules still
union the cells they overlap. AABB tests stay capped per oversized rect.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tables): query overlapping grid cells for oversized cluster rects

Index-order scans starved later overlaps once a per-rect check cap filled
with disjoint drawings. Oversized spans now probe the cells they cover,
with Y-banded oversized-to-oversized unions so stacked page-wide rules
stay linear.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tables): range-query cluster grid cells for oversized rects

Scan only occupied rows in the oversized rect's Y range, then X-partition
those keys, so unrelated drawings are not visited. Band oversized-to-oversized
unions on the short axis instead of a per-rect huge-Y fallback.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tables): union crossing oversized cluster rects across orientation bands

Wide and tall page-spanning rules are indexed on different axes, so a
crossing pair never shared a bucket. Query the tall X-index from each wide
or dual-oversized rect, and insert dual-oversized spans into every coarse
Y cell they cover.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tables): skip quadratic wide-by-tall clustering when the product is huge

Cross-orientation union is only needed for a handful of page-spanning rules.
When |wide|×|tall| exceeds the per-cell pair cap, skip that pass so mixed
oversized drawings cannot go quadratic. Pair counts in a range query no
longer reset per band.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tables): count unique oversized candidates when querying X/Y bands

A tall rule occupying several X cells was charged once per cell against the
pair budget, which could skip a later overlapping partner. Deduplicate `j`
per query so the cap applies to distinct rects.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-13 13:34:03 -07:00
Abimael MartellandCursor f114e79c8b fix(detector): bound Tj/TJ operand lookback to the previous operator (#380)
* fix(detector): bound Tj/TJ operand lookback to the previous operator

A missing `[` before `TJ` walked the entire prefix for every operator, so a compact `] TJ` stream was quadratic. Stop each lookback at the previous text/font operator so total work stays linear.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(detector): skip strings and comments when scanning text operators

A `Tj` token inside a literal string was treated as an operator and pinned the lookback floor, so the real `Tj` could not see its operand. Skip literals, hex strings, and comments before matching operators.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(detector): skip inline image data before string/hex scanning

A `(` or `<` byte in `BI`/`ID` sample data could enter string or hex state and hide every later text operator. Jump from `BI` to `EI` before applying those delimiter states.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(detector): skip inline images by declared size, not the first EI

Sample bytes can contain `EI` followed by a token-like character. When Width/Height are present and the image is uncompressed, jump that many bytes before looking for `EI`; DCT images use JPEG EOI, and the generic scan requires the following bytes to look like PDF content.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(detector): only trust inline-image length when the dict is complete

Require Width, Height, bits-per-component, and a known color space before skipping by size; pad each row to a byte; treat image masks as 1-bit. Drop the post-EI binary heuristic so a following non-ASCII string does not hide later text operators.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(detector): keep a strict EI fallback for filtered inline images

Exact-length skips still accept a following non-ASCII string. Fallback scans require printable PDF after `EI` unless the next token starts a string, name, or array. Boolean image-mask values must end at a token boundary.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(detector): stop the EI printable check at the next string token

A fallback scan of `EI` then `BT (` plus high-byte text was rejected because the 16-byte window included the string payload. Count binary-ness only until `(`, `<`, `[`, or `/`.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(detector): treat Tj/TJ as operators only after a string/array closer

Inline-image EI scanning cannot be made complete in this heuristic, and each attempt produced a new counterexample. Count Tj/TJ only when the previous token is `)`, `>`, or `]`: that keeps `] TJ` lookback linear and ignores `Tj` inside `(Hello Tj World)` without parsing BI/ID/EI.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-13 10:49:58 -07:00
Abimael MartellandCursor 544538b99f fix(extractor): bound ToUnicode bfrange expansion during subset remap (#379)
* fix(extractor): bound ToUnicode bfrange expansion during subset remap

Repeated full-width beginbfrange entries were expanded into individual
CID inserts on every copy. Stop after 65,536 assignments, matching the
existing /W and Encoding caps.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(extractor): document bfrange remap truncation and assert visit count

The 65,536 cap counts overwrites so repeated ranges cannot keep expanding.
The test now checks the assignment count, not just HashMap size (u16 keys
are always ≤ 65,536).

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-13 09:19:57 -07:00
Abimael MartellandCursor c8ba909407 fix(extractor): bound Encoding CMap cidrange expansion (#375)
* fix(extractor): bound Encoding CMap cidrange expansion

Repeating full-width begincidrange declarations re-inserted the entire
16-bit domain on every copy. Stop after 65,536 assignments, matching the
existing /W cap.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(extractor): clarify Encoding cidrange cap counts insert operations

The bound includes overwrites so repeated full-width ranges cannot keep
working after the map is full. Unique-key coverage alone would re-open
the CPU blow-up.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(extractor): distinguish /W insert vs unique-key CID caps

Encoding cidrange and /W width assignment count every insert; the /W
unicode heuristic caps unique CIDs with the same 65,536 bound.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-13 08:36:28 -07:00
Abimael MartellandCursor 076183e2e4 fix(extractor): cap content-stream decode before allocating operators (#373)
* fix(extractor): cap content-stream decode before allocating operators

The 1M operation limit ran after lopdf materialized the full vector, so a
compact page of q/Q pairs could still abort under memory pressure.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(extractor): treat NUL and form-feed as PDF whitespace in op counting

Names must stop on the full PDF whitespace set so a following operator is
not absorbed into /Name, which would undercount and skip the decode cap.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(extractor): scan inline-image EI with the full PDF whitespace set

A missed EI terminator used to consume the rest of the stream and drop
later operators from the decode cap. If EI is absent, keep scanning.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-12 16:13:12 -07:00
Abimael Martell ec6e54afb8 fix(extractor): merge small-caps runs so they stop reading as table columns (#371)
* fix(xobjects): track text line matrix and handle T*/TL/'/"/Tc/Tw in Form XObjects

The Form XObject text extractor in xobjects.rs is a separate hand-rolled
implementation of the operator state machine in content_stream.rs, and it had
drifted well out of parity:

- No text line matrix (TLM). `Td`/`TD` were applied to the text matrix already
  advanced by `Tj`/`TJ`, so every line began where the previous line *ended*
  instead of at the line start. Lines marched off the right edge and were
  dropped as off-page.
- `T*` was not handled at all, so it never advanced to the next line.
- `TL`, `'` and `"` were missing, and `TD` never set the leading as a side
  effect.
- `Tc`/`Tw` were hardcoded to 0.0 when computing advance widths, drifting
  positions and inserting spurious spaces.
- Text state (Tc/Tw/TL/Tf) is part of the graphics state but was not saved or
  restored by `q`/`Q`.

This matters well beyond an edge case: producers that emit a page stream of
just `q /X Do Q` and put all content in a Form XObject are common in
print-to-PDF and typesetting workflows, so this parser is on the hot path for
whole classes of real documents.

Measured on nycourts.gov 199AD3d.pdf (1370 pages, PDFlib producer, every page
wrapped in a Form XObject, 1331 pages using T*), word recall against a
pdftotext reference goes from 19.2% to 97.7% — 116k extracted words to 582k
against a 578k-word reference. On a 10-page subset, sequence similarity goes
from 27.3% to 97.7%, against 99.8% for Mistral OCR.

pdf-evals: 195 passed / 7 failed, byte-identical to the origin/main baseline
with the same failure list — no regressions.

Adds 7 unit tests covering the line-matrix-relative `Td`, `T*`, `TD` setting
leading, `'`, `"`, `Tc` advance widths, and `q`/`Q` text-state restore, all
driven through a page whose content is only `q /X1 Do Q`.

* fix(xobjects): restore fill colour across q/Q in Form XObjects

A white fill set inside a q/Q pair leaked past the Q, so all subsequent
text was treated as invisible and dropped. Save and restore fill_is_white
with the rest of the graphics state.

This was the cause of several long-standing extraction failures where
whole passages went missing or degraded into per-character garbage:
cambridge_excerpt (+8.5KB of recovered text), MTUAeroEngines (+7.4KB),
2025_findings-acl_668 (+1.4KB), HTM_02-01_Part_A (+1.3KB), and
HuttoISDWorkPerks / ebgt7isj04ophcq, which both went from exploded
per-character tables to clean prose.

Adds a regression test that fails without the restore (the text after Q
is dropped entirely).

Reported by cubic on #369.

* fix(extractor): merge small-caps runs so they stop reading as table columns

Typesetters render small caps as a full-size capital immediately followed by
shrunken capitals in the same font — `(R) Tj` at 9.98pt, then `(OLANDO) Tj`
at 6.74pt, touching. The 20% font-size band in merge_text_items split those
into separate items, and since table column boundaries cluster on item *start*
positions (find_column_boundaries never consults widths), each fragment
started far enough from the last to become its own column.

The result was garbled pseudo-tables. From 199AD3d.pdf p.5:

  |OLANDO|COSTA|NIL|INGH||
  |---|---|---|---|---|
  |IANNE|ENWICK|ETER|OULTON||

  |R|T. A|, P.J.|A|C. S|
  |---|---|---|---|---|

now:

  |ROLANDO T. ACOSTA, P.J.|ANIL C. SINGH|
  |---|---|
  |DIANNE T. RENWICK|PETER H. MOULTON|

Adds is_small_caps_continuation, gated tightly enough to exclude the other
reasons a smaller run follows a larger one: superscripts and footnote markers
(requires an uppercase *letter*, so digits never qualify), drop caps (the
following body text is mixed case), and adjacent cells or separate words
(requires the runs to be visually contiguous). A small-caps junction is
mid-word, so it also suppresses the space that would otherwise be inserted
("T. A" + "COSTA" -> "T. ACOSTA", not "T. A COSTA").

On 199AD3d.pdf this drops false-positive tables from 65 to 52 and lifts word
recall against a pdftotext reference from 97.7% to 97.8%.

Verified by running both binaries over all 203 eval PDFs and diffing outputs:
19 differ, 184 byte-identical. Beyond the reporter volume the merge also fixes:

  Waters-Edge          two more garbled heading-tables — "##### B. C R S" plus
                       "||OMPLIANCE|EPORTING YSTEM|" became
                       "##### B. COMPLIANCE REPORTING SYSTEM"
  cn-student-handbook  six TOC entries had collapsed to their initials;
                       "U M S H..." is now
                       "UNIVERSITY MISSION AND STUDENT HANDBOOK..."
  546403               "(FAA)" + "ADVISORY CIRCULARS ( )" + a stray "CONT"
                       became "(FAA) ADVISORY CIRCULARS (CONT)"
  ERP-2025             "T ABLE B-1" -> "TABLE B-1"
  DMP-Keypad           "THINLINE" + stray "TM KEYPADS" -> "THINLINETM KEYPADS"
  zhaw / Stijn         subscripted math variables: "*T* *G*" -> "*TG*"

One known regression, called out rather than hidden:
PA_PVEM_Sen_Waldo_Fernández (+1689 bytes). Its running footer genuinely is
small caps, so the merge correctly assembles the text (400 items -> 275), but
the now-contiguous footer lines align well enough that the heuristic detector
turns the wrapped document title into a 4-column table where it previously
rendered as bold prose.

Attempting to fix that in the detector was a dead end and is not included
here: gating the `num_cols >= 3` bypass in has_table_like_content on "no cell
has >= 12 words" removed 143 tables across 44 files, including correct ones —
MCF5235RM went 505 -> 483 and its register glossary degraded into a fused
column plus a ~100-word cell, because rejecting a good candidate lets a worse
fallback win. No wordiness threshold separates the two cases: PA_PVEM's
longest cell is 14 words while MCF5235RM's legitimate cells are <= 10. A
positional signal (running-header/footer bands, or cross-page repetition) is
the way in, and belongs in its own change.

Adds 7 unit tests: the two-column small-caps row from the reporter volume, and
rejection of superscript digits, drop caps, separate words, lowercase
continuations, and out-of-band size ratios.

* fix(extractor): tighten small-caps gate to cross-band junctions only

Three findings from cubic on #371, all valid.

1. Space suppression was too broad. `is_small_caps_continuation` accepted size
   ratios up to 0.92, which is *inside* the 20% band that merge_text_items
   already treats as the same size. Two similarly-sized uppercase words with a
   small real word gap therefore had their space suppressed even though the
   normal path would have merged them correctly with a space ("SEE" + "ALSO" ->
   "SEEALSO"). The helper now requires the junction to *cross* the band, since
   rescuing junctions the band would break is its only purpose; within-band
   pairs keep the normal word-spacing logic. The band is now a shared
   MERGE_FONT_SIZE_BAND constant so the helper and its caller cannot drift.

2. A trailing digit was skipped. The backward search for "the capital we are
   continuing" skipped non-alphabetic characters, so text ending in a footnote
   marker ("ANGELA M. MAZZARELLI1") found the earlier uppercase letter and
   accepted the join. It now checks the actual trailing character.

   Rejecting every trailing digit outright turned out to cost real quality, so
   this keeps one narrow exception. In meetings_in_mass_july_2023 the source
   reads "TUESDAY, JULY 4TH" with the ordinal suffix set as a smaller run; a
   blanket digit rejection reverted that to "TUESDAY, JULY 4" plus a stray "TH"
   leaking onto the next line, which is what main produces and what pdftotext
   shows is wrong. Only the four English ordinal suffixes (TH/ST/ND/RD) may
   follow a digit; anything else after one is treated as a footnote marker and
   rejected, so the case cubic raised stays blocked.

3. A test's name did not match its data. `small_caps_merge_does_not_swallow_a_
   second_column` claimed to exercise the 72pt column gap but contained only
   the second column's items, so it just re-tested the happy-path merge. It now
   holds the full nine-item row and asserts exactly two merged results,
   "ROLANDO T. ACOSTA, P.J." and "ANIL C. SINGH", which genuinely exercises the
   gap.

Both new guards were verified to be load-bearing: removing either one makes its
test fail.

The document that motivated the change is unaffected — small caps there run at
a 0.675 ratio, far outside the band — and 199AD3d.pdf p.5 still produces the
correct two-column justices table.

900 unit tests pass (10 covering small caps), fmt and clippy clean.
2026-08-12 15:33:18 -07:00
Abimael Martell 89dd20d02c fix(xobjects): track text line matrix and handle T*/TL/'/"/Tc/Tw in Form XObjects (#369)
* fix(xobjects): track text line matrix and handle T*/TL/'/"/Tc/Tw in Form XObjects

The Form XObject text extractor in xobjects.rs is a separate hand-rolled
implementation of the operator state machine in content_stream.rs, and it had
drifted well out of parity:

- No text line matrix (TLM). `Td`/`TD` were applied to the text matrix already
  advanced by `Tj`/`TJ`, so every line began where the previous line *ended*
  instead of at the line start. Lines marched off the right edge and were
  dropped as off-page.
- `T*` was not handled at all, so it never advanced to the next line.
- `TL`, `'` and `"` were missing, and `TD` never set the leading as a side
  effect.
- `Tc`/`Tw` were hardcoded to 0.0 when computing advance widths, drifting
  positions and inserting spurious spaces.
- Text state (Tc/Tw/TL/Tf) is part of the graphics state but was not saved or
  restored by `q`/`Q`.

This matters well beyond an edge case: producers that emit a page stream of
just `q /X Do Q` and put all content in a Form XObject are common in
print-to-PDF and typesetting workflows, so this parser is on the hot path for
whole classes of real documents.

Measured on nycourts.gov 199AD3d.pdf (1370 pages, PDFlib producer, every page
wrapped in a Form XObject, 1331 pages using T*), word recall against a
pdftotext reference goes from 19.2% to 97.7% — 116k extracted words to 582k
against a 578k-word reference. On a 10-page subset, sequence similarity goes
from 27.3% to 97.7%, against 99.8% for Mistral OCR.

pdf-evals: 195 passed / 7 failed, byte-identical to the origin/main baseline
with the same failure list — no regressions.

Adds 7 unit tests covering the line-matrix-relative `Td`, `T*`, `TD` setting
leading, `'`, `"`, `Tc` advance widths, and `q`/`Q` text-state restore, all
driven through a page whose content is only `q /X1 Do Q`.

* fix(xobjects): restore fill colour across q/Q in Form XObjects

A white fill set inside a q/Q pair leaked past the Q, so all subsequent
text was treated as invisible and dropped. Save and restore fill_is_white
with the rest of the graphics state.

This was the cause of several long-standing extraction failures where
whole passages went missing or degraded into per-character garbage:
cambridge_excerpt (+8.5KB of recovered text), MTUAeroEngines (+7.4KB),
2025_findings-acl_668 (+1.4KB), HTM_02-01_Part_A (+1.3KB), and
HuttoISDWorkPerks / ebgt7isj04ophcq, which both went from exploded
per-character tables to clean prose.

Adds a regression test that fails without the restore (the text after Q
is dropped entirely).

Reported by cubic on #369.
2026-08-12 14:32:50 -07:00
Abimael MartellandCursor 3d33ff3dbd fix(extractor): bound CID /W range expansion (#372)
Type0 /W parsing and the Unicode-CID heuristic expanded every CID in every
range. Repeating a full-width [0 65535 w] entry therefore re-materialized
the same 65,536-key domain on every copy, growing a temporary vector and
HashMap work without bound.

Cap expansion at the 16-bit CID domain, collect unique CIDs for the
median heuristic, and stop width assignment once that many entries have
been written. Legitimate compact /W arrays are unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-12 14:27:29 -07:00
Abimael MartellandCursor 75e9b09593 fix(extractor): bound Form XObject expansion per page (#370)
* fix(extractor): bound Form XObject expansion with invocation and operation budgets

Nested Form XObjects were only limited by recursion depth (5). An acyclic
graph where each form invokes the next N times still expands to N^depth
work, so a small PDF can force millions of nested /Do evaluations.

Share a per-page FormWalkBudget that caps 10,000 Form invocations and
1,000,000 operations walked across those expansions. Extraction stops
when either cap is hit. Legitimate shallow nesting is unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(extractor): share Form XObject budget across invisible-layer retry

extract_page_text_items created a fresh FormWalkBudget on every call, so
the invisible-text retry could consume a second full expansion budget for
the same page. Own the budget at the call site and pass it into both
passes.

Also charge form operations independently of the invocation cap so a form
that was already admitted can finish its stream.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-12 14:03:19 -07:00
18 changed files with 1993 additions and 146 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "pdf-inspector"
version = "1.14.1"
version = "1.14.2"
edition = "2021"
autobins = false
authors = ["Firecrawl Team"]
+2 -2
View File
@@ -851,7 +851,7 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
[[package]]
name = "pdf-inspector"
version = "1.14.1"
version = "1.14.2"
dependencies = [
"env_logger",
"include_dir",
@@ -867,7 +867,7 @@ dependencies = [
[[package]]
name = "pdf-inspector-napi"
version = "1.14.1"
version = "1.14.2"
dependencies = [
"napi",
"napi-build",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "pdf-inspector-napi"
version = "1.14.1"
version = "1.14.2"
edition = "2021"
[lib]
+6 -6
View File
@@ -8,12 +8,12 @@
"@napi-rs/cli": "^3.4.1",
},
"optionalDependencies": {
"@firecrawl/pdf-inspector-darwin-arm64": "1.14.1",
"@firecrawl/pdf-inspector-linux-arm64-gnu": "1.14.1",
"@firecrawl/pdf-inspector-linux-arm64-musl": "1.14.1",
"@firecrawl/pdf-inspector-linux-x64-gnu": "1.14.1",
"@firecrawl/pdf-inspector-linux-x64-musl": "1.14.1",
"@firecrawl/pdf-inspector-win32-x64-msvc": "1.14.1",
"@firecrawl/pdf-inspector-darwin-arm64": "1.14.2",
"@firecrawl/pdf-inspector-linux-arm64-gnu": "1.14.2",
"@firecrawl/pdf-inspector-linux-arm64-musl": "1.14.2",
"@firecrawl/pdf-inspector-linux-x64-gnu": "1.14.2",
"@firecrawl/pdf-inspector-linux-x64-musl": "1.14.2",
"@firecrawl/pdf-inspector-win32-x64-msvc": "1.14.2",
},
},
},
+7 -7
View File
@@ -1,6 +1,6 @@
{
"name": "@firecrawl/pdf-inspector",
"version": "1.14.1",
"version": "1.14.2",
"description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
"main": "index.js",
"types": "index.d.ts",
@@ -52,11 +52,11 @@
"@napi-rs/cli": "^3.4.1"
},
"optionalDependencies": {
"@firecrawl/pdf-inspector-linux-x64-gnu": "1.14.1",
"@firecrawl/pdf-inspector-linux-x64-musl": "1.14.1",
"@firecrawl/pdf-inspector-linux-arm64-gnu": "1.14.1",
"@firecrawl/pdf-inspector-linux-arm64-musl": "1.14.1",
"@firecrawl/pdf-inspector-darwin-arm64": "1.14.1",
"@firecrawl/pdf-inspector-win32-x64-msvc": "1.14.1"
"@firecrawl/pdf-inspector-linux-x64-gnu": "1.14.2",
"@firecrawl/pdf-inspector-linux-x64-musl": "1.14.2",
"@firecrawl/pdf-inspector-linux-arm64-gnu": "1.14.2",
"@firecrawl/pdf-inspector-linux-arm64-musl": "1.14.2",
"@firecrawl/pdf-inspector-darwin-arm64": "1.14.2",
"@firecrawl/pdf-inspector-win32-x64-msvc": "1.14.2"
}
}
+1 -1
View File
@@ -6,7 +6,7 @@ build-backend = "maturin"
name = "pdf-inspector"
# Keep package versions in sync with `python3 scripts/version.py <version>`.
# CI publishes automatically when the synchronized change lands on main.
version = "1.14.1"
version = "1.14.2"
description = "Fast PDF inspection, classification, and text extraction with smart scanned vs text-based detection"
readme = "docs/python.md"
license = { text = "MIT" }
+1 -1
View File
@@ -975,7 +975,7 @@ result = pdf_inspector.<span class="fn">process_pdf</span>(<span class="str">"do
<script>
(() => {
const MAX_FILE_SIZE = 25 * 1024 * 1024;
const WASM_MODULE_URL = "https://cdn.jsdelivr.net/npm/@firecrawl/pdf-inspector-wasm@1.14.1/pdf_inspector_wasm.js";
const WASM_MODULE_URL = "https://cdn.jsdelivr.net/npm/@firecrawl/pdf-inspector-wasm@1.14.2/pdf_inspector_wasm.js";
const input = document.querySelector("#pdf-input");
const dropZone = document.querySelector("#drop-zone");
const filePanel = document.querySelector("#demo-file");
+98 -24
View File
@@ -1382,7 +1382,13 @@ fn scan_content_for_text_operators(
let is_word_end =
|pos: usize| -> bool { pos + 1 >= content.len() || content[pos + 1].is_ascii_whitespace() };
// Simple state machine to find operators
// Simple state machine to find operators.
// Each Tj/TJ/Tf lookback stops at the previous text/font operator so a
// malformed `] TJ` (no `[`) cannot rescan the entire prefix — that was
// quadratic in the number of operators.
// `Tj`/`TJ` are only counted when the preceding token closes a string or
// array (')', '>', ']'), so `Tj` inside `(Hello Tj World)` cannot pin the floor.
let mut operand_floor = 0usize;
let mut i = 0;
while i < content.len() {
let b = content[i];
@@ -1392,14 +1398,15 @@ fn scan_content_for_text_operators(
let next = content[i + 1];
if next == b'j' || next == b'J' {
// Verify it's an operator (followed by whitespace or newline)
if i + 2 >= content.len()
if (i + 2 >= content.len()
|| content[i + 2].is_ascii_whitespace()
|| content[i + 2] == b'\n'
|| content[i + 2] == b'\r'
|| content[i + 2] == b'\r')
&& preceding_operand_closer(content, i, operand_floor)
{
text_ops += 1;
// Scan backward for text string operand to collect unique chars
collect_text_chars_before(content, i, unique_chars);
collect_text_chars_before(content, i, unique_chars, operand_floor);
operand_floor = i;
}
} else if next == b'f' {
// Tf = set font operator
@@ -1415,12 +1422,10 @@ fn scan_content_for_text_operators(
|| content[i + 2] == b'<'
|| content[i + 2] == b'/'
{
font_changes += 1;
// Extract the font name operand preceding the size + Tf.
// Pattern: /FontName <size> Tf
// Scan backward past the size number and whitespace to find /Name.
if let Some(name) = extract_font_name_before_tf(content, i) {
if let Some(name) = extract_font_name_before_tf(content, i, operand_floor) {
used_font_names.insert(name);
font_changes += 1;
operand_floor = i;
}
}
}
@@ -1466,6 +1471,20 @@ fn scan_content_for_text_operators(
(text_ops, image_count, path_ops, font_changes)
}
/// True when the token before `op_pos` (skipping whitespace, not crossing
/// `floor`) is a string/array closer. Used so `Tj` inside `(Hello Tj World)`
/// is not treated as an operator.
fn preceding_operand_closer(content: &[u8], op_pos: usize, floor: usize) -> bool {
let mut j = op_pos;
while j > floor {
j -= 1;
if !content[j].is_ascii_whitespace() {
return matches!(content[j], b')' | b'>' | b']');
}
}
false
}
/// Extract the font name operand from content stream bytes preceding a Tf operator.
///
/// The Tf operator syntax is: `/FontName size Tf`
@@ -1473,25 +1492,27 @@ fn scan_content_for_text_operators(
/// whitespace to find the `/Name` token.
///
/// Returns the font name bytes (without the leading `/`), e.g. `b"F1"` for `/F1`.
fn extract_font_name_before_tf(content: &[u8], tf_pos: usize) -> Option<Vec<u8>> {
/// `floor` is the start of the previous text/font operator (or 0); lookback
/// must not cross it.
fn extract_font_name_before_tf(content: &[u8], tf_pos: usize, floor: usize) -> Option<Vec<u8>> {
// Scan backward past whitespace before "Tf"
let mut j = tf_pos;
while j > 0 && content[j - 1].is_ascii_whitespace() {
while j > floor && content[j - 1].is_ascii_whitespace() {
j -= 1;
}
// Scan backward past the size number (digits, '.', '-')
while j > 0
while j > floor
&& (content[j - 1].is_ascii_digit() || content[j - 1] == b'.' || content[j - 1] == b'-')
{
j -= 1;
}
// Scan backward past whitespace between font name and size
while j > 0 && content[j - 1].is_ascii_whitespace() {
while j > floor && content[j - 1].is_ascii_whitespace() {
j -= 1;
}
// Now j should point just after the font name. Scan backward to find '/'.
let name_end = j;
while j > 0 && content[j - 1] != b'/' {
while j > floor && content[j - 1] != b'/' {
// Font names consist of regular characters (not whitespace, not delimiters)
if content[j - 1].is_ascii_whitespace() || content[j - 1] == b'(' || content[j - 1] == b')'
{
@@ -1499,7 +1520,7 @@ fn extract_font_name_before_tf(content: &[u8], tf_pos: usize) -> Option<Vec<u8>>
}
j -= 1;
}
if j == 0 || content[j - 1] != b'/' {
if j <= floor || content[j - 1] != b'/' {
return None;
}
// j-1 is the '/', font name is content[j..name_end]
@@ -1514,16 +1535,24 @@ fn extract_font_name_before_tf(content: &[u8], tf_pos: usize) -> Option<Vec<u8>>
/// and collect unique non-whitespace bytes from it.
///
/// Handles both literal strings `(...)` and hex strings `<...>`.
fn collect_text_chars_before(content: &[u8], op_pos: usize, unique_chars: &mut HashSet<u8>) {
/// `floor` is the start of the previous text/font operator (or 0); lookback
/// must not cross it, or a missing `[` before `TJ` rescans the whole prefix.
fn collect_text_chars_before(
content: &[u8],
op_pos: usize,
unique_chars: &mut HashSet<u8>,
floor: usize,
) {
// Walk backward past whitespace to find the closing delimiter
let mut j = op_pos;
while j > 0 {
while j > floor {
j -= 1;
if !content[j].is_ascii_whitespace() {
break;
}
}
if j == 0 {
// All whitespace, or we landed on the previous operator token.
if j == floor {
return;
}
@@ -1533,7 +1562,7 @@ fn collect_text_chars_before(content: &[u8], op_pos: usize, unique_chars: &mut H
// Literal string: scan backward for matching '('
let mut depth = 1i32;
let mut k = j;
while k > 0 && depth > 0 {
while k > floor && depth > 0 {
k -= 1;
match content[k] {
b')' if k == 0 || content[k - 1] != b'\\' => depth += 1,
@@ -1552,7 +1581,7 @@ fn collect_text_chars_before(content: &[u8], op_pos: usize, unique_chars: &mut H
} else if closing == b'>' {
// Hex string: scan backward for '<'
let mut k = j;
while k > 0 {
while k > floor {
k -= 1;
if content[k] == b'<' {
break;
@@ -1582,7 +1611,7 @@ fn collect_text_chars_before(content: &[u8], op_pos: usize, unique_chars: &mut H
} else if closing == b']' {
// TJ array: scan backward for '[' and collect from all strings inside
let mut k = j;
while k > 0 {
while k > floor {
k -= 1;
if content[k] == b'[' {
break;
@@ -2014,6 +2043,51 @@ mod tests {
assert_eq!(imgs3, 0);
}
#[test]
fn test_scan_content_successive_tj_collects_each_operand() {
// Lookback is floored at the previous Tj/TJ/Tf so later operators must
// still see their own operands.
let content = b"[(Hello)] TJ [(World)] TJ (More) Tj";
let mut uchars = HashSet::new();
let (ops, _, _, _) =
scan_content_for_text_operators(content, &mut uchars, &mut HashSet::new());
assert_eq!(ops, 3);
for &ch in b"HeloWrdM" {
assert!(uchars.contains(&ch), "missing char {}", ch as char);
}
}
#[test]
fn test_scan_content_tj_inside_literal_is_not_an_operator() {
// `Tj` followed by space inside a literal must not count as an operator
// or pin the lookback floor; the real `Tj` still collects the string.
let content = b"BT (Hello Tj World) Tj ET";
let mut uchars = HashSet::new();
let (ops, _, _, _) =
scan_content_for_text_operators(content, &mut uchars, &mut HashSet::new());
assert_eq!(ops, 1);
for &ch in b"HeloTjWrd" {
assert!(uchars.contains(&ch), "missing char {}", ch as char);
}
}
#[test]
fn test_scan_content_malformed_tj_lookback_stays_linear() {
// `] TJ` with no `[` used to walk the entire prefix for every operator
// (quadratic). 30k repeats is enough that a prefix rescan would dominate
// the test runtime; with the floor it is a single linear pass.
let n = 30_000usize;
let mut content = Vec::with_capacity(n * 5);
for _ in 0..n {
content.extend_from_slice(b"] TJ\n");
}
let mut uchars = HashSet::new();
let (ops, _, _, _) =
scan_content_for_text_operators(&content, &mut uchars, &mut HashSet::new());
assert_eq!(ops, n as u32);
assert!(uchars.is_empty());
}
#[test]
fn test_image_dominated_detection() {
// Do operators are no longer counted as images by scan_content_for_text_operators.
@@ -2772,14 +2846,14 @@ mod tests {
fn test_extract_font_name_basic() {
// Standard pattern: /F1 12 Tf
let content = b"/F1 12 Tf";
let name = extract_font_name_before_tf(content, 6); // 'T' is at index 6
let name = extract_font_name_before_tf(content, 6, 0); // 'T' is at index 6
assert_eq!(name, Some(b"F1".to_vec()));
}
#[test]
fn test_extract_font_name_long_name() {
let content = b"/ArialMT-Bold 9.5 Tf";
let name = extract_font_name_before_tf(content, 18);
let name = extract_font_name_before_tf(content, 18, 0);
assert_eq!(name, Some(b"ArialMT-Bold".to_vec()));
}
+328
View File
@@ -0,0 +1,328 @@
//! Bounded content-stream decoding.
//!
//! `lopdf::content::Content::decode` materializes every operator before any
//! caller can apply a limit. A compact page of `q Q` pairs can therefore
//! allocate hundreds of megabytes and abort. Count operators first (without
//! allocating `Operation` objects) and skip decode when the cap is exceeded.
use crate::PdfError;
use lopdf::content::Content;
/// Maximum content-stream operators decoded for a page or a single Form
/// XObject. Matches the previous post-decode skip threshold.
pub(crate) const MAX_PAGE_OPERATIONS: usize = 1_000_000;
/// Decode `data` unless it contains more than `max_operations` operators.
///
/// Returns `Ok(None)` when the stream exceeds the cap, so callers can skip
/// extraction without first allocating the operation vector.
pub(crate) fn decode_content_bounded(
data: &[u8],
max_operations: usize,
) -> Result<Option<Content>, PdfError> {
if content_exceeds_operation_limit(data, max_operations) {
return Ok(None);
}
Content::decode(data)
.map(Some)
.map_err(|e| PdfError::Parse(e.to_string()))
}
fn content_exceeds_operation_limit(data: &[u8], max_operations: usize) -> bool {
count_content_operators(data, max_operations.saturating_add(1)) > max_operations
}
/// Count operators using the same token rules as lopdf's content parser,
/// stopping at `limit`. Does not allocate `Operation` / `Object` values.
fn count_content_operators(data: &[u8], limit: usize) -> usize {
let mut i = 0;
let mut count = 0;
while i < data.len() && count < limit {
skip_content_space(data, &mut i);
if i >= data.len() {
break;
}
if data[i] == b'%' {
skip_comment(data, &mut i);
continue;
}
match data[i] {
b'(' => i = skip_literal_string(data, i),
b'<' => {
if data.get(i + 1) == Some(&b'<') {
i += 2;
} else {
i = skip_hex_string(data, i);
}
}
b'>' => {
i += 1;
if data.get(i) == Some(&b'>') {
i += 1;
}
}
b'[' | b']' => i += 1,
b'/' => skip_name(data, &mut i),
b'+' | b'-' | b'.' => skip_number(data, &mut i),
b if b.is_ascii_digit() => skip_number(data, &mut i),
b if is_operator_byte(b) => {
let start = i;
i += 1;
while i < data.len() && is_operator_byte(data[i]) {
i += 1;
}
let token = &data[start..i];
if token == b"true" || token == b"false" || token == b"null" {
continue;
}
count += 1;
if token == b"BI" && (i >= data.len() || is_content_space(data[i])) {
i = skip_inline_image_after_bi(data, i);
}
}
_ => i += 1,
}
}
count
}
fn is_content_space(b: u8) -> bool {
// PDF whitespace (ISO 32000): NUL, tab, LF, FF, CR, space. Names must
// stop on these so a following operator is not absorbed into `/Name`.
matches!(b, b'\0' | b'\t' | b'\n' | b'\x0c' | b'\r' | b' ')
}
fn is_operator_byte(b: u8) -> bool {
b.is_ascii_alphabetic() || matches!(b, b'*' | b'\'' | b'"')
}
fn is_delimiter(b: u8) -> bool {
matches!(
b,
b'(' | b')' | b'<' | b'>' | b'[' | b']' | b'{' | b'}' | b'/' | b'%'
)
}
fn skip_content_space(data: &[u8], i: &mut usize) {
while *i < data.len() && is_content_space(data[*i]) {
*i += 1;
}
}
fn skip_comment(data: &[u8], i: &mut usize) {
while *i < data.len() && data[*i] != b'\n' && data[*i] != b'\r' {
*i += 1;
}
}
fn skip_literal_string(data: &[u8], mut i: usize) -> usize {
let mut depth = 1i32;
i += 1;
while i < data.len() && depth > 0 {
match data[i] {
b'\\' => {
i += 1;
if i < data.len() {
i += 1;
}
}
b'(' => {
depth += 1;
i += 1;
}
b')' => {
depth -= 1;
i += 1;
}
_ => i += 1,
}
}
i
}
fn skip_hex_string(data: &[u8], mut i: usize) -> usize {
i += 1;
while i < data.len() && data[i] != b'>' {
i += 1;
}
if i < data.len() {
i += 1;
}
i
}
fn skip_name(data: &[u8], i: &mut usize) {
*i += 1;
while *i < data.len() && !is_content_space(data[*i]) && !is_delimiter(data[*i]) {
*i += 1;
}
}
fn skip_number(data: &[u8], i: &mut usize) {
if *i < data.len() && matches!(data[*i], b'+' | b'-') {
*i += 1;
}
while *i < data.len() && data[*i].is_ascii_digit() {
*i += 1;
}
if *i < data.len() && data[*i] == b'.' {
*i += 1;
while *i < data.len() && data[*i].is_ascii_digit() {
*i += 1;
}
}
}
/// After a `BI` operator, skip inline-image data through `EI`.
/// Uses the same PDF whitespace set as `is_content_space`. If `EI` is not
/// found, leave the cursor in place so later operators are still counted
/// (undercounting would let decode allocate the full vector).
fn skip_inline_image_after_bi(data: &[u8], mut i: usize) -> usize {
skip_content_space(data, &mut i);
let rest = &data[i..];
if let Some(pos) = rest.windows(4).position(|w| {
is_content_space(w[0]) && w[1] == b'E' && w[2] == b'I' && is_content_space(w[3])
}) {
return i + pos + 3;
}
i
}
#[cfg(test)]
mod tests {
use super::*;
fn lopdf_op_count(data: &[u8]) -> usize {
Content::decode(data)
.map(|c| c.operations.len())
.unwrap_or(0)
}
/// DoS safety: never report fewer operators than lopdf would allocate.
/// Overcount is acceptable (skip a page); undercount would re-open decode.
fn assert_count_does_not_undercount(data: &[u8]) {
let ours = count_content_operators(data, usize::MAX);
match Content::decode(data) {
Ok(content) => assert!(
ours >= content.operations.len(),
"undercount: ours={ours} lopdf={} for {:?}",
content.operations.len(),
String::from_utf8_lossy(data)
),
Err(_) => {}
}
}
#[test]
fn operator_count_matches_lopdf_for_typical_streams() {
let samples: &[&[u8]] = &[
b"q 1 0 0 1 0 0 cm BT /F1 12 Tf 72 720 Td (Hello) Tj ET Q",
b"q Q q Q",
b"BT /F1 12 Tf 12 TL 1 0 0 1 100 512 Tm (first) Tj (struck) ' ET",
b"1 0 0 rg 0 0 10 10 re f",
b"true false null q",
b"% comment\nq Q\n",
b"[ (a) 1 (b) ] TJ",
b"1 0 0 1 0 0 cm /Im0 Do",
];
for data in samples {
assert_eq!(
count_content_operators(data, usize::MAX),
lopdf_op_count(data),
"count mismatch for {}",
String::from_utf8_lossy(data)
);
}
}
#[test]
fn strings_and_comments_are_not_operators() {
let data = b"(q Q Tj) Tj % q Q\nET";
assert_eq!(
count_content_operators(data, usize::MAX),
lopdf_op_count(data)
);
assert_eq!(count_content_operators(data, usize::MAX), 2); // Tj, ET
}
#[test]
fn inline_image_counts_as_one_operator() {
let data = b"BI /W 2 /H 2 /CS /RGB /BPC 8 ID \x00\x01\x02\x03 EI q";
assert_eq!(
count_content_operators(data, usize::MAX),
lopdf_op_count(data)
);
assert_eq!(count_content_operators(data, usize::MAX), 2); // BI, q
}
#[test]
fn inline_image_ei_accepts_pdf_whitespace() {
let tab = b"BI /W 1 /H 1 ID \xff\tEI\t q Q";
let nul = b"BI /W 1 /H 1 ID \xff\x00EI\x00 q Q";
let ff = b"BI /W 1 /H 1 ID \xff\x0cEI\x0c q Q";
for data in [tab.as_slice(), nul.as_slice(), ff.as_slice()] {
assert_count_does_not_undercount(data);
assert!(
count_content_operators(data, usize::MAX) >= 3,
"BI plus following q Q must remain visible after EI, got {} for {:?}",
count_content_operators(data, usize::MAX),
String::from_utf8_lossy(data)
);
}
}
#[test]
fn decode_is_skipped_when_operator_cap_is_exceeded() {
let mut data = Vec::new();
for _ in 0..20 {
data.extend_from_slice(b"q Q\n");
}
assert!(decode_content_bounded(&data, 10).unwrap().is_none());
let decoded = decode_content_bounded(&data, 50).unwrap().unwrap();
assert_eq!(decoded.operations.len(), 40);
}
#[test]
fn name_whitespace_does_not_swallow_following_operator() {
// NUL / form-feed end a name (PDF whitespace). Absorbing `q` into
// `/x` would undercount and let decode allocate the operator vector.
let mut nul_sep = Vec::new();
let mut ff_sep = Vec::new();
for _ in 0..8_000 {
nul_sep.extend_from_slice(b"/x\x00q");
ff_sep.extend_from_slice(b"/x\x0cq");
}
assert_count_does_not_undercount(&nul_sep);
assert_count_does_not_undercount(&ff_sep);
assert!(count_content_operators(&ff_sep, usize::MAX) >= 8_000);
}
#[test]
fn edge_streams_do_not_undercount_vs_lopdf() {
let samples: &[&[u8]] = &[
b".5 0 0 .5 0 0 cm",
b"+1 -2 3.0 rg",
b"<0041> Tj",
b"(unbalanced",
b"BI /W 1 /H 1 ID \xff\xff no EI here q Q q Q",
b"q\x00Q\x00q\x00Q",
b"/F1\x0c12 Tf (Hi) Tj",
b"{ 1 2 add } cvx",
];
for data in samples {
assert_count_does_not_undercount(data);
}
}
#[test]
fn million_q_pairs_are_rejected_without_decode() {
let mut data = Vec::with_capacity((MAX_PAGE_OPERATIONS + 1) * 2);
for _ in 0..=MAX_PAGE_OPERATIONS {
data.extend_from_slice(b"q\n");
}
assert!(content_exceeds_operation_limit(&data, MAX_PAGE_OPERATIONS));
assert!(decode_content_bounded(&data, MAX_PAGE_OPERATIONS)
.unwrap()
.is_none());
}
}
+41 -15
View File
@@ -19,7 +19,7 @@ use super::fonts::{
CMapDecisionCache, FontStyleCache,
};
use super::underline::UnderlineLine;
use super::xobjects::{extract_form_xobject_text, get_page_xobjects, XObjectType};
use super::xobjects::{extract_form_xobject_text, get_page_xobjects, FormWalkBudget, XObjectType};
use super::{get_number, image_bbox_from_ctm, multiply_matrices};
/// Strip PDF comments (% to end of line) from content stream bytes.
@@ -149,9 +149,8 @@ pub(crate) fn extract_page_text_items(
font_cmaps: &FontCMaps,
include_invisible: bool,
style_cache: &mut FontStyleCache,
form_budget: &mut FormWalkBudget,
) -> Result<(PageExtraction, bool, bool, bool), PdfError> {
use lopdf::content::Content;
let mut items = Vec::new();
let mut rects: Vec<PdfRect> = Vec::new();
let mut clip_rects: Vec<PdfRect> = Vec::new();
@@ -255,18 +254,20 @@ pub(crate) fn extract_page_text_items(
// Content::decode parser, causing it to skip operators like ET and Q.
let content_data = strip_pdf_comments(&content_data);
let content = Content::decode(&content_data).map_err(|e| PdfError::Parse(e.to_string()))?;
const MAX_OPERATIONS: usize = 1_000_000;
if content.operations.len() > MAX_OPERATIONS {
log::warn!(
"page {}: skipping extraction — {} operations exceeds limit ({})",
page_num,
content.operations.len(),
MAX_OPERATIONS
);
return Ok(((Vec::new(), Vec::new(), Vec::new()), false, false, false));
}
let content = match super::content_decode::decode_content_bounded(
&content_data,
super::content_decode::MAX_PAGE_OPERATIONS,
)? {
Some(content) => content,
None => {
log::warn!(
"page {}: skipping extraction — content stream exceeds {} operations",
page_num,
super::content_decode::MAX_PAGE_OPERATIONS
);
return Ok(((Vec::new(), Vec::new(), Vec::new()), false, false, false));
}
};
// Graphics state tracking
let mut ctm = [1.0f32, 0.0, 0.0, 1.0, 0.0, 0.0]; // Current Transformation Matrix
@@ -916,6 +917,7 @@ pub(crate) fn extract_page_text_items(
&ctm,
&mut cmap_decisions,
style_cache,
form_budget,
);
items.extend(form_items);
}
@@ -1285,6 +1287,12 @@ pub(crate) fn extract_page_text_items(
}
}
if form_budget.was_truncated() {
log::warn!(
"page {page_num}: Form XObject expansion truncated (invocation or operation budget reached); nested form text may be incomplete"
);
}
// Underline detection reads only painted ink: `re` rects confirmed by
// a paint operator plus filled-subpath rects — never clip-only rects,
// which draw nothing.
@@ -1544,6 +1552,7 @@ mod tests {
&font_cmaps,
false,
&mut FontStyleCache::new(),
&mut FormWalkBudget::new(),
)
.unwrap();
items
@@ -1773,6 +1782,7 @@ BT /F1 12 Tf 0 1 -1 0 240 100 Tm (WORLD) Tj ET
&font_cmaps,
false,
&mut FontStyleCache::new(),
&mut FormWalkBudget::new(),
)
.unwrap();
let ((items, rects, lines), _has_gid, _coords_rotated, _skipped_invisible) = result;
@@ -1863,6 +1873,7 @@ BT 30 700 Tm <41> Tj ET";
&font_cmaps,
false,
&mut FontStyleCache::new(),
&mut FormWalkBudget::new(),
)
.unwrap();
let text = items
@@ -1926,4 +1937,19 @@ BT 30 700 Tm <41> Tj ET";
let output = strip_pdf_comments(input);
assert_eq!(output, b"(x\\\\) Tj \nET\n");
}
#[test]
fn oversized_content_stream_skips_extraction() {
let mut content =
Vec::with_capacity((super::super::content_decode::MAX_PAGE_OPERATIONS + 1) * 2);
for _ in 0..=super::super::content_decode::MAX_PAGE_OPERATIONS {
content.extend_from_slice(b"q\n");
}
content.extend_from_slice(b"BT /F1 12 Tf 72 720 Td (Hello) Tj ET\n");
let items = extract_simple_items(&content);
assert!(
items.is_empty(),
"pages over the operator cap must not be decoded"
);
}
}
+107 -16
View File
@@ -482,7 +482,11 @@ pub(crate) fn parse_cid_w_array(
widths: &mut HashMap<u16, u16>,
) {
let mut i = 0;
let mut assigned = 0usize;
while i < w_array.len() {
if assigned >= crate::tounicode::MAX_CID_W_EXPANSION {
return;
}
let start_cid = match &w_array[i] {
Object::Integer(n) => *n as u16,
Object::Real(n) => *n as u16,
@@ -501,12 +505,14 @@ pub(crate) fn parse_cid_w_array(
Object::Array(arr) => {
// [c [w1 w2 ...]] — consecutive widths starting at c
for (j, w_obj) in arr.iter().enumerate() {
let w = match w_obj {
Object::Integer(n) => *n as u16,
Object::Real(n) => *n as u16,
_ => continue,
};
widths.insert(start_cid + j as u16, w);
if !assign_cid_width(
widths,
start_cid.wrapping_add(j as u16),
w_obj,
&mut assigned,
) {
return;
}
}
i += 1;
}
@@ -514,12 +520,14 @@ pub(crate) fn parse_cid_w_array(
// Could be a reference to an array
if let Ok(Object::Array(arr)) = doc.get_object(*r) {
for (j, w_obj) in arr.iter().enumerate() {
let w = match w_obj {
Object::Integer(n) => *n as u16,
Object::Real(n) => *n as u16,
_ => continue,
};
widths.insert(start_cid + j as u16, w);
if !assign_cid_width(
widths,
start_cid.wrapping_add(j as u16),
w_obj,
&mut assigned,
) {
return;
}
}
i += 1;
} else {
@@ -542,8 +550,8 @@ pub(crate) fn parse_cid_w_array(
continue;
}
};
for cid in start_cid..=end {
widths.insert(cid, w);
if !assign_cid_width_range(widths, start_cid, end, w, &mut assigned) {
return;
}
i += 1;
}
@@ -561,8 +569,8 @@ pub(crate) fn parse_cid_w_array(
continue;
}
};
for cid in start_cid..=end {
widths.insert(cid, w);
if !assign_cid_width_range(widths, start_cid, end, w, &mut assigned) {
return;
}
i += 1;
}
@@ -573,6 +581,45 @@ pub(crate) fn parse_cid_w_array(
}
}
fn assign_cid_width(
widths: &mut HashMap<u16, u16>,
cid: u16,
w_obj: &Object,
assigned: &mut usize,
) -> bool {
let w = match w_obj {
Object::Integer(n) => *n as u16,
Object::Real(n) => *n as u16,
_ => return true,
};
if *assigned >= crate::tounicode::MAX_CID_W_EXPANSION {
return false;
}
widths.insert(cid, w);
*assigned += 1;
true
}
fn assign_cid_width_range(
widths: &mut HashMap<u16, u16>,
start: u16,
end: u16,
w: u16,
assigned: &mut usize,
) -> bool {
if start > end {
return true;
}
for cid in start..=end {
if *assigned >= crate::tounicode::MAX_CID_W_EXPANSION {
return false;
}
widths.insert(cid, w);
*assigned += 1;
}
true
}
/// Compute the width of a string in text space units,
/// given raw bytes and font width info.
/// Returns width in text space units (font_units * units_scale * font_size).
@@ -2313,4 +2360,48 @@ end",
// invalid CMap result — so it must not clear the gid flag.
assert!(gid_flagged(Some("<01> <FFFD>\n<02> <FFFD>")));
}
#[test]
fn parse_cid_w_array_range_and_consecutive() {
use super::parse_cid_w_array;
use lopdf::{Document, Object};
use std::collections::HashMap;
let doc = Document::new();
let mut widths = HashMap::new();
let w = vec![
Object::Integer(10),
Object::Integer(12),
Object::Integer(500),
Object::Integer(20),
Object::Array(vec![Object::Integer(100), Object::Integer(200)]),
];
parse_cid_w_array(&doc, &w, &mut widths);
assert_eq!(widths.get(&10), Some(&500));
assert_eq!(widths.get(&11), Some(&500));
assert_eq!(widths.get(&12), Some(&500));
assert_eq!(widths.get(&20), Some(&100));
assert_eq!(widths.get(&21), Some(&200));
}
#[test]
fn parse_cid_w_array_repeated_full_ranges_stay_bounded() {
use super::parse_cid_w_array;
use crate::tounicode::MAX_CID_W_EXPANSION;
use lopdf::{Document, Object};
use std::collections::HashMap;
let doc = Document::new();
let mut widths = HashMap::new();
let mut w = Vec::new();
for _ in 0..5_000 {
w.push(Object::Integer(0));
w.push(Object::Integer(65535));
w.push(Object::Integer(500));
}
parse_cid_w_array(&doc, &w, &mut widths);
assert!(widths.len() <= MAX_CID_W_EXPANSION);
assert_eq!(widths.get(&0), Some(&500));
assert_eq!(widths.get(&65535), Some(&500));
}
}
+239 -3
View File
@@ -3,6 +3,7 @@
//! This module extracts text with position information for structure detection.
mod base14;
mod content_decode;
pub(crate) mod content_stream;
mod fonts;
mod layout;
@@ -37,6 +38,7 @@ pub(crate) use layout::group_prefiltered_items_into_lines_with_thresholds_and_re
pub(crate) use layout::is_newspaper_layout;
pub(crate) use layout::ColumnRegion;
pub use layout::{group_into_lines, group_into_lines_preserving_all_text};
pub(crate) use xobjects::FormWalkBudget;
// ---------------------------------------------------------------------------
// Public API
@@ -282,6 +284,7 @@ fn extract_positioned_text_impl(
font_cmaps,
include_invisible,
&mut style_cache,
&mut FormWalkBudget::new(),
);
let ((mut items, mut rects, mut lines), has_gid_fonts, coords_rotated, _skipped_invisible) =
match page_result {
@@ -887,6 +890,92 @@ fn tracked_run_space_floor(group: &[&TextItem], start: usize) -> Option<(usize,
Some((end, floor * fs))
}
/// Fractional font-size band within which `merge_text_items` treats two runs as
/// the same size. Shared with `is_small_caps_continuation`, which exists only to
/// rescue junctions this band would otherwise break.
const MERGE_FONT_SIZE_BAND: f32 = 0.20;
/// Detect a small-caps continuation: typesetters render small caps as a
/// full-size capital immediately followed by shrunken capitals in the same
/// font (`(R) Tj` at 9.98pt, then `(OLANDO) Tj` at 6.74pt). Those runs are one
/// word, but the font-size band in `merge_text_items` would split them,
/// leaving "R" and "OLANDO" as separate items — which then read as separate
/// table columns, since column boundaries cluster on item start positions.
///
/// Gated tightly so it cannot absorb the other reasons a smaller run follows a
/// larger one:
/// - runs the size band already accepts — excluded by requiring the junction
/// to *cross* the band, so within-band pairs keep the normal word-spacing
/// logic instead of having their space suppressed
/// - superscripts / footnote markers — excluded by requiring an uppercase
/// *letter* on both sides, so digits never qualify
/// - drop caps — excluded because the body text that follows is mixed case
/// - adjacent table cells or separate words — excluded by requiring the runs
/// to be visually contiguous (essentially no gap)
fn is_small_caps_continuation(
text_so_far: &str,
first: &TextItem,
next: &TextItem,
gap: f32,
) -> bool {
// Must shrink. Real small caps sit near 0.7-0.8 of the full cap height;
// anything smaller is a superscript or a different run entirely.
if first.font_size <= 0.0 || next.font_size >= first.font_size {
return false;
}
// Only rescue junctions the size band would have broken. Within-band pairs
// merge on their own, and suppressing their space would swallow real word
// gaps between two similarly-sized uppercase words.
if (next.font_size - first.font_size).abs() <= first.font_size * MERGE_FONT_SIZE_BAND {
return false;
}
if next.font_size / first.font_size < 0.55 {
return false;
}
// Visually contiguous: the capital and its small caps touch. A real word
// space or a column gap disqualifies.
if !(-first.font_size * 0.2..=first.font_size * 0.15).contains(&gap) {
return false;
}
// The continuation must be all-uppercase letters (digits and lowercase
// both disqualify), and must contain at least one letter.
let mut saw_letter = false;
for ch in next.text.chars() {
if ch.is_alphabetic() {
saw_letter = true;
if !ch.is_uppercase() {
return false;
}
} else if ch.is_numeric() {
return false;
}
}
if !saw_letter {
return false;
}
// What we are continuing must itself end in a capital. Check the actual
// trailing character rather than skipping back to the nearest letter: after
// "ANGELA M. MAZZARELLI1" the run to continue is the footnote marker, not
// the "I" before it.
let trimmed = text_so_far.trim_end();
if trimmed.chars().last().is_some_and(|c| c.is_numeric()) {
// One legitimate exception: an ordinal suffix set as a smaller run,
// e.g. "JULY 4" + "TH". Only the four English suffixes qualify —
// anything else after a digit is a footnote marker or numeric suffix.
return matches!(trimmed_suffix(next), "TH" | "ST" | "ND" | "RD");
}
trimmed
.chars()
.rev()
.find(|c| c.is_alphabetic())
.is_some_and(|c| c.is_uppercase())
}
/// The continuation run's text, trimmed — used to spot ordinal suffixes.
fn trimmed_suffix(next: &TextItem) -> &str {
next.text.trim()
}
pub(crate) fn merge_text_items(items: Vec<TextItem>) -> Vec<TextItem> {
if items.is_empty() {
return items;
@@ -945,8 +1034,16 @@ pub(crate) fn merge_text_items(items: Vec<TextItem>) -> Vec<TextItem> {
let mut j = i + 1;
while j < group.len() {
let next = group[j];
// Must be similar font size (within 20%)
if (next.font_size - first.font_size).abs() > first.font_size * 0.20 {
// A small-caps junction is mid-word: it both survives the
// font-size band below and must never take a space.
let small_caps_join =
is_small_caps_continuation(&text, first, next, next.x - end_x);
// Must be similar font size, except for genuine small-caps
// runs, where the shrunken capitals are the same word as the
// full-size initial (see helper).
if (next.font_size - first.font_size).abs() > first.font_size * MERGE_FONT_SIZE_BAND
&& !small_caps_join
{
break;
}
// Never merge across style boundaries: the merged item
@@ -1000,7 +1097,7 @@ pub(crate) fn merge_text_items(items: Vec<TextItem>) -> Vec<TextItem> {
Some((run_end, floor)) if j <= run_end => floor,
_ => threshold,
};
if needs_bullet_space || gap > effective_threshold {
if !small_caps_join && (needs_bullet_space || gap > effective_threshold) {
text.push(' ');
}
text.push_str(&next.text);
@@ -3004,6 +3101,145 @@ mod tests {
}
}
/// Small caps as typesetters emit them: a full-size capital at 9.98pt
/// immediately followed by shrunken capitals at 6.74pt, touching.
/// Modelled on `199AD3d.pdf` p.5 ("ROLANDO T. ACOSTA, P.J.").
#[test]
fn small_caps_run_merges_into_one_word() {
let items = vec![
make_item_fs("R", 144.36, 581.84, 7.20, 9.98),
make_item_fs("OLANDO", 151.56, 581.84, 30.56, 6.74),
make_item_fs("T. A", 185.45, 581.84, 17.58, 9.98),
make_item_fs("COSTA", 203.94, 581.84, 23.15, 6.74),
make_item_fs(", P.J.", 227.09, 581.84, 22.56, 9.98),
];
let merged = merge_text_items(items);
assert_eq!(merged.len(), 1, "got {:?}", merged);
assert_eq!(merged[0].text, "ROLANDO T. ACOSTA, P.J.");
}
/// The full two-column row: both names must merge independently and the
/// 72pt column gap between them must survive as an item boundary.
#[test]
fn small_caps_merge_does_not_swallow_a_second_column() {
let items = vec![
// Column 1: "ROLANDO T. ACOSTA, P.J." ending at x=249.65
make_item_fs("R", 144.36, 581.84, 7.20, 9.98),
make_item_fs("OLANDO", 151.56, 581.84, 30.56, 6.74),
make_item_fs("T. A", 185.45, 581.84, 17.58, 9.98),
make_item_fs("COSTA", 203.94, 581.84, 23.15, 6.74),
make_item_fs(", P.J.", 227.09, 581.84, 22.56, 9.98),
// Column 2 starts at x=321.96 — a 72pt gap.
make_item_fs("A", 321.96, 581.84, 7.20, 9.98),
make_item_fs("NIL", 329.17, 581.84, 12.72, 6.74),
make_item_fs("C. S", 345.04, 581.84, 19.59, 9.98),
make_item_fs("INGH", 364.62, 581.84, 19.08, 6.74),
];
let merged = merge_text_items(items);
let texts: Vec<&str> = merged.iter().map(|i| i.text.as_str()).collect();
assert_eq!(
texts,
vec!["ROLANDO T. ACOSTA, P.J.", "ANIL C. SINGH"],
"column gap should keep the two names apart"
);
}
#[test]
fn small_caps_merge_keeps_word_space_between_same_size_capitals() {
// Two uppercase words at sizes the merge band already accepts (9.98 and
// 9.0, a 10% drop) separated by a real word gap. The small-caps path
// must not claim this junction and swallow the space.
let items = vec![
make_item_fs("SEE", 100.0, 500.0, 18.0, 9.98),
make_item_fs("ALSO", 119.2, 500.0, 24.0, 9.0),
];
let merged = merge_text_items(items);
assert_eq!(merged.len(), 1, "got {:?}", merged);
assert_eq!(merged[0].text, "SEE ALSO");
}
#[test]
fn trailing_digit_is_not_a_capital_awaiting_small_caps() {
// "...MAZZARELLI1" ends in a footnote marker; the backward search for an
// uppercase letter must not skip the digit and glue the next run.
assert!(!is_small_caps_continuation(
"ANGELA M. MAZZARELLI1",
&make_item_fs("ANGELA", 100.0, 500.0, 40.0, 9.98),
&make_item_fs("SHULMAN", 140.0, 500.0, 30.0, 6.74),
0.0,
));
}
#[test]
fn ordinal_suffix_after_a_digit_still_merges() {
// "TUESDAY, JULY 4" + "TH" is one word in the source; the digit guard
// must not block the four English ordinal suffixes.
for suffix in ["TH", "ST", "ND", "RD"] {
assert!(
is_small_caps_continuation(
"TUESDAY, JULY 4",
&make_item_fs("JULY", 100.0, 500.0, 30.0, 12.0),
&make_item_fs(suffix, 130.0, 500.0, 8.0, 8.0),
0.0,
),
"{suffix} should merge after a digit"
);
}
}
#[test]
fn superscript_footnote_marker_is_not_a_small_caps_continuation() {
// A digit must never qualify — otherwise footnote markers get glued on
// without the superscript handling.
assert!(!is_small_caps_continuation(
"MAZZARELLI",
&make_item_fs("MAZZARELLI", 100.0, 500.0, 50.0, 9.98),
&make_item_fs("1", 150.0, 503.0, 3.0, 6.74),
0.0,
));
}
#[test]
fn drop_cap_is_not_a_small_caps_continuation() {
// Mixed-case body text after a large initial is a drop cap, not small
// caps.
assert!(!is_small_caps_continuation(
"T",
&make_item_fs("T", 100.0, 500.0, 20.0, 30.0),
&make_item_fs("he court held", 120.0, 500.0, 60.0, 10.0),
0.0,
));
}
#[test]
fn separate_word_is_not_a_small_caps_continuation() {
// A real word space disqualifies even when both runs are uppercase.
let first = make_item_fs("SEE", 100.0, 500.0, 20.0, 9.98);
let next = make_item_fs("ALSO", 128.0, 500.0, 25.0, 6.74);
assert!(!is_small_caps_continuation("SEE", &first, &next, 8.0));
}
#[test]
fn lowercase_continuation_is_not_small_caps() {
assert!(!is_small_caps_continuation(
"SMALL",
&make_item_fs("SMALL", 100.0, 500.0, 30.0, 9.98),
&make_item_fs("caps", 130.0, 500.0, 20.0, 6.74),
0.0,
));
}
#[test]
fn too_small_a_ratio_is_not_small_caps() {
// 0.4 ratio is a superscript/sub-run, outside the small-caps band.
assert!(!is_small_caps_continuation(
"A",
&make_item_fs("A", 100.0, 500.0, 7.0, 10.0),
&make_item_fs("BC", 107.0, 500.0, 8.0, 4.0),
0.0,
));
}
#[test]
fn test_merge_subscript_items_chemical_formula() {
// NH₃: "NH" at fs=8 followed by subscript "3" at fs=4.7
+600 -21
View File
@@ -16,6 +16,76 @@ use super::{get_number, image_bbox_from_ctm, multiply_matrices};
const MAX_FORM_XOBJECT_DEPTH: u8 = 5;
/// Upper bound on Form XObject invocations during a single page extraction.
/// Depth alone is not enough: an acyclic DAG where each form invokes the next
/// N times expands to N^depth work before the depth cap is reached.
const MAX_FORM_XOBJECT_INVOCATIONS: usize = 10_000;
/// Upper bound on content-stream operations walked across all Form XObject
/// expansions for a page. Nested forms are decoded independently of the
/// page-level operation cap, so this keeps total form work in the same
/// ballpark as that page cap.
const MAX_FORM_XOBJECT_OPERATIONS: usize = 1_000_000;
/// Shared budget for Form XObject expansion on a page. Bounds both nested DAG
/// expansion and repeated sibling `/Do` invocations of the same form.
pub(crate) struct FormWalkBudget {
invocations: usize,
operations: usize,
max_invocations: usize,
max_operations: usize,
truncated: bool,
}
impl FormWalkBudget {
pub(crate) fn new() -> Self {
Self::with_limits(MAX_FORM_XOBJECT_INVOCATIONS, MAX_FORM_XOBJECT_OPERATIONS)
}
fn with_limits(max_invocations: usize, max_operations: usize) -> Self {
Self {
invocations: 0,
operations: 0,
max_invocations,
max_operations,
truncated: false,
}
}
fn exhausted(&mut self) -> bool {
if self.invocations >= self.max_invocations || self.operations >= self.max_operations {
self.truncated = true;
true
} else {
false
}
}
fn charge_invocation(&mut self) -> bool {
if self.exhausted() {
return false;
}
self.invocations += 1;
true
}
/// Charge one walked content-stream operator. Independent of the
/// invocation cap so a form that was already admitted can finish its
/// stream (up to the operation cap).
fn charge_operation(&mut self) -> bool {
if self.operations >= self.max_operations {
self.truncated = true;
return false;
}
self.operations += 1;
true
}
pub(crate) fn was_truncated(&self) -> bool {
self.truncated
}
}
pub(crate) enum XObjectType {
Image,
Form(ObjectId),
@@ -109,6 +179,7 @@ fn collect_xobjects_from_dict(
}
/// Extract text items from a Form XObject
#[allow(clippy::too_many_arguments)]
pub(crate) fn extract_form_xobject_text(
doc: &Document,
form_id: ObjectId,
@@ -117,6 +188,7 @@ pub(crate) fn extract_form_xobject_text(
parent_ctm: &[f32; 6],
cmap_decisions: &mut CMapDecisionCache,
style_cache: &mut FontStyleCache,
budget: &mut FormWalkBudget,
) -> Vec<TextItem> {
extract_form_xobject_text_inner(
doc,
@@ -127,6 +199,7 @@ pub(crate) fn extract_form_xobject_text(
cmap_decisions,
style_cache,
0,
budget,
)
}
@@ -140,11 +213,14 @@ fn extract_form_xobject_text_inner(
cmap_decisions: &mut CMapDecisionCache,
style_cache: &mut FontStyleCache,
depth: u8,
budget: &mut FormWalkBudget,
) -> Vec<TextItem> {
use lopdf::content::Content;
let mut items = Vec::new();
if !budget.charge_invocation() {
return items;
}
// Get the Form XObject stream
let Ok(Object::Stream(stream)) = doc.get_object(form_id) else {
return items;
@@ -156,8 +232,12 @@ fn extract_form_xobject_text_inner(
Err(_) => stream.content.clone(),
};
// Decode the content stream
let Ok(content) = Content::decode(&content_data) else {
// Decode the content stream. Cap before lopdf materializes the operator
// vector — the walk budget cannot help if decode itself allocates first.
let Ok(Some(content)) = super::content_decode::decode_content_bounded(
&content_data,
super::content_decode::MAX_PAGE_OPERATIONS,
) else {
return items;
};
@@ -246,19 +326,55 @@ fn extract_form_xobject_text_inner(
let mut current_font = String::new();
let mut current_font_size: f32 = 12.0;
let mut text_matrix = [1.0f32, 0.0, 0.0, 1.0, 0.0, 0.0];
// Text line matrix (TLM) — Td/TD/T* move relative to the start of the
// current line, not to the position left by the last show operator.
let mut line_matrix = [1.0f32, 0.0, 0.0, 1.0, 0.0, 0.0];
let mut text_leading: f32 = 0.0; // TL parameter (text-space units)
let mut char_spacing: f32 = 0.0; // Tc parameter
let mut word_spacing: f32 = 0.0; // Tw parameter
let mut in_text_block = false;
let mut fill_is_white = false;
let mut ctm = base_ctm;
let mut ctm_stack: Vec<[f32; 6]> = Vec::new();
// Text state (Tc/Tw/TL/Tf) and the fill colour are part of the graphics
// state and must be saved/restored by q/Q alongside the CTM.
#[derive(Clone)]
struct GraphicsState {
ctm: [f32; 6],
char_spacing: f32,
word_spacing: f32,
text_leading: f32,
current_font: String,
current_font_size: f32,
fill_is_white: bool,
}
let mut ctm_stack: Vec<GraphicsState> = Vec::new();
for op in &content.operations {
if !budget.charge_operation() {
break;
}
match op.operator.as_str() {
"q" => {
ctm_stack.push(ctm);
ctm_stack.push(GraphicsState {
ctm,
char_spacing,
word_spacing,
text_leading,
current_font: current_font.clone(),
current_font_size,
fill_is_white,
});
}
"Q" => {
if let Some(saved) = ctm_stack.pop() {
ctm = saved;
ctm = saved.ctm;
char_spacing = saved.char_spacing;
word_spacing = saved.word_spacing;
text_leading = saved.text_leading;
current_font = saved.current_font;
current_font_size = saved.current_font_size;
fill_is_white = saved.fill_is_white;
}
}
"cm" => {
@@ -276,7 +392,7 @@ fn extract_form_xobject_text_inner(
let xobj_name = String::from_utf8_lossy(name).to_string();
match form_xobjects.get(&xobj_name) {
Some(XObjectType::Form(nested_id)) => {
if depth < MAX_FORM_XOBJECT_DEPTH {
if depth < MAX_FORM_XOBJECT_DEPTH && !budget.exhausted() {
let nested_items = extract_form_xobject_text_inner(
doc,
*nested_id,
@@ -286,6 +402,7 @@ fn extract_form_xobject_text_inner(
cmap_decisions,
style_cache,
depth + 1,
budget,
);
items.extend(nested_items);
}
@@ -321,6 +438,7 @@ fn extract_form_xobject_text_inner(
"BT" => {
in_text_block = true;
text_matrix = [1.0, 0.0, 0.0, 1.0, 0.0, 0.0];
line_matrix = text_matrix;
}
"ET" => {
in_text_block = false;
@@ -333,12 +451,33 @@ fn extract_form_xobject_text_inner(
current_font_size = get_number(&op.operands[1]).unwrap_or(12.0);
}
}
"TL" => {
// Set text leading (used by T*, ', and ")
if let Some(tl) = op.operands.first().and_then(get_number) {
text_leading = tl;
}
}
"Tc" => {
if let Some(tc) = op.operands.first().and_then(get_number) {
char_spacing = tc;
}
}
"Tw" => {
if let Some(tw) = op.operands.first().and_then(get_number) {
word_spacing = tw;
}
}
"Td" | "TD" => {
// Move text position: TLM = T(tx,ty) x TLM; Tm = TLM
if op.operands.len() >= 2 {
let tx = get_number(&op.operands[0]).unwrap_or(0.0);
let ty = get_number(&op.operands[1]).unwrap_or(0.0);
text_matrix[4] += tx * text_matrix[0] + ty * text_matrix[2];
text_matrix[5] += tx * text_matrix[1] + ty * text_matrix[3];
line_matrix[4] += tx * line_matrix[0] + ty * line_matrix[2];
line_matrix[5] += tx * line_matrix[1] + ty * line_matrix[3];
text_matrix = line_matrix;
if op.operator == "TD" {
text_leading = -ty;
}
}
}
"Tm" => {
@@ -347,8 +486,20 @@ fn extract_form_xobject_text_inner(
text_matrix[i] =
get_number(operand).unwrap_or(if i == 0 || i == 3 { 1.0 } else { 0.0 });
}
line_matrix = text_matrix;
}
}
"T*" => {
// Move to start of next line: equivalent to `0 -TL Td`
let tl = if text_leading != 0.0 {
text_leading
} else {
current_font_size * 1.2
};
line_matrix[4] += (-tl) * line_matrix[2];
line_matrix[5] += (-tl) * line_matrix[3];
text_matrix = line_matrix;
}
"g" => {
if let Some(gray) = op.operands.first().and_then(get_number) {
fill_is_white = gray > 0.95;
@@ -384,17 +535,33 @@ fn extract_form_xobject_text_inner(
_ => fill_is_white = false,
}
}
"Tj" => {
if in_text_block && !op.operands.is_empty() {
"Tj" | "'" | "\"" => {
// `'` = move to next line then show; `"` = set word/char spacing,
// move to next line, then show (string is the last operand).
if op.operator != "Tj" {
if op.operator == "\"" && op.operands.len() >= 3 {
word_spacing = get_number(&op.operands[0]).unwrap_or(word_spacing);
char_spacing = get_number(&op.operands[1]).unwrap_or(char_spacing);
}
let tl = if text_leading != 0.0 {
text_leading
} else {
current_font_size * 1.2
};
line_matrix[4] += (-tl) * line_matrix[2];
line_matrix[5] += (-tl) * line_matrix[3];
text_matrix = line_matrix;
}
if let (true, Some(show_operand)) = (in_text_block, op.operands.last()) {
if fill_is_white {
if let Some(font_info) = font_widths.get(&current_font) {
if let Some(raw_bytes) = get_operand_bytes(&op.operands[0]) {
if let Some(raw_bytes) = get_operand_bytes(show_operand) {
let w_ts = compute_string_width_ts(
raw_bytes,
font_info,
current_font_size,
0.0,
0.0,
char_spacing,
word_spacing,
);
text_matrix[4] += w_ts * text_matrix[0];
text_matrix[5] += w_ts * text_matrix[1];
@@ -403,7 +570,7 @@ fn extract_form_xobject_text_inner(
continue;
}
if let Some(text) = extract_text_from_operand(
&op.operands[0],
show_operand,
&current_font,
font_base_names.get(&current_font).map(|s| s.as_str()),
font_cmaps,
@@ -419,13 +586,13 @@ fn extract_form_xobject_text_inner(
* type3_scales.get(&current_font).copied().unwrap_or(1.0);
let (x, y) = (combined[4], combined[5]);
let width = if let Some(font_info) = font_widths.get(&current_font) {
if let Some(raw_bytes) = get_operand_bytes(&op.operands[0]) {
if let Some(raw_bytes) = get_operand_bytes(show_operand) {
let w_ts = compute_string_width_ts(
raw_bytes,
font_info,
current_font_size,
0.0,
0.0,
char_spacing,
word_spacing,
);
text_matrix[4] += w_ts * text_matrix[0];
text_matrix[5] += w_ts * text_matrix[1];
@@ -548,8 +715,8 @@ fn extract_form_xobject_text_inner(
raw_bytes,
fi,
current_font_size,
0.0,
0.0,
char_spacing,
word_spacing,
);
}
}
@@ -683,3 +850,415 @@ pub(crate) fn get_form_fonts<'a>(
fonts
}
#[cfg(test)]
mod tests {
use super::*;
use crate::extractor::content_stream::extract_page_text_items;
use lopdf::{dictionary, Dictionary, Stream};
/// Build an acyclic Form XObject DAG: `levels` form objects, each non-leaf
/// invoking the next form `branches` times. The leaf draws a single `(X)`.
/// Returns `(doc, root_form_id)`.
fn form_dag(branches: usize, levels: usize) -> (Document, ObjectId) {
assert!(levels >= 2);
let mut doc = Document::new();
let font_id = doc.add_object(dictionary! {
"Type" => "Font",
"Subtype" => "Type1",
"BaseFont" => "Helvetica",
});
let ids: Vec<ObjectId> = (0..levels).map(|_| doc.new_object_id()).collect();
for level in 0..levels {
let stream = if level + 1 == levels {
Stream::new(
dictionary! {
"Type" => "XObject",
"Subtype" => "Form",
"BBox" => vec![0.into(), 0.into(), 100.into(), 100.into()],
"Resources" => dictionary! {
"Font" => dictionary! {
"F1" => Object::Reference(font_id),
},
},
},
b"BT /F1 10 Tf 10 10 Td (X) Tj ET\n".to_vec(),
)
} else {
let next_name = format!("Fm{}", level + 1);
let content = format!("/{next_name} Do\n").repeat(branches);
let mut xobjects = Dictionary::new();
xobjects.set(next_name, Object::Reference(ids[level + 1]));
let mut resources = Dictionary::new();
resources.set("XObject", Object::Dictionary(xobjects));
let mut dict = dictionary! {
"Type" => "XObject",
"Subtype" => "Form",
"BBox" => vec![0.into(), 0.into(), 100.into(), 100.into()],
};
dict.set("Resources", Object::Dictionary(resources));
Stream::new(dict, content.into_bytes())
};
doc.set_object(ids[level], Object::Stream(stream));
}
(doc, ids[0])
}
fn page_invoking_form(mut doc: Document, form_id: ObjectId) -> (Document, ObjectId) {
let content_id = doc.add_object(Object::Stream(Stream::new(
dictionary! {},
b"/Fm0 Do\n".to_vec(),
)));
let page_id = doc.add_object(dictionary! {
"Type" => "Page",
"Contents" => Object::Reference(content_id),
"Resources" => dictionary! {
"XObject" => dictionary! {
"Fm0" => Object::Reference(form_id),
},
},
"MediaBox" => vec![0.into(), 0.into(), 612.into(), 792.into()],
});
let pages_id = doc.add_object(dictionary! {
"Type" => "Pages",
"Count" => Object::Integer(1),
"Kids" => vec![Object::Reference(page_id)],
});
let catalog_id = doc.add_object(dictionary! {
"Type" => "Catalog",
"Pages" => Object::Reference(pages_id),
});
doc.trailer.set("Root", Object::Reference(catalog_id));
(doc, page_id)
}
fn extract_form(
doc: &Document,
form_id: ObjectId,
budget: &mut FormWalkBudget,
) -> Vec<TextItem> {
extract_form_xobject_text(
doc,
form_id,
1,
&FontCMaps::from_doc(doc),
&[1.0, 0.0, 0.0, 1.0, 0.0, 0.0],
&mut CMapDecisionCache::new(),
&mut FontStyleCache::new(),
budget,
)
}
#[test]
fn nested_form_still_extracts_leaf_text() {
let (doc, root) = form_dag(1, 3);
let items = extract_form(&doc, root, &mut FormWalkBudget::new());
assert_eq!(items.len(), 1);
assert_eq!(items[0].text, "X");
}
#[test]
fn acyclic_form_dag_within_budget_keeps_all_leaves() {
// 4 sibling invocations across 4 nested levels → 4^4 leaf drawings.
// Default budgets are far above 256, so legitimate nesting is intact.
let (doc, root) = form_dag(4, 5);
let items = extract_form(&doc, root, &mut FormWalkBudget::new());
assert_eq!(items.len(), 4usize.pow(4));
assert!(items.iter().all(|item| item.text == "X"));
}
#[test]
fn acyclic_form_dag_stops_at_invocation_budget() {
// Same DAG as above would draw 256 leaves; a tiny invocation cap must
// stop expansion rather than walking the full tree.
let (doc, root) = form_dag(4, 5);
let mut budget = FormWalkBudget::with_limits(20, MAX_FORM_XOBJECT_OPERATIONS);
let items = extract_form(&doc, root, &mut budget);
assert!(
items.len() < 4usize.pow(4),
"invocation budget must truncate DAG expansion; got {} items",
items.len()
);
assert!(budget.was_truncated());
}
#[test]
fn form_operations_stop_at_budget() {
let mut doc = Document::new();
let font_id = doc.add_object(dictionary! {
"Type" => "Font",
"Subtype" => "Type1",
"BaseFont" => "Helvetica",
});
let mut content = b"q Q\n".repeat(50);
content.extend_from_slice(b"BT /F1 10 Tf 10 10 Td (X) Tj ET\n");
let form_id = doc.add_object(Object::Stream(Stream::new(
dictionary! {
"Type" => "XObject",
"Subtype" => "Form",
"BBox" => vec![0.into(), 0.into(), 100.into(), 100.into()],
"Resources" => dictionary! {
"Font" => dictionary! {
"F1" => Object::Reference(font_id),
},
},
},
content,
)));
let mut budget = FormWalkBudget::with_limits(MAX_FORM_XOBJECT_INVOCATIONS, 10);
let items = extract_form(&doc, form_id, &mut budget);
assert!(
items.is_empty(),
"operation budget must stop before the trailing text show"
);
assert!(budget.was_truncated());
}
#[test]
fn page_level_form_dag_stays_within_production_budget() {
// A page-level `/Do` of an 8-wide, 6-level Form DAG would expand to
// 8^5 = 32_768 leaf drawings without a budget. The production
// invocation cap must keep extraction bounded.
let (doc, root) = form_dag(8, 6);
let (doc, page_id) = page_invoking_form(doc, root);
let font_cmaps = FontCMaps::from_doc(&doc);
let ((items, _, _), _, _, _) = extract_page_text_items(
&doc,
page_id,
1,
&font_cmaps,
false,
&mut FontStyleCache::new(),
&mut FormWalkBudget::new(),
)
.unwrap();
assert!(
items.len() <= MAX_FORM_XOBJECT_INVOCATIONS,
"page-level Form expansion must stay within the invocation cap; got {}",
items.len()
);
assert!(
!items.is_empty(),
"budget must still allow some nested form text through"
);
}
#[test]
fn shared_form_budget_spans_two_extraction_passes() {
// The invisible-layer retry calls extract_page_text_items twice for
// the same page; both passes must share one budget.
let (doc, root) = form_dag(1, 2);
let (doc, page_id) = page_invoking_form(doc, root);
let font_cmaps = FontCMaps::from_doc(&doc);
// Root + leaf = 2 invocations on the first pass.
let mut budget = FormWalkBudget::with_limits(2, MAX_FORM_XOBJECT_OPERATIONS);
let ((first, _, _), _, _, _) = extract_page_text_items(
&doc,
page_id,
1,
&font_cmaps,
false,
&mut FontStyleCache::new(),
&mut budget,
)
.unwrap();
assert_eq!(first.iter().filter(|item| item.text == "X").count(), 1);
assert!(!budget.was_truncated());
let ((second, _, _), _, _, _) = extract_page_text_items(
&doc,
page_id,
1,
&font_cmaps,
true,
&mut FontStyleCache::new(),
&mut budget,
)
.unwrap();
assert!(
second.iter().all(|item| item.text != "X"),
"second pass must not get a fresh invocation budget"
);
assert!(budget.was_truncated());
}
/// Build a document whose page draws *all* of its content through a single
/// Form XObject — the shape emitted by print-to-PDF producers like PDFlib,
/// where the page stream itself is only `q /X1 Do Q`.
fn doc_with_form_content(form_content: &[u8]) -> (Document, ObjectId) {
let mut doc = Document::new();
let widths: Vec<Object> = (0..=255).map(|_| 600.into()).collect();
let font_id = doc.add_object(dictionary! {
"Type" => "Font",
"Subtype" => "Type1",
"BaseFont" => "Helvetica",
"FirstChar" => 0,
"LastChar" => 255,
"Widths" => Object::Array(widths),
});
let form_id = doc.add_object(Object::Stream(Stream::new(
dictionary! {
"Type" => "XObject",
"Subtype" => "Form",
"BBox" => vec![0.into(), 0.into(), 612.into(), 792.into()],
"Resources" => dictionary! {
"Font" => dictionary! { "F1" => Object::Reference(font_id) },
},
},
form_content.to_vec(),
)));
let content_id = doc.add_object(Object::Stream(Stream::new(
dictionary! {},
b"q /X1 Do Q".to_vec(),
)));
let page_id = doc.add_object(dictionary! {
"Type" => "Page",
"Contents" => Object::Reference(content_id),
"Resources" => dictionary! {
"XObject" => dictionary! { "X1" => Object::Reference(form_id) },
},
"MediaBox" => vec![0.into(), 0.into(), 612.into(), 792.into()],
});
let pages_id = doc.add_object(dictionary! {
"Type" => "Pages",
"Count" => Object::Integer(1),
"Kids" => vec![Object::Reference(page_id)],
});
let catalog_id = doc.add_object(dictionary! {
"Type" => "Catalog",
"Pages" => Object::Reference(pages_id),
});
doc.trailer.set("Root", Object::Reference(catalog_id));
(doc, page_id)
}
fn form_items(form_content: &[u8]) -> Vec<TextItem> {
let (doc, page_id) = doc_with_form_content(form_content);
let font_cmaps = FontCMaps::from_doc(&doc);
let ((items, _, _), _, _, _) = extract_page_text_items(
&doc,
page_id,
1,
&font_cmaps,
false,
&mut FontStyleCache::new(),
&mut FormWalkBudget::new(),
)
.unwrap();
items
}
fn find<'a>(items: &'a [TextItem], text: &str) -> &'a TextItem {
items
.iter()
.find(|item| item.text == text)
.unwrap_or_else(|| {
let found: Vec<&String> = items.iter().map(|i| &i.text).collect();
panic!("no item {text:?} in {found:?}")
})
}
#[test]
fn t_star_inside_form_moves_to_next_line() {
// T* was previously unhandled inside Form XObjects, so every line after
// the first piled onto the preceding baseline and drifted right.
let items =
form_items(b"BT /F1 12 Tf 12 TL 1 0 0 1 100 700 Tm (first) Tj T* (second) Tj ET");
let first = find(&items, "first");
let second = find(&items, "second");
assert!((first.y - 700.0).abs() < 0.1, "first y = {}", first.y);
assert!((second.y - 688.0).abs() < 0.1, "second y = {}", second.y);
assert!((second.x - 100.0).abs() < 0.1, "second x = {}", second.x);
}
#[test]
fn td_inside_form_is_relative_to_line_start_not_shown_text() {
// Td moves relative to the text *line* matrix. Applying it to the
// matrix already advanced by Tj marched each line off the right edge.
let items = form_items(b"BT /F1 12 Tf 1 0 0 1 100 700 Tm (AAAAA) Tj 0 -12 Td (B) Tj ET");
let b = find(&items, "B");
assert!((b.x - 100.0).abs() < 0.1, "B x = {} (expected 100)", b.x);
assert!((b.y - 688.0).abs() < 0.1, "B y = {}", b.y);
}
#[test]
fn td_inside_form_sets_leading_for_later_t_star() {
// `TD` sets the leading to -ty as a side effect; a following T* must
// reuse it.
let items = form_items(
b"BT /F1 12 Tf 1 0 0 1 100 700 Tm (one) Tj 0 -15 TD (two) Tj T* (three) Tj ET",
);
assert!((find(&items, "two").y - 685.0).abs() < 0.1);
let three = find(&items, "three");
assert!((three.y - 670.0).abs() < 0.1, "three y = {}", three.y);
assert!((three.x - 100.0).abs() < 0.1, "three x = {}", three.x);
}
#[test]
fn quote_operator_inside_form_moves_to_next_line() {
let items = form_items(b"BT /F1 12 Tf 12 TL 1 0 0 1 100 700 Tm (first) Tj (second) ' ET");
let second = find(&items, "second");
assert!((second.y - 688.0).abs() < 0.1, "second y = {}", second.y);
assert!((second.x - 100.0).abs() < 0.1, "second x = {}", second.x);
}
#[test]
fn double_quote_operator_inside_form_sets_spacing_and_moves() {
// `aw ac (string) "` — set word spacing and char spacing, then T* and show.
let items =
form_items(b"BT /F1 12 Tf 12 TL 1 0 0 1 100 700 Tm (first) Tj 0 0 (second) \" ET");
let second = find(&items, "second");
assert!((second.y - 688.0).abs() < 0.1, "second y = {}", second.y);
assert!((second.x - 100.0).abs() < 0.1, "second x = {}", second.x);
}
#[test]
fn char_spacing_inside_form_widens_advance() {
// Tc was hardcoded to 0 in the form parser, so advance widths drifted.
// 2 glyphs x 600/1000 x 12pt = 14.4, plus 2 x Tc(2.0) = 18.4.
let items = form_items(b"BT /F1 12 Tf 1 0 0 1 100 700 Tm 2 Tc (AB) Tj ET");
let ab = find(&items, "AB");
assert!((ab.width - 18.4).abs() < 0.1, "AB width = {}", ab.width);
}
#[test]
fn q_restores_fill_colour_inside_form() {
// A white fill set inside q/Q must not leak past the Q — otherwise the
// following black text is treated as invisible and dropped entirely.
let items = form_items(
b"BT /F1 12 Tf 12 TL 1 0 0 1 100 700 Tm q 1 g (hidden) Tj Q T* (visible) Tj ET",
);
assert!(
items.iter().any(|item| item.text == "visible"),
"text after Q was dropped: {:?}",
items.iter().map(|i| &i.text).collect::<Vec<_>>()
);
assert!(
!items.iter().any(|item| item.text == "hidden"),
"white-filled text should still be suppressed"
);
}
#[test]
fn q_restores_text_state_inside_form() {
// Tc/TL live in the graphics state; `Q` must roll them back.
let items =
form_items(b"BT /F1 12 Tf 12 TL 1 0 0 1 100 700 Tm q 30 TL (a) Tj Q T* (b) Tj ET");
let b = find(&items, "b");
assert!(
(b.y - 688.0).abs() < 0.1,
"b y = {} (leading should restore to 12)",
b.y
);
}
}
+12 -1
View File
@@ -826,7 +826,10 @@ pub fn extract_text_in_regions_mem(
let height = get_page_height(&doc, page_id).unwrap_or(792.0);
page_heights.insert(*page_num, height);
// Extract text items for this page
// Extract text items for this page. The Form XObject budget is shared
// with the invisible-layer retry below so one page cannot consume two
// full expansion budgets.
let mut form_budget = extractor::FormWalkBudget::new();
let ((mut items, _rects, _lines), mut has_gid, mut coords_rotated, skipped_invisible) =
extractor::content_stream::extract_page_text_items(
&doc,
@@ -835,6 +838,7 @@ pub fn extract_text_in_regions_mem(
&font_cmaps,
false,
&mut style_cache,
&mut form_budget,
)?;
// OCR-layer fallback: scanned pages often carry their text as an
// invisible (Tr 3) layer behind the page raster. The visible-only
@@ -863,6 +867,7 @@ pub fn extract_text_in_regions_mem(
&font_cmaps,
true,
&mut style_cache,
&mut form_budget,
)
{
let inv_alnum = non_placeholder_alnum(&inv_items);
@@ -1045,6 +1050,7 @@ pub fn extract_tables_in_regions_mem(
&font_cmaps,
false,
&mut style_cache,
&mut extractor::FormWalkBudget::new(),
)?;
let threshold = text_utils::fix_letterspaced_items(&mut items);
if threshold > 0.10 {
@@ -1356,6 +1362,7 @@ pub fn detect_vector_grid_in_region_mem(
&font_cmaps,
false,
&mut extractor::FontStyleCache::new(),
&mut extractor::FormWalkBudget::new(),
)?;
text_utils::fix_letterspaced_items(&mut items);
@@ -1550,6 +1557,7 @@ mod vector_grid_tests {
&cmaps,
false,
&mut crate::extractor::FontStyleCache::new(),
&mut crate::extractor::FormWalkBudget::new(),
)
.unwrap();
@@ -1593,6 +1601,7 @@ mod vector_grid_tests {
&cmaps,
false,
&mut crate::extractor::FontStyleCache::new(),
&mut crate::extractor::FormWalkBudget::new(),
)
.unwrap();
@@ -2330,6 +2339,7 @@ pub fn extract_tables_with_structure_cells_mem(
&font_cmaps,
false,
&mut style_cache,
&mut extractor::FormWalkBudget::new(),
)?;
let threshold = text_utils::fix_letterspaced_items(&mut items);
if threshold > 0.10 {
@@ -3132,6 +3142,7 @@ fn detect_tsr_quality_issue(
&font_cmaps,
false,
&mut extractor::FontStyleCache::new(),
&mut extractor::FormWalkBudget::new(),
)?;
let adaptive_threshold = text_utils::fix_letterspaced_items(&mut items);
let coords = if coords_rotated {
+335 -15
View File
@@ -1,6 +1,6 @@
//! Rectangle-based table detection using union-find clustering.
use std::collections::HashMap;
use std::collections::{BTreeMap, HashMap, HashSet};
use log::debug;
@@ -78,19 +78,111 @@ pub(crate) fn rects_overlap(a: &(f32, f32, f32, f32), b: &(f32, f32, f32, f32),
!(a_right < b_left || b_right < a_left || a_top < b_bottom || b_top < a_bottom)
}
fn grid_coord(value: f32, cell: f32) -> i32 {
(value / cell).floor().clamp(-1_000_000.0, 1_000_000.0) as i32
}
/// Inclusive grid range. `None` if the rect covers more cells than we will
/// materialize — those rects are clustered via a bounded fallback.
fn grid_span(lo: f32, hi: f32, cell: f32) -> Option<std::ops::RangeInclusive<i32>> {
let a = grid_coord(lo.min(hi), cell);
let b = grid_coord(lo.max(hi), cell);
let span = b.saturating_sub(a);
if span > 64 {
return None;
}
Some(a..=b)
}
fn union_bucket_pairs(
uf: &mut UnionFind,
rects: &[(f32, f32, f32, f32)],
bucket: &[usize],
tolerance: f32,
) {
let m = bucket.len();
let mut pairs = 0usize;
'cell: for a in 0..m {
let i = bucket[a];
if uf.component_size(i) >= MAX_CLUSTER_RECTS {
continue;
}
for &j in &bucket[a + 1..] {
if pairs >= MAX_CLUSTER_PAIRS_PER_CELL {
break 'cell;
}
if uf.component_size(j) >= MAX_CLUSTER_RECTS {
continue;
}
pairs += 1;
if rects_overlap(&rects[i], &rects[j], tolerance) {
uf.union(i, j);
if uf.component_size(i) >= MAX_CLUSTER_RECTS {
break;
}
}
}
}
}
fn union_rect_against_bands(
uf: &mut UnionFind,
rects: &[(f32, f32, f32, f32)],
i: usize,
bands: &BTreeMap<i32, Vec<usize>>,
lo: i32,
hi: i32,
tolerance: f32,
) {
if uf.component_size(i) >= MAX_CLUSTER_RECTS {
return;
}
let mut pairs = 0usize;
let mut seen = HashSet::new();
for (_, bucket) in bands.range(lo..=hi) {
for &j in bucket {
if !seen.insert(j) {
continue;
}
if pairs >= MAX_CLUSTER_PAIRS_PER_CELL {
return;
}
if i == j || uf.component_size(j) >= MAX_CLUSTER_RECTS {
continue;
}
pairs += 1;
if rects_overlap(&rects[i], &rects[j], tolerance) {
uf.union(i, j);
if uf.component_size(i) >= MAX_CLUSTER_RECTS {
return;
}
}
}
}
}
/// Maximum component size for rect clustering. No real table has thousands
/// of cell rects — once a component exceeds this, it is a vector drawing or
/// page-spanning clipping path. We skip overlap checks for rects already in
/// an oversized component, keeping the original O(n²) loop but making it
/// effectively O(n) for pathological pages.
/// an oversized component.
const MAX_CLUSTER_RECTS: usize = 2000;
/// Pairwise-disjoint rects never merge, so a component-size cap does not
/// stop an all-pairs loop. Rects are hashed into this many points of grid
/// and compared only against others in the same cell.
const CLUSTER_GRID_CELL: f32 = 64.0;
/// All-pairs AABB tests allowed inside one grid cell. A real table cell is
/// tens of points wide, so a 64-pt cell holds a handful of neighbors — not
/// thousands of stacked drawings.
const MAX_CLUSTER_PAIRS_PER_CELL: usize = 16_384;
/// Cluster rects by spatial overlap using union-find.
/// Returns groups of rect indices; only groups with ≥ `min_size` rects are returned.
///
/// Skips overlap checks for rects whose component has already exceeded
/// [`MAX_CLUSTER_RECTS`], so pages with tens of thousands of vector-drawing
/// rects complete in milliseconds instead of minutes.
/// Overlap tests run inside a uniform grid so far-apart rects are never
/// compared, and each cell is pair-capped so a dense stack cannot go
/// quadratic or starve an independent table in another cell.
pub(crate) fn cluster_rects(
rects: &[(f32, f32, f32, f32)],
tolerance: f32,
@@ -98,23 +190,144 @@ pub(crate) fn cluster_rects(
) -> Vec<Vec<usize>> {
let n = rects.len();
let mut uf = UnionFind::new(n);
let cell = CLUSTER_GRID_CELL.max(tolerance * 4.0);
for i in 0..n {
// If rect i is already in an oversized component, no point comparing
// it against further rects — the component won't be used for table
// detection anyway.
let mut grid: HashMap<(i32, i32), Vec<usize>> = HashMap::new();
let mut large: Vec<usize> = Vec::new();
for (idx, &(x, y, w, h)) in rects.iter().enumerate() {
match (
grid_span(x - tolerance, x + w + tolerance, cell),
grid_span(y - tolerance, y + h + tolerance, cell),
) {
(Some(xs), Some(ys)) => {
for gx in xs {
for gy in ys.clone() {
grid.entry((gx, gy)).or_default().push(idx);
}
}
}
_ => large.push(idx),
}
}
let mut keys: Vec<_> = grid.keys().copied().collect();
keys.sort_unstable();
let mut keys_by_y: BTreeMap<i32, Vec<i32>> = BTreeMap::new();
for &key in &keys {
union_bucket_pairs(&mut uf, rects, &grid[&key], tolerance);
keys_by_y.entry(key.1).or_default().push(key.0);
}
// Oversized spans skip insert. Range-query occupied cells they cover so
// later X-ranges are not starved and we do not scan unrelated rows.
for &i in &large {
if uf.component_size(i) >= MAX_CLUSTER_RECTS {
continue;
}
for j in (i + 1)..n {
if rects_overlap(&rects[i], &rects[j], tolerance) {
uf.union(i, j);
// Check if the merged component just exceeded the cap —
// if so, no need to test more pairs for rect i.
let (x, y, w, h) = rects[i];
let x_lo = grid_coord(x - tolerance, cell);
let x_hi = grid_coord(x + w + tolerance, cell);
let y_lo = grid_coord(y - tolerance, cell);
let y_hi = grid_coord(y + h + tolerance, cell);
for (&gy, gxs) in keys_by_y.range(y_lo..=y_hi) {
let start = gxs.partition_point(|&gx| gx < x_lo);
for &gx in &gxs[start..] {
if gx > x_hi {
break;
}
let bucket = &grid[&(gx, gy)];
let mut pairs = 0usize;
for &j in bucket {
if pairs >= MAX_CLUSTER_PAIRS_PER_CELL {
break;
}
if uf.component_size(j) >= MAX_CLUSTER_RECTS {
continue;
}
pairs += 1;
if rects_overlap(&rects[i], &rects[j], tolerance) {
uf.union(i, j);
if uf.component_size(i) >= MAX_CLUSTER_RECTS {
break;
}
}
}
if uf.component_size(i) >= MAX_CLUSTER_RECTS {
break;
}
}
if uf.component_size(i) >= MAX_CLUSTER_RECTS {
break;
}
}
}
// Oversized-vs-oversized: band on the short axis so stacked or side-by-side
// page-spanning rules stay linear. Wide vs tall pairs are matched by
// querying the tall X-index; dual-oversized rects occupy every coarse-Y
// cell they span.
let mut large_x: BTreeMap<i32, Vec<usize>> = BTreeMap::new();
let mut large_y: BTreeMap<i32, Vec<usize>> = BTreeMap::new();
let mut large_coarse_y: BTreeMap<i32, Vec<usize>> = BTreeMap::new();
let mut wide: Vec<usize> = Vec::new();
let mut dual: Vec<usize> = Vec::new();
for &i in &large {
let (x, y, w, h) = rects[i];
let xs = grid_span(x - tolerance, x + w + tolerance, cell);
let ys = grid_span(y - tolerance, y + h + tolerance, cell);
match (xs, ys) {
(Some(xs), _) => {
for gx in xs {
large_x.entry(gx).or_default().push(i);
}
}
(_, Some(ys)) => {
wide.push(i);
for gy in ys {
large_y.entry(gy).or_default().push(i);
}
}
_ => {
dual.push(i);
let coarse = cell * 64.0;
match grid_span(y - tolerance, y + h + tolerance, coarse) {
Some(ys) => {
for gy in ys {
large_coarse_y.entry(gy).or_default().push(i);
}
}
None => {
large_coarse_y.entry(i32::MIN).or_default().push(i);
}
}
}
}
}
for bands in [&large_x, &large_y, &large_coarse_y] {
for bucket in bands.values() {
union_bucket_pairs(&mut uf, rects, bucket, tolerance);
}
}
// Cross-orientation is |wide|×|tall| if every wide rule spans the page.
// Skip that pass when the product cannot be a table (a few rules).
let tall_n = large
.len()
.saturating_sub(wide.len())
.saturating_sub(dual.len());
let cross_n =
(wide.len() + dual.len()).saturating_mul(tall_n) + dual.len().saturating_mul(wide.len());
if cross_n > 0 && cross_n <= MAX_CLUSTER_PAIRS_PER_CELL {
for &i in wide.iter().chain(&dual) {
let (x, _, w, _) = rects[i];
let x_lo = grid_coord(x - tolerance, cell);
let x_hi = grid_coord(x + w + tolerance, cell);
union_rect_against_bands(&mut uf, rects, i, &large_x, x_lo, x_hi, tolerance);
}
for &i in &dual {
let (_, y, _, h) = rects[i];
let y_lo = grid_coord(y - tolerance, cell);
let y_hi = grid_coord(y + h + tolerance, cell);
union_rect_against_bands(&mut uf, rects, i, &large_y, y_lo, y_hi, tolerance);
}
}
@@ -3810,6 +4023,113 @@ mod tests {
assert_eq!(groups[0].len(), 2);
}
#[test]
fn test_cluster_rects_overlapping_grid_still_clusters() {
// Neighboring cells overlap; the grid must still union the whole table.
let mut rects = Vec::new();
for row in 0..4 {
for col in 0..4 {
rects.push((col as f32 * 9.0, row as f32 * 9.0, 10.0, 10.0));
}
}
let groups = cluster_rects(&rects, 0.0, 1);
assert_eq!(groups.len(), 1);
assert_eq!(groups[0].len(), 16);
}
#[test]
fn test_cluster_rects_many_disjoint_stays_subquadratic() {
// Pairwise-disjoint rects never merge, so a component-size cap does
// not stop all-pairs overlap tests. Spread in X so they land in
// different grid cells; 8k is enough that n² tests would dominate.
let n = 8_000usize;
let rects: Vec<(f32, f32, f32, f32)> =
(0..n).map(|i| (i as f32 * 20.0, 0.0, 10.0, 10.0)).collect();
let groups = cluster_rects(&rects, 0.0, 2);
assert!(groups.is_empty());
}
#[test]
fn test_cluster_rects_stacked_disjoint_does_not_starve_later_table() {
// Same X, spread in Y: a spatial grid must still union an overlapping
// pair in another region of the page.
let n = 8_000usize;
let mut rects: Vec<(f32, f32, f32, f32)> =
(0..n).map(|i| (0.0, i as f32 * 20.0, 10.0, 10.0)).collect();
rects.push((500.0, 0.0, 10.0, 10.0));
rects.push((508.0, 0.0, 10.0, 10.0));
let groups = cluster_rects(&rects, 0.0, 2);
assert_eq!(groups.len(), 1);
assert_eq!(groups[0].len(), 2);
}
#[test]
fn test_cluster_rects_oversized_span_still_unions() {
// Wider than 64 grid cells; must still union the small overlapping rect.
let rects = vec![(0.0, 0.0, 5000.0, 10.0), (4900.0, 0.0, 10.0, 10.0)];
let groups = cluster_rects(&rects, 0.0, 1);
assert_eq!(groups.len(), 1);
assert_eq!(groups[0].len(), 2);
}
#[test]
fn test_cluster_rects_many_oversized_spans_all_get_a_pass() {
// More than 32 huge rects: the last one must still union its overlap.
let mut rects: Vec<(f32, f32, f32, f32)> = (0..40)
.map(|i| (0.0, i as f32 * 20.0, 5000.0, 10.0))
.collect();
rects.push((4900.0, 39.0 * 20.0, 10.0, 10.0));
let groups = cluster_rects(&rects, 0.0, 2);
assert_eq!(groups.len(), 1);
assert_eq!(groups[0].len(), 2);
}
#[test]
fn test_cluster_rects_oversized_not_starved_by_earlier_disjoint() {
// 9k earlier disjoint drawings would exhaust an index-order cap of
// 8,192 before the overlapping cell is visited.
let mut rects: Vec<(f32, f32, f32, f32)> = (0..9_000)
.map(|i| (10_000.0, i as f32 * 20.0, 10.0, 10.0))
.collect();
let wide = rects.len();
rects.push((0.0, 0.0, 5000.0, 10.0));
let target = rects.len();
rects.push((4900.0, 0.0, 10.0, 10.0));
let groups = cluster_rects(&rects, 0.0, 2);
assert!(
groups
.iter()
.any(|g| g.contains(&wide) && g.contains(&target)),
"wide rule and far-end cell must share a cluster"
);
}
#[test]
fn test_cluster_rects_wide_and_tall_oversized_union() {
let rects = vec![(0.0, 0.0, 5000.0, 10.0), (0.0, 0.0, 10.0, 5000.0)];
let groups = cluster_rects(&rects, 0.0, 2);
assert_eq!(groups.len(), 1);
assert_eq!(groups[0].len(), 2);
}
#[test]
fn test_cluster_rects_dual_oversized_spans_coarse_y() {
let rects = vec![(0.0, 0.0, 5000.0, 5000.0), (0.0, 4500.0, 5000.0, 5000.0)];
let groups = cluster_rects(&rects, 0.0, 2);
assert_eq!(groups.len(), 1);
assert_eq!(groups[0].len(), 2);
}
#[test]
fn test_cluster_rects_many_wide_and_tall_stays_subquadratic() {
let mut rects = Vec::with_capacity(4_000);
for i in 0..2_000 {
rects.push((0.0, i as f32 * 20.0, 5000.0, 10.0));
rects.push((i as f32 * 20.0, 0.0, 10.0, 5000.0));
}
let _groups = cluster_rects(&rects, 0.0, 2);
}
// --- snap_edges ---
#[test]
+211 -29
View File
@@ -540,18 +540,14 @@ impl ToUnicodeCMap {
/// Remap a CMap that references pre-subsetting GIDs to sequential post-subsetting GIDs.
/// Collects all source CIDs, sorts them, and reassigns to 1, 2, 3, ...
///
/// Range expansion stops after `MAX_CID_W_EXPANSION` CID visits, counting
/// overwrites, so repeated full-width `bfrange`s cannot re-expand the
/// 16-bit domain. Later overlapping ranges that would have introduced new
/// CIDs after that many visits are truncated.
pub fn remap_to_sequential(&self) -> ToUnicodeCMap {
let mut cid_to_unicode: HashMap<u16, String> = HashMap::new();
// Expand ranges first
for &(start, end, base) in &self.ranges {
for cid in start..=end {
let unicode_cp = base + (cid - start) as u32;
if let Some(ch) = char::from_u32(unicode_cp) {
cid_to_unicode.insert(cid, ch.to_string());
}
}
}
expand_bfranges_for_remap(&self.ranges, &mut cid_to_unicode, MAX_CID_W_EXPANSION);
// char_map entries override range entries
for (&cid, unicode) in &self.char_map {
@@ -576,6 +572,33 @@ impl ToUnicodeCMap {
}
}
/// Expand `bfrange` entries into individual CID→Unicode inserts.
/// Returns how many CIDs were visited. Counts overwrites so a repeated
/// full-width range cannot keep working after `max_assignments`.
fn expand_bfranges_for_remap(
ranges: &[(u16, u16, u32)],
cid_to_unicode: &mut HashMap<u16, String>,
max_assignments: usize,
) -> usize {
let mut assigned = 0usize;
'ranges: for &(start, end, base) in ranges {
if start > end {
continue;
}
for cid in start..=end {
if assigned >= max_assignments {
break 'ranges;
}
assigned += 1;
let unicode_cp = base + (cid - start) as u32;
if let Some(ch) = char::from_u32(unicode_cp) {
cid_to_unicode.insert(cid, ch.to_string());
}
}
}
assigned
}
/// Parse a hex string to u16
fn parse_hex_u16(hex: &str) -> Option<u16> {
u16::from_str_radix(hex.trim(), 16).ok()
@@ -1561,23 +1584,31 @@ fn parse_encoding_cmap_stream(data: &[u8]) -> Option<EncodingCMap> {
}
let mut map = HashMap::new();
let mut assigned = 0usize;
let mut pos = 0;
while let Some(start) = text[pos..].find("begincidchar") {
let section_start = pos + start + "begincidchar".len();
if let Some(end) = text[section_start..].find("endcidchar") {
let section = &text[section_start..section_start + end];
parse_cidchar_section(section, &mut map, &mut src_hex_lengths);
if !parse_cidchar_section(section, &mut map, &mut src_hex_lengths, &mut assigned) {
break;
}
pos = section_start + end;
} else {
break;
}
}
pos = 0;
while let Some(start) = text[pos..].find("begincidrange") {
while assigned < MAX_CID_W_EXPANSION {
let Some(start) = text[pos..].find("begincidrange") else {
break;
};
let section_start = pos + start + "begincidrange".len();
if let Some(end) = text[section_start..].find("endcidrange") {
let section = &text[section_start..section_start + end];
parse_cidrange_section(section, &mut map, &mut src_hex_lengths);
if !parse_cidrange_section(section, &mut map, &mut src_hex_lengths, &mut assigned) {
break;
}
pos = section_start + end;
} else {
break;
@@ -1612,7 +1643,8 @@ fn parse_cidchar_section(
section: &str,
map: &mut HashMap<u16, u16>,
src_hex_lengths: &mut Vec<usize>,
) {
assigned: &mut usize,
) -> bool {
let mut chars = section.chars().peekable();
loop {
while chars.peek().is_some_and(|c| c.is_whitespace()) {
@@ -1643,16 +1675,20 @@ fn parse_cidchar_section(
}
}
if let (Some(code), Ok(cid)) = (parse_hex_u16(&src_hex), cid_str.parse::<u16>()) {
map.insert(code, cid);
if !assign_encoding_cid(map, code, cid, assigned) {
return false;
}
}
}
true
}
fn parse_cidrange_section(
section: &str,
map: &mut HashMap<u16, u16>,
src_hex_lengths: &mut Vec<usize>,
) {
assigned: &mut usize,
) -> bool {
let mut chars = section.chars().peekable();
loop {
while chars.peek().is_some_and(|c| c.is_whitespace()) {
@@ -1703,12 +1739,34 @@ fn parse_cidrange_section(
) else {
continue;
};
if start > end {
continue;
}
let mut cid = start_cid;
for code in start..=end {
map.insert(code, cid);
if !assign_encoding_cid(map, code, cid, assigned) {
return false;
}
cid = cid.saturating_add(1);
}
}
true
}
fn assign_encoding_cid(
map: &mut HashMap<u16, u16>,
code: u16,
cid: u16,
assigned: &mut usize,
) -> bool {
// Count overwrites: unique-key coverage alone would not stop a repeated
// full-width range from re-inserting all 65,536 codes.
if *assigned >= MAX_CID_W_EXPANSION {
return false;
}
map.insert(code, cid);
*assigned += 1;
true
}
fn parse_binary_cmap_encoding(data: &[u8]) -> Result<EncodingCMap, String> {
@@ -1832,6 +1890,13 @@ fn merge_cmaps(mut base: ToUnicodeCMap, overlay: ToUnicodeCMap) -> ToUnicodeCMap
base
}
/// Shared 16-bit CID expansion cap (65,536).
/// Encoding `begincidrange`, `/W` width assignment, and ToUnicode sequential
/// remap count every insert, including overwrites, so a repeated full-width
/// range cannot keep working after the domain is filled. The `/W` unicode
/// heuristic caps unique CIDs with the same number.
pub(crate) const MAX_CID_W_EXPANSION: usize = 65_536;
/// Check if a CIDFont's /W (widths) array contains CID values that look like
/// Unicode codepoints rather than low-value GIDs.
///
@@ -1843,20 +1908,23 @@ pub(crate) fn cid_values_look_like_unicode(cid_font_dict: &lopdf::Dictionary) ->
_ => return false,
};
// The /W array format: [cid [w1 w2 ...]] or [cid_start cid_end w]
// We extract all CID values (the first element of each group).
let mut cids: Vec<u16> = Vec::new();
// The /W array format: [cid [w1 w2 ...]] or [cid_start cid_end w].
// Collect unique CIDs only: repeating a full-width range must not grow a
// temporary vector (or the sort) with the range length on every copy.
let mut seen = HashSet::new();
let mut i = 0;
while i < w_arr.len() {
while i < w_arr.len() && seen.len() < MAX_CID_W_EXPANSION {
if let Ok(cid) = w_arr[i].as_i64() {
cids.push(cid as u16);
// Skip the width data
let start = cid as u16;
if i + 1 < w_arr.len() {
match &w_arr[i + 1] {
Object::Array(widths) => {
// [cid [w1 w2 ...]] — CIDs are cid, cid+1, ..., cid+len-1
for j in 1..widths.len() {
cids.push((cid as u16).wrapping_add(j as u16));
for j in 0..widths.len() {
if seen.len() >= MAX_CID_W_EXPANSION {
break;
}
seen.insert(start.wrapping_add(j as u16));
}
i += 2;
}
@@ -1864,9 +1932,7 @@ pub(crate) fn cid_values_look_like_unicode(cid_font_dict: &lopdf::Dictionary) ->
// [cid_start cid_end w] — range of CIDs
if i + 2 < w_arr.len() {
if let Ok(cid_end) = w_arr[i + 1].as_i64() {
for c in (cid as u16)..=(cid_end as u16) {
cids.push(c);
}
record_unique_cid_range(start, cid_end as u16, &mut seen);
}
i += 3;
} else {
@@ -1875,6 +1941,7 @@ pub(crate) fn cid_values_look_like_unicode(cid_font_dict: &lopdf::Dictionary) ->
}
}
} else {
seen.insert(start);
i += 1;
}
} else {
@@ -1882,10 +1949,11 @@ pub(crate) fn cid_values_look_like_unicode(cid_font_dict: &lopdf::Dictionary) ->
}
}
if cids.is_empty() {
if seen.is_empty() {
return false;
}
let mut cids: Vec<u16> = seen.into_iter().collect();
cids.sort_unstable();
let median = cids[cids.len() / 2];
// Unicode text CIDs are typically >= 0x20 (space) with letters at 0x41+.
@@ -1894,6 +1962,18 @@ pub(crate) fn cid_values_look_like_unicode(cid_font_dict: &lopdf::Dictionary) ->
median >= 0x41
}
fn record_unique_cid_range(start: u16, end: u16, seen: &mut HashSet<u16>) {
if start > end {
return;
}
for cid in start..=end {
if seen.len() >= MAX_CID_W_EXPANSION {
return;
}
seen.insert(cid);
}
}
/// Build a ToUnicodeCMap from predefined CID→Unicode mapping based on CIDSystemInfo.
///
/// Supports Adobe-Korea1 (Korean) character collection. Can be extended for
@@ -2867,6 +2947,33 @@ endbfrange
assert!(remapped.ranges.is_empty());
}
#[test]
fn remap_to_sequential_repeated_full_bfranges_stay_bounded() {
// 5,000 copies of `<0003> <ffff>` must stop after 65,536 CID visits,
// not 5,000 × ~65,533 expansions.
let ranges = vec![(3u16, 65535u16, 0x41u32); 5_000];
let mut map = std::collections::HashMap::new();
let assigned = expand_bfranges_for_remap(&ranges, &mut map, MAX_CID_W_EXPANSION);
assert_eq!(assigned, MAX_CID_W_EXPANSION);
assert!(map.len() <= MAX_CID_W_EXPANSION);
let mut body = String::new();
let mut remaining = 5_000usize;
while remaining > 0 {
let n = remaining.min(100);
body.push_str(&format!("{n} beginbfrange\n"));
for _ in 0..n {
body.push_str("<0003> <ffff> <0041>\n");
}
body.push_str("endbfrange\n");
remaining -= n;
}
let data = format!("1 begincodespacerange\n<0000> <ffff>\nendcodespacerange\n{body}");
let cmap = ToUnicodeCMap::parse(data.as_bytes()).unwrap();
let remapped = cmap.remap_to_sequential();
assert_eq!(remapped.lookup(1), Some("A".to_string()));
}
#[test]
fn test_min_source_cid() {
let cmap_content = r#"
@@ -3298,4 +3405,79 @@ endbfrange
"An indirect /Subtype naming CIDFontType2 must still reach the remap"
);
}
#[test]
fn cid_values_look_like_unicode_letter_range() {
let mut dict = lopdf::Dictionary::new();
dict.set(
"W",
Object::Array(vec![
Object::Integer(0x41),
Object::Integer(0x5A),
Object::Integer(500),
]),
);
assert!(cid_values_look_like_unicode(&dict));
}
#[test]
fn cid_values_look_like_unicode_low_gids() {
let mut dict = lopdf::Dictionary::new();
dict.set(
"W",
Object::Array(vec![
Object::Integer(0),
Object::Array(vec![Object::Integer(500); 10]),
]),
);
assert!(!cid_values_look_like_unicode(&dict));
}
#[test]
fn cid_values_look_like_unicode_repeated_full_ranges_stay_bounded() {
// Repeating `[0 65535 w]` must not materialize 65,536 CIDs per copy.
let mut w = Vec::new();
for _ in 0..5_000 {
w.push(Object::Integer(0));
w.push(Object::Integer(65535));
w.push(Object::Integer(500));
}
let mut dict = lopdf::Dictionary::new();
dict.set("W", Object::Array(w));
assert!(cid_values_look_like_unicode(&dict));
}
#[test]
fn encoding_cidrange_maps_a_normal_range() {
let data = b"1 begincodespacerange\n<0000> <FFFF>\nendcodespacerange\n\
1 begincidrange\n<0041> <0043> 65\nendcidrange\n";
let enc = parse_encoding_cmap_stream(data).unwrap();
assert_eq!(enc.map.get(&0x41), Some(&65));
assert_eq!(enc.map.get(&0x42), Some(&66));
assert_eq!(enc.map.get(&0x43), Some(&67));
assert_eq!(enc.map.len(), 3);
assert_eq!(enc.code_byte_length, 2);
}
#[test]
fn encoding_cidrange_repeated_full_ranges_stay_bounded() {
// 5,000 copies of `<0000> <ffff> 0` must not re-expand the 16-bit
// domain on every declaration.
let mut body = String::new();
let mut remaining = 5_000usize;
while remaining > 0 {
let n = remaining.min(100);
body.push_str(&format!("{n} begincidrange\n"));
for _ in 0..n {
body.push_str("<0000> <ffff> 0\n");
}
body.push_str("endcidrange\n");
remaining -= n;
}
let data = format!("1 begincodespacerange\n<0000> <FFFF>\nendcodespacerange\n{body}");
let enc = parse_encoding_cmap_stream(data.as_bytes()).unwrap();
assert!(enc.map.len() <= MAX_CID_W_EXPANSION);
assert_eq!(enc.map.get(&0), Some(&0));
assert_eq!(enc.map.get(&65535), Some(&65535));
}
}
+2 -2
View File
@@ -724,7 +724,7 @@ checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e"
[[package]]
name = "pdf-inspector"
version = "1.14.1"
version = "1.14.2"
dependencies = [
"env_logger",
"include_dir",
@@ -740,7 +740,7 @@ dependencies = [
[[package]]
name = "pdf-inspector-wasm"
version = "1.14.1"
version = "1.14.2"
dependencies = [
"console_error_panic_hook",
"js-sys",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "pdf-inspector-wasm"
version = "1.14.1"
version = "1.14.2"
edition = "2021"
authors = ["Firecrawl Team"]
description = "Browser WebAssembly bindings for pdf-inspector"