Releases fix(regions) #351 — invisible (Tr 3) OCR text layers served from
the region extractor instead of falling back to GPU OCR.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(regions): serve invisible (Tr 3) OCR text layers instead of needs_ocr
Scanned pages (archive.org-style digitizations) carry their text as an
invisible render-mode-3 layer behind the page raster. extract_text_in_regions
extracted only visible items, so such pages yielded nothing but
'[Image: ...]' placeholders and every region fell back to OCR — while the
markdown path already includes the invisible layer for Mixed PDFs. The two
extractors disagreed about the same page.
Mirror the markdown path's gate, page-scoped: when the visible pass is
effectively textless (<40 non-placeholder alphanumerics), retry with
invisible text included and adopt the retry only when it contributes real,
non-garbage text. Pages with real visible text never retry, so double-layer
PDFs (visible text plus an invisible accessibility copy) are unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: strict zero-visible adoption gate, whole-layer garbage check, doc placement, discriminating tests
- Adoption now requires ZERO visible text on the page: the invisible pass
returns visible items too, so adopting alongside any visible text would
duplicate it. Strict gate instead of fuzzy dedupe; the dead
inv_alnum > visible_alnum condition goes with it.
- Garbage check judges the whole recovered layer, not the first 200 items.
- Constant/helper moved above the doc block so rustdoc stays attached to
extract_text_in_regions_mem.
- Tests: any-visible-text-blocks-adoption case (single short visible line)
with exactly-once assertions; mode-0 guard asserts occurrence count.
- Re-verified against real scanned-book pages: all recover (9.2-11.7K chars,
needs_ocr=false) — pure OCR-layer scans carry no visible text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: gate adoption on visible-item presence, not alphanumeric mass
Punctuation-only visible text (zero alphanumerics) could still adopt the
invisible pass; its OCR twin in the layer would duplicate the glyphs. The
gate is now item-presence: any non-image item with non-whitespace text
blocks adoption (whitespace-only artifacts still tolerated). Pinned by a
punctuation-only fixture test; real scanned-book pages re-verified — all
recover unchanged (they carry no visible items at all).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: punctuation-gate test pins preservation, comment states fixture scope
The fixture's visible punctuation is a separate block, not mirrored in the
invisible layer — so it pins the gate, not the duplication scenario. The
doc comment now says so, and a positive assertion checks the punctuation
survives exactly once.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: retry only when invisible text was actually skipped; negative gate tests
- extract_page_text_items now reports skipped_invisible (4th return): the
Tr-3 suppression sites set it, so the region extractor retries only when
a recoverable layer exists. Blank pages and image-only scans without an
OCR layer — the common scanned case — no longer pay a second
content-stream parse.
- Negative tests: below-floor watermark layer and symbol-garbage layer are
both rejected (region keeps only the raster placeholder). needs_ocr
semantics for placeholder-only regions deliberately unchanged — that
contract predates this PR and downstream pipelines handle it.
- Real scanned-book pages re-verified: recovery unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: flag the ' show-text suppression path; require nonempty strings
- P1: invisible layers shown via the ' operator never set skipped_invisible,
so those pages kept the placeholder and fell back to GPU OCR. The '
suppression path now flags too — pinned by a fixture whose layer is shown
entirely via ' (nothing through Tj/TJ).
- P3: all three flag sites (Tj, TJ, ') require a nonempty string operand —
numeric-only TJ kerning arrays and empty shows no longer trigger the
invisible reparse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Tagged PDFs carry a structure tree with real heading roles (H1..H6), and
the core already parses it (structure_tree::StructTree) and threads MCIDs
onto TextItem — but neither surfaced through the bindings.
- Expose TextItem.mcid (Option<i64>) through the napi and pyo3 bindings,
matching the core field added with the marked-content extractor.
- Add StructRole::name(), the inverse of from_name, so roles have a
stable string form.
- Add extract_structure_elements / extract_structure_elements_mem to the
core: one (page, mcid, role) entry per marked-content reference, sorted
by (page, mcid), empty for untagged PDFs. Pages are 1-indexed to match
TextItem.page, so results join directly against
extract_text_with_positions output.
- Bind it as extractStructureElements (napi) and
extract_structure_elements / extract_structure_elements_bytes (pyo3),
with type-stub updates in pdf_inspector.pyi.
- Cover the join in Rust integration tests, napi test.mjs, and pytest,
using the existing firecrawl_docs_tagged.pdf fixture (tagged) and
thermo-freon12.pdf (untagged).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(structure-tree): bound tagged /K parsing against alias/cycle DoS
A struct element that references itself (or an ancestor) through /K — e.g.
/K [n 0 R n 0 R] — made parse_struct_element_dict branch exponentially:
the depth cap (64) alone still permits 2^depth materialized nodes, so a
~830-byte PDF exhausts memory (OOM, exit 134).
Add a StructWalk carrying (1) an active-path set of object IDs so a node
that references itself/an ancestor is not re-expanded (breaks self- and
mutual-reference cycles cheaply), and (2) a global node budget
(MAX_STRUCT_NODES) that caps total materialization for aliased/DAG-shaped
graphs of distinct objects the path guard cannot catch.
Adds regression tests for self-alias, mutual-alias, and the aliased-DAG
budget cap.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(structure-tree): charge /K content refs against the node budget
The per-node budget only covered materialized struct elements and child
recursion; bare MCIDs and MCR dicts in a /K array append to content_refs
without charging it, so one element with a very wide /K array could still
allocate content_refs without bound. Charge every /K array item before
handling it, and stop the top-level /K loop once the budget is spent, so
content refs and loop work are bounded too. Adds a wide-MCID-array test.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(structure-tree): charge /K budget per materialized item, not per array entry
Charging every /K array item double-counted structural children (charged
here and again at their node entry) and charged cycle-skipped references
that materialize nothing, draining the budget up to ~2x faster than the
per-node semantics and risking early truncation of large legitimate trees.
Charge only the unbounded content-ref items (bare MCIDs and MCR dicts);
structural children remain charged once at their node entry.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* refactor(structure-tree): charge every content ref uniformly via helper
Route all budget charges through StructWalk::charge() so every
marked-content reference is charged once, including the single-value /K
branches (bare integer and MCR dict) that previously appended without
charging. charge() also guards against underflow, so charging after the
node-entry charge (which can leave the budget at 0) is safe. Makes the
documented per-item budget contract hold uniformly across all branches.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* feat(structure-tree): log once when the node budget truncates parsing
Add a one-shot truncation flag on StructWalk, set the first time the
budget is exhausted, and emit a single warn! after parsing so an operator
can tell when a (very large or malformed) tagged tree was cut off. Avoids
per-item log spam; negligible overhead on the normal path.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(structure-tree): flag truncation at budget guards, not just in charge
The truncation flag was only set inside charge() on the budget==0 branch,
but the dominant skip paths use budget==0 guards that break/return before
charge() is ever called with an empty budget, so the flag (and the warn!)
almost never fired. Route those guards through a new exhausted() that sets
the flag when it skips remaining work. Adds a parser-level test that would
have caught the missed warning.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(structure-tree): flag cycle/depth skips and charge bare MCIDs fully
Two review follow-ups:
- Cycle-broken and depth-capped /K skips dropped tagged content without
setting the truncation flag, so the one-shot warning never fired for
malformed/over-deep trees. Mark those skips via note_skipped() and
broaden the warning to cover non-budget truncation.
- A bare /K MCID materializes a wrapper node AND a content reference but
charged only one budget unit, allowing ~2x the advertised budget for
such content; charge both.
Adds tests: cycle-skip flags truncation, and bare MCID charges two units.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(structure-tree): charge MCR-dict wrappers the same two units as bare MCIDs
A top-level MCR /K dict flows through parse_kid -> parse_struct_element_dict
and materializes a Span node + one content ref (two items) but was charged
only one unit at node entry, while the bare-MCID path charges two. Charge
the content reference in the MCR branch too so the per-item budget is
uniform across both wrapper paths. Adds a symmetric test.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(structure-tree): reserve leaf-wrapper budget units atomically
A leaf MCID wrapper (bare MCID or MCR dict) materializes a node + one
content ref and charged the two units via separate charge() calls. At the
last unit the first charge succeeded and the second failed, consuming a
unit without emitting the wrapper and denying it to a later element that
would have fit. Add charge_n() to reserve both units atomically (or
neither), and detect MCR before the node charge so it reserves both up
front. Adds a boundary test asserting the leftover unit is preserved.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(structure-tree): stop scanning wide /K once a leaf reservation stalls
The atomic charge_n(2) left budget nonzero (==1) when it failed, so
exhausted() (budget==0) never broke the root /K loop and a crafted wide
array of leaf wrappers was scanned in full after no leaf could fit. Add a
stalled flag set on an insufficient reservation and fold it into
exhausted(); charge()-based (one-unit) loops are unaffected since they
reach budget 0 exactly. Adds a test that a one-unit budget still allows a
one-unit item but a failed two-unit reservation stops the scan.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(structure-tree): add traversal budget and stop charging non-materializing dicts
Two review follow-ups on budget accounting:
- Wide /K arrays of non-materializing items (unsupported value types, OBJR
dicts, cycle back-edges) consumed no node budget, so the loop scanned the
whole array. Add a separate work budget charged per examined /K item and
break the loops when it is spent, bounding traversal even when nothing
materializes.
- OBJR dicts and dicts without a valid /S were charged the node budget before
being recognized and skipped, draining the shared budget and truncating
real content later. Hoist the OBJR check and /S validation above the node
charge so only materializing nodes consume it (matching the MCR hoisting).
Adds tests for the work-budget bound, wide unsupported /K, and non-materializing
dicts not charging the node budget.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* docs(structure-tree): mention traversal budget in truncation warning
The one-shot truncation warning listed the node budget, cycle, and depth
as causes but not the new traversal (work) budget, so a work-budget
truncation printed a misleading message. Include MAX_STRUCT_WORK so
malformed-PDF debugging identifies the actual limit hit.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* feat(napi): add processPdfAsync, classifyPdfAsync, extractPagesMarkdownAsync
The Node bindings are synchronous, so every call parses on the event
loop thread — up to hundreds of milliseconds of dead loop per document
in a server. Add additive AsyncTask-based variants that run the same
shared implementations on the libuv thread pool and return promises.
The existing synchronous exports keep their names, signatures, and
behaviour; each sync/async pair shares one implementation. Panics in
compute() are caught and surfaced as rejections, matching the sync
error contract.
Closes#336
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(napi): read async task buffers in place instead of copying
Review feedback on #337: buffer.to_vec() copied the whole PDF on the
event loop before the task was queued, so large inputs still stalled
the loop and doubled peak memory. The tasks now hold the napi Buffer
itself — its ref pins the JS allocation for the task's lifetime and
the backing store is stable, so compute() reads it directly from the
worker thread. Callers must not mutate the buffer until the promise
settles (same contract as Node's async fs APIs); documented on each
export and in the README.
The suggested removal of ts_return_type was checked and rejected:
without it napi-rs generates Promise<unknown> for AsyncTask returns.
A comment now records that finding.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(napi): copy async task input on the JS thread for soundness
Review feedback on #337: holding the napi Buffer and reading it from
the libuv worker was unsound. Buffer derefs straight to the JS-side
allocation, so a caller mutating it before the promise settled would
race the worker's reads — undefined behavior, not a recoverable error,
and the documented don't-mutate contract was unenforceable. Deferring
the copy to compute() would not help: any off-thread read races the
same way. The JS thread is the only race-free place to take the copy,
because JS is single-threaded and nothing can mutate the buffer during
the synchronous part of the call.
Revert to an owned Vec<u8> copied at call time. The cost is one memcpy,
negligible next to the parse the async variants exist to unblock. Docs
now state the buffer may be reused or mutated immediately, and a test
locks in the copy semantics by mutating the input while a parse is in
flight.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* Bound column-detection histogram size
Derive the projection histogram from a clamped bin count and skip
non-finite page widths. Extreme or malformed text-item coordinates
(from the content-stream text matrix) could otherwise drive a very
large allocation. 65,536 bins is ~9x the largest legal page, so real
layouts are unaffected. Adds regression tests.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* Exclude non-finite coordinates from page bounds
Items at NaN/inf positions are now skipped when folding the page bounds,
so a malformed coordinate can no longer escape as a ColumnRegion
boundary, and an all-non-finite page returns no columns. Bad items are
dropped individually rather than failing the page, so one stray glyph
does not disable column detection.
Addresses review feedback on the finite-width guard.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* Trim far-outlier coordinates from page bounds
Gutter margins, spanning-item width and the XY-cut margin are all
fractions of page_width, so a single far-but-finite item (x=50_000 is
enough) set the scale for the whole page: real gutters fell inside the
rejected margin band and a genuine two-column page collapsed to one
region. When the span exceeds one legal page (14_400 units), re-derive
the bounds from items clustered around the median x. Outliers keep their
text because column assignment buckets by nearest overlap.
The MAX_BINS ceiling stays as an allocation bound that does not depend
on this heuristic.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* Harden bounds trimming against widths and wide layouts
Check both item edges when trimming: a malformed width at an ordinary
position poisoned x_max just as a malformed position poisoned x_min, so
a huge width still collapsed a two-column page to one region.
Only trim when the far items are a small minority (<=10%). A genuinely
large-format page has content spread across its full width, so it now
keeps its true bounds instead of being reduced to the median cluster.
Correct the MAX_PAGE_EXTENT comment: 14_400 units is the traditional
Acrobat architectural limit, not a format cap. PDF 2.0 sets no page-size
limit and UserUnit scales physical size, so this is a heuristic.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* Scale bin width so the histogram spans the whole page
Clamping the bin count alone left anything past MAX_BINS * BIN_WIDTH
(~131k points) outside the histogram, folded into the final bin. A page
wide enough to hit that lost real gutters: with a visible gutter inside
the covered range the XY-cut fallback never runs, so a three-column
layout silently reported two. Derive bin_width from page_width instead,
keeping the same allocation ceiling and degrading only resolution.
Also anchor the trimming median on the same finite left/right items that
bounds() accepts, so a malformed width cannot shift which items count as
strays.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* Require geometric evidence before detaching far content
The median-window trim narrowed any content more than one page from the
centre, so a valid large page with a sparse far sidebar lost the sidebar
from its bounds and its text fell into column 0. An item-count minority
rule cannot tell that layout from malformed coordinates.
Group content into clusters separated by more than a whole page of
continuous emptiness, and only drop a cluster that is both detached by
such a void and a small minority of items. Real content does not leave a
gap that large; a stray coordinate sits alone beyond one.
A single run wider than one page is treated as a malformed width, which
also covers the huge-width case the cluster sweep cannot see (such an
item spans everything and leaves no gap).
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* Judge run width against page content, not a fixed extent
Treating any run wider than 14_400 units as malformed penalised valid
large pages: one made entirely of such runs reported no columns at all,
and a mixed page lost the right edge of every long run.
Judge width relative to the page's own content instead. Positions cannot
be inflated by a bogus width, so the spread of the core cluster is a
sound scale: a run wider than that spread plus one page is malformed.
A genuinely large page keeps its genuinely long runs, while a 1e12-wide
run beside ordinary text is still rejected.
Cluster on positions rather than filled intervals, so a bogus width can
no longer merge everything into one cluster, and keep ordinary pages on
an O(n) fast path that skips the sort entirely.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* Update SECURITY.md reporting channels
Clarify that email is the only required channel and point the
alternative at Firecrawl's Bugcrowd disclosure engagement instead of
the private-advisory link, which is not enabled on this repo.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* Make Bugcrowd the preferred reporting channel
Bugcrowd's disclosure engagement is the primary channel; email to
help@firecrawl.dev is offered as the alternative.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
Use hex.get(i..i+2) instead of &hex[i..i+2] so a non-hex, non-ASCII
destination in a /ToUnicode CMap can no longer trigger a UTF-8
char-boundary panic. An even byte length does not guarantee the byte
offset falls on a char boundary; get() returns None on a non-boundary
or out-of-range index, folding cleanly into the existing flow.
Add regression tests covering a multi-byte destination char and a
replacement-char byte.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
Use str::get instead of a byte-length check plus slice when parsing the
uniXXXX glyph-name form. The byte-length guard only proved the index was
in bounds, not on a UTF-8 char boundary, so a glyph name containing
non-ASCII bytes could cause a slice on a non-boundary index. Switch to a
checked slice that folds into the existing Option flow, and add tests.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(links): guard AcroForm /Kids traversal against cycles and huge trees
A crafted PDF whose AcroForm field lists itself (or another ancestor) in
/Kids caused walk_form_fields to recurse indefinitely, overflowing the
stack and aborting pdf2md (exit 134) — an application-level DoS from a
~730-byte input.
Track visited field object IDs to break /Kids cycles, and cap total
field-node traversal at 100k nodes to bound pathologically large trees.
Adds regression tests for self-cycle and mutual-cycle field graphs.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(links): cap AcroForm /Kids recursion depth to stop deep-chain overflow
The visited-set guard stops cyclic /Kids graphs, but a long *acyclic*
chain of distinct fields still recurses to the chain length and overflows
the stack (a ~1.6MB PDF with 20k linked fields aborts pdf2md, exit 134)
before the 100k node budget is reached.
Add an explicit recursion depth cap (100 levels — far above any legitimate
form hierarchy) so stack usage is bounded independently of node count.
Adds a deep-acyclic-chain regression test.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(links): enforce form-field node budget before insertion
The node-budget guard inserted each field ID into the visited set before
checking the budget, so the check triggered an early return but never
actually capped the set. A field with a huge /Kids array kept inserting
post-budget IDs, letting visited (memory and work) grow with the crafted
input rather than stopping at MAX_FORM_FIELD_NODES.
Check depth and budget before inserting, so visited can never exceed the
cap. Adds a wide-tree regression test.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(links): stop /Fields and /Kids iteration once node budget is spent
Checking the budget before insertion capped the visited set, but callers
still iterated every remaining entry of a wide /Fields or /Kids array
after the budget was exhausted — each walk returned immediately, yet the
O(N) sibling iteration let a single multi-million-entry array burn
extraction CPU unbounded. Break out of both the top-level and recursive
loops once visited reaches the cap, making the budget a true
traversal-work cap. Adds a top-level wide-/Fields regression test.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(links): charge examined entries against the field-node budget
The budget counted only distinct visited nodes, so /Fields or /Kids
arrays full of invalid (non-reference) or duplicate entries never grew
visited and ran to completion regardless of size — the node budget did
not actually cap traversal work.
Introduce FieldWalkBudget tracking both visited nodes and total entries
examined; charge every array entry (valid, invalid, or duplicate) and
stop once either hits MAX_FORM_FIELD_NODES. Adds a regression test with a
huge /Kids array of duplicate + null entries.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(links): iterate /Fields and /Kids arrays by borrow, not clone
Both arrays were cloned in full before the budget check, so a crafted
oversized /Fields or /Kids array forced an O(n) allocation and copy
regardless of the cap. resolve_array already returns a borrow tied to the
document and the walker only needs a shared &Document, so iterate the
borrowed arrays directly — the early break now bounds how many entries
are even touched, before any per-array allocation.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* docs(links): correct wide-array test comments to match range assertions
The two wide-array tests assert item counts within a range near the
budget, not an exact value (charging entries in the entry guard shifts
the boundary by one or two). Fix the stale comments that claimed exact
counts.
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* fix(tables): exclude script attachments and tiny numeric fragments from detection
Split out of #242 (draft) so it can be reviewed on its own evidence.
Display equations with sub/superscripts form phantom small-font table
regions: the subscripts cluster with nearby small text (footnotes, axis
labels) into fake multi-column grids. Two guards:
- Script attachment: a small-font item horizontally adjacent to a
larger-font item at a genuine baseline offset is a sub/superscript,
not a table cell, and is excluded from candidates. A real baseline
offset is required so a small cell beside a larger same-baseline
label is never filtered. Attachment targets are indexed by Y and
scanned through a bounded window rather than a full-page sweep.
The body-font pass applies the same exclusion but only for
heading-sized anchors (>= 1.15x base), so body-size table cells
beside slightly larger labels are untouched.
- Tiny numeric fragments: a <=2-row grid whose every cell is a bare
1-2 digit number carries no tabular information. Restricted to the
small-font pass, where the pattern is overwhelmingly exponent
clusters; body-font numeric grids are unaffected.
Corpus impact: 20 of 186 documents, measured against a control build of
main so main's own drift is excluded. Table rows fall in 17 of 18
inspected documents and no content is lost — 2103_07786 drops all 21
rows, every one a math fragment ('|X 42 43|1|||'); Stijn_SB_doc drops
173 rows of footnote text that had been shredded into cells, with word
count slightly UP and footnote markers intact. M2019_mordeste gains 11
rows from a 3-column grid re-detected as 2-column, neither clearly
better nor worse.
Note the 20 documents is far more than the 3 that per-change ablation
suggested: that figure measured sole-cause attribution inside the
original combined PR, where other heuristics changed the same files and
masked this one. Reach and sole-cause are different measurements.
805 unit + 148 integration tests pass, clippy clean, in-repo snapshots
unchanged.
* fix(tables): suppress script column evidence instead of dropping candidates
Reworked after reviewing the corpus diffs: the first approach removed
sub/superscripts from table candidates entirely, which had two failure
modes beyond the intended fix.
- Legitimate cell content was displaced. citizen-sr-282 is a calculator
manual whose engineering-notation table lists M = 10^6, k = 10^3.
Those exponents are superscripts, so they were dropped from the table
and resurfaced elsewhere in the reading order ('9 mega 6 kilo = 10 3
milli').
- Removing items changed the candidate geometry, so different spurious
structure could form from what remained.
Scripts are now kept as candidates and excluded only from the geometry:
they cannot create a column (find_column_boundaries), cannot qualify a
region on their own (find_table_regions / _strict), but are still
assigned to cells. Column alignment is validated against ALL items
including scripts — validating only the non-script subset would let a
region manufacture alignment by ignoring its awkward items, which is
what block-diagram pages did.
Corpus: 20 of 186 documents, net -359 table rows, no content lost.
Token-level comparison shows the only text changes are merges in the
right direction: 'X' + '10' becomes 'X10', 'L' + 'g' becomes 'Lg' —
subscripts joining their base instead of floating free.
Remaining artifact: MCF5235RM (and its _nxp duplicate) gains a small
spurious table from a block-diagram label line, and M2019_mordeste
gains 11 rows from a 3-column grid re-detected as 2-column. Both are
borderline regions where the previous output was also wrong; documented
rather than tuned away.
805 unit + 148 integration tests pass, clippy clean.
* fix(tables): use a heading-anchored script mask in the body-font pass
Cubic review of #264: a single script mask computed with a 0.0 anchor
was applied to both passes, including body-font region qualification and
geometry. The body pass is supposed to require a heading-sized anchor —
that distinction existed before the geometry rework and was lost in it.
Why it matters: body-pass candidates are themselves body-sized
(0.85..1.05x base). A cell at the low end of that band, say 8.5pt,
sitting beside a 10.5pt label clears the inherent 'anchor >= 1.2x cell'
rule (10.2) and so was flagged as a script attachment. At body sizes a
slightly larger neighbour is a bold label or column header, not the base
of a superscript, so flagging it stripped real cells out of the region
evidence and column geometry and could lose the table entirely.
Two masks now: the small-font pass keeps the 0.0 anchor, the body pass
requires >= 1.15x base. Note the threshold only bites below base size —
for a cell at base, 1.2x-of-cell already exceeds 1.15x-of-base — which
is exactly the 0.85..1.0x band cubic identified.
Corpus: 20 documents, net -367 table rows (was -359 with the single
mask), so the body pass now keeps 8 rows of real table it had been
discarding. 958 tests pass, clippy clean.
* fix(markdown): reject headings that end on a relational verb
A heading candidate ending in 'equals', 'denotes', 'implies' and the
like is the first half of a sentence, not a title. This shows up when a
block dissolves and strands its lead-in ahead of the formula it
introduced — opendataloader 01030000000144 produced
## Note that the exact error equals
M - Q(h) = e - 2.7525... = -0.0342....
Deliberately a very short list. Broader variants were tried and
measured, then rejected:
- Function words (of/and/for/the): a heading that WRAPS across lines
ends on exactly those. Destroyed real IRS Publication 17 headings —
'Casualty and' -> 'Casualty and Theft Losses', 'Rule 10. You Must Be
at' -> '... At Least Age 25'. 52 documents affected, -619 headings.
- Copulas and auxiliaries (is/are/be/have): same failure. 'Rule 15.
Your AGI Must Be', 'What Medical Expenses Are' and 'When Can a Roth
IRA Be' are real wrapped headings, while 'the tax burden should be'
is a genuine fragment. The trailing word cannot separate them; that
needs the next line's context, which this text-only predicate lacks.
The verbs kept never end a heading in any register, so they are safe
without context. Standalone the guard is a no-op on both benchmarks
(0 documents on opendataloader, 4 on pdf-evals with no net heading
change) — its value is as a companion to the table filter in this PR,
which is what strands these lead-ins.
Combined effect on opendataloader (200 docs, vs a control build of
main), where the table filter alone regressed:
table filter + this guard
overall -0.0003 +0.0003
mhs -0.0019 +0.0003
doc ...144 -0.063 +0.053
doc ...144 mhs -0.203 +0.028
* review: gate the dangling-verb veto on sentence case, drop 'yields'
Cubic review of a5a6e8f — both findings valid.
1. 'yields' is also a plural noun. 'Bond Yields', 'Crop Yields' and
'Dividend Yields' are real section titles in financial documents,
which this corpus contains. Removed from the list; my claim that
these verbs 'never end a heading in any register' was wrong for it.
2. A wrapped title-case heading whose first line ends on one of these
verbs would be suppressed if the heading preprocessor failed to
merge it.
Both are fixed by the same gate, which is the discriminator I was
missing: case. A heading is title case ('Bond Yields', 'The Theorem
Implies'); a stranded lead-in is sentence case ('Note that the exact
error equals', 'the method yields'). The veto now applies only when
every content word is NOT capitalized, so titles are spared regardless
of their final word.
This is also why the earlier function-word and copula variants failed:
they had no way to tell 'Rule 15. Your AGI Must Be' from 'the tax
burden should be'. Case separates those two as well.
No measured cost. opendataloader is unchanged from the previous
revision — overall +0.0003, mhs +0.0003, doc 01030000000144 still
0.732 -> 0.785 — and pdf-evals still 20 documents. 963 tests pass,
clippy clean.
* review: exempt section-numbered lines from the dangling-verb veto
Valid ordering bug. heading.rs consults is_heading_fragment at line 282
and only applies its numbered-prefix allowance at line 288, so the veto
pre-empted it: '1. What the model implies' is sentence case and ends on
a listed verb, so it was discarded before numbering could vouch for it.
Numbering is independent evidence of a heading, so the veto now skips
any line opening with a section number.
Acceptance is deliberately a little broader than heading::parse_numbering
(which requires a trailing delimiter) because '2.3 Section Title' is
written without one, and being permissive in a veto exemption can only
avoid suppressing headings. Two guards keep it from swallowing prose:
- a bare single number needs a delimiter ('1.' yes, '3 apples' no)
- roman numerals always need one, since a leading 'I' is the pronoun far
more often than a section number
Not reused from convert::starts_with_section_number, which deliberately
demands two components because it bypasses isolation checks — that would
reject the reviewer's single-'1.' case.
No measured change: opendataloader still overall +0.0003 / mhs +0.0003
with doc 01030000000144 at 0.732 -> 0.785, pdf-evals still 20 documents,
target case still suppressed. 964 tests pass, clippy clean.
* review: share roman_value so the veto exemption matches the parser
Valid. My numbering predicate accepted tokens heading::parse_numbering
rejects — lowercase 'iv)', alphabetical 'd)', over-long 'MMMM.' — because
it case-folded and allowed D and M. Anything the parser rejects is not
numbering, so exempting it let ordinary list items bypass the
dangling-verb veto and reach font-based heading promotion.
Rather than restate the grammar, roman_value is now pub(super) and the
exemption calls it, so the two cannot drift. Its rules apply as written:
uppercase I/V/X/L/C only, at most 8 characters, positive total.
Decimal numbering keeps its slightly broader acceptance (bare '2.3' with
no trailing delimiter), which is deliberate and documented — that form is
common in real headings and being permissive in a veto exemption cannot
manufacture a heading, only decline to suppress one. The roman case is
different because single letters collide with alphabetical list markers.
No measured change: opendataloader overall +0.0003 / mhs +0.0003, doc
01030000000144 still 0.732 -> 0.785. 964 tests pass, clippy clean.
* test: cover the roman length bound with a nine-character token
Valid P3. The 'MMMM.' case fails on the unsupported M, not on length, so
the 8-character bound in roman_value had no coverage and could regress
silently. Added a nine-'I' token, which is rejected only by the bound,
plus an eight-'I' token that must stay exempt to pin the boundary from
both sides.
* fix(tables): stop dropping body-band scripts from the candidate set
Valid: the body-font pass filtered scripts out of body_candidates
itself, so body_script_flags and its two downstream uses were dead. The
mask filters region_evidence and feeds detect_table_in_region's is_script
closure, but neither ever saw a script item because the candidate set no
longer contained any.
Consequences: a body-band sub/superscript attached to a heading-sized
anchor was dropped from the table outright rather than assigned to a
cell, so its text was lost — the opposite of what both the
body_script_flags comment ('they stay candidates') and the
detect_table_in_region docstring ('they remain eligible for cell
assignment') describe, and inconsistent with the small-font pass.
Root cause: the geometry rework removed the candidate-level filter from
the small-font pass, but the body one had been reflowed onto a single
line by rustfmt so the same edit missed it. Adding body_script_flags in
a later review then wired a mask that the surviving filter made
unreachable.
No measured change on either benchmark — pdf-evals still 20 documents
and -367 table rows, opendataloader still overall +0.0003 / mhs +0.0003
with one document changed — because the combination it affects (a
body-sized script attached to a heading-sized anchor) does not occur in
either corpus. The fix is for correctness and consistency between the
two passes, not for a score.
964 tests pass, clippy clean.
* fix(tounicode): skip subset GID remap for CIDFontType0 (CFF) descendants
The sequential-GID repair in try_remap_subset_cmap assumes CIDs are glyph
indices that a subsetter can renumber. That holds for CIDFontType2
(TrueType) but not for CIDFontType0 (CFF), where CIDs are resolved through
the CFF charset, so a valid ToUnicode CMap stays valid after subsetting.
For CFF fonts the corrupting path was unavoidable: CIDToGIDMap is
CIDFontType2-only (PDF 32000-1:2008, 9.7.4.2), so the branch that repairs
the CMap correctly can never be taken, and any CFF font whose /W array
starts at a low CID fell through into remap_to_sequential. Japanese
Adobe-Japan1 documents extracted as long runs of a single unrelated kanji.
Guard both repair paths on a CIDFontType2 descendant, placed before the
CIDToGIDMap branch so a CIDToGIDMap wrongly attached to a CFF font by a
malformed producer is ignored too.
On a National Diet Library proceedings PDF: 1233 U+FFFD in 69099 chars
before, 0 in 68331 after; character 3-gram recall against a hand-written
ground truth 0.354 -> 0.605. The PDF from #118 (CIDFontType2) extracts
byte-identically before and after.
Two existing tests build descendant dicts without a /Subtype and set
CIDToGIDMap, which is CIDFontType2-only, so the fixtures now say what they
already meant. Without that, test_try_remap_skipped_when_w_covers_cmap
would keep passing while no longer exercising the W-coverage logic.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tounicode): resolve indirect /Subtype, skip only explicit non-CIDFontType2
Addresses review feedback on the guard added in the previous commit.
- /Subtype may be an indirect reference, and as_name() does not dereference
it. A genuine CIDFontType2 font storing /Subtype indirectly would have been
read as "not CIDFontType2", returning early and losing the repair it needs —
reintroducing the corruption this PR fixes, for those fonts. Resolve the
reference through the document before comparing.
- Bail out only when /Subtype is explicitly a non-CIDFontType2 name. A missing
or unresolvable /Subtype now keeps the pre-existing behaviour instead of
silently disabling the repair. As a result the two existing tests no longer
need fixture changes, and this commit reverts those; the diff against main
is now additive only.
- The CFF regression test now attaches a real CIDToGIDMap stream rather than
/Identity, which get_cid_to_gid_map treats as "no map". With the stream, the
test also fails if the guard is moved back below the CIDToGIDMap branch —
verified by moving it and watching it fail.
- Added test_try_remap_resolves_indirect_subtype.
cargo fmt --check, cargo clippy -- -D warnings and cargo test (862 tests) pass.
The Diet PDF still extracts with 0 U+FFFD and the #118 PDF is still unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: recover from a corrupted startxref pointer
Fixes#228.
A PDF whose startxref pointer has been corrupted to point at the wrong
byte offset — a single flipped digit, which is what damaged writers
emit in the wild — was entirely unprocessable: every entry point
(classify_pdf, extract_pages_markdown, process_pdf) raised "Invalid
PDF structure", even though the file's object data, real xref table,
and trailer were all completely intact just past the wrong pointer.
Both pypdf and pdfium recover from this by locating the real table
directly instead of trusting the pointer; lopdf doesn't.
Added a new repair candidate (alongside the existing
missing-%%EOF-marker and stripped-leading-bytes repairs in
repair_pdf_container_candidates): scan the buffer for the real,
standalone `xref` keyword and append a corrected trailing
`startxref`/`%%EOF` block. lopdf's own get_xref_start always reads the
*last* `%%EOF` in the final 512 bytes of the buffer and the
`startxref` value immediately before it, so the appended block
transparently supersedes the corrupted one already in the file — no
in-place byte surgery on content the original writer produced.
Scoped to classic (non-stream) xref tables, matching the reported
repro and the common case; a corrupted pointer into a cross-reference
*stream* (`N 0 obj << /Type /XRef ...>>`, some PDF 1.5+ writers) would
need the containing object's number, not just a byte offset — out of
scope here.
Verified against the issue's exact repro (a valid one-page PDF with a
single corrupted byte in its startxref offset): before this fix,
process_pdf/classify_pdf/extract_pages_markdown all raised "Invalid
PDF structure"; after, both the page count and the real extracted text
("Order Detail Report by Account", "WIDGET ASSEMBLY", the dollar
amount) come back correctly. New regression test added.
Full suite (859 tests, 1 new) passes; cargo clippy --all-targets
-- -D warnings unchanged at 28 pre-existing/unrelated errors.
* fix: validate xref table shape and scan in a single reverse pass
Addresses cubic-dev-ai's review of #230.
- P2 (correctness/safety): the recovery candidate trusted the last
standalone "xref" token unconditionally, without confirming it's
actually a cross-reference table. A coincidental "xref" substring
inside unrelated content — a stream, a string, uncompressed
metadata — could get "repaired" against a bogus offset, letting
lopdf load successfully against garbage instead of returning a
clean error: a real failure turned into silent data corruption on
the fallback path. Added looks_like_xref_subsection_header, which
confirms a plausible classic xref subsection header (`<start-id>
<count>`, e.g. "0 6" — the shape every real classic table starts
with) actually follows the candidate token before accepting it.
find_last_valid_xref_table_start now walks backward from the end of
the buffer until it finds a token that both stands alone *and*
validates, rather than accepting the first (rightmost) standalone
match unconditionally.
- P2 (performance): the old scan re-invoked
`buf[..search_end].windows(4).rposition(...)` on a shrinking prefix
every time a candidate token failed the boundary check, which is
quadratic on a pathological buffer with many non-standalone "xref"
occurrences. Rewrote as a single reverse byte-index walk — O(n)
regardless of how many false candidates it has to reject along the
way.
Added direct unit tests on the byte-level scan (more precise than
constructing adversarial full PDFs, and the coincidental-match
scenario can't be represented in an integration-test fixture anyway
since reportlab compresses page content by default): a coincidental
standalone "xref" with no subsection header is rejected; a real
classic table is found; a coincidental match positioned *after* the
real table in the buffer doesn't shadow it; "xref" as a substring of
"startxref" still doesn't match. The original #228 repro (corrupted
startxref pointer, real table otherwise intact) is unaffected —
verified manually in addition to the existing integration test.
Full suite (863 tests, 5 new) passes; cargo clippy --all-targets
-- -D warnings unchanged at 28 pre-existing/unrelated errors.
* fix: reject xref subsection count runs with trailing garbage
looks_like_xref_subsection_header validated that a count run of digits
followed the whitespace separator, but never checked what came after
it. A coincidental "xref\n0 6garbage" in stream/literal content would
still validate as a real subsection header shape and get repaired
against a bogus offset.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Abimael Martell <1450169+abimaelmartell@users.noreply.github.com>
* fix: extract_pages_markdown's needs_ocr now agrees with classify_pdf
Fixes#227.
extract_pages_markdown_mem computed its per-page needs_ocr entirely
from text-quality signals: decoding/garble issues, empty markdown, GID
fonts, garbage-text ratio. It had no awareness of the page's image
content at all — so a page that is fundamentally a full-page scan
with a little genuine native text drawn over it (a header, a stamp, a
cover-sheet annotation) extracts that text cleanly, trips none of the
text-quality checks, and reports needs_ocr=false — while
classify_pdf/detect_pdf_type correctly see the dominant background
image and flag the same page as needing OCR. Two public APIs
answering the same question, silently disagreeing, in the unsafe
direction (skipping OCR on a page that needs it).
Exposed detector::analyze_page_images at crate visibility (was
private) and call it per page in extract_pages_markdown_mem's loop —
the same "large background image" signal (>50% page coverage) that
already powers has_template_image in classify_pdf/detect_pdf_type,
rather than reimplementing image-area detection a second time with
its own thresholds that could drift out of sync again. When it's
true, the page is flagged needs_ocr (with OCR_REASON_SCANNED added
to ocr_reasons_by_page, matching how the same signal is already
reported elsewhere) and its markdown is blanked, exactly like the
existing text-quality-triggered needs_ocr paths already do — no
special-casing added for "cleanly-extracted-but-still-a-scan" text.
Verified against the issue's exact repro (a full-page raster with one
native text line drawn over it, built via reportlab/pillow): before
this fix, extract_pages_markdown_bytes reported page 0
needs_ocr=False with the header line as markdown while
classify_pdf_bytes correctly flagged pages_needing_ocr=[0]; after,
both agree needs_ocr=True and the page's markdown is empty. Confirmed
no regression on a normal text-based fixture (nexo-price-en.pdf:
needs_ocr stays False, full markdown returned). New Rust regression
test added exercising both APIs against the same fixture.
Full suite (860 tests, 1 new) passes; cargo clippy --all-targets
-- -D warnings unchanged at 28 pre-existing/unrelated errors.
* fix: gate has_template_image behind the same OCR signals classify_pdf uses
extract_pages_markdown_mem was treating has_template_image alone as
sufficient to force needs_ocr=true and discard the page's markdown, but
classify_pdf/detect_pdf_type never treats that raw signal alone as
needing OCR. A text page with a full-bleed watermark, letterhead, or
large figure would get its clean markdown wrongly blanked and routed
to OCR.
Added page_template_image_needs_ocr(), mirroring the two distinct
signals classify_pdf actually uses to decide a template-image page
needs OCR: the looks_like_scan gate (image_count <= 1, few text ops,
low alphanumeric diversity) used for Mixed-type routing, and the
insufficient-text-volume signal (text_operator_count < 10) that routes
a page with a dominant background image and only a couple of native
text calls to PdfType::ImageBased independent of looks_like_scan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix: match per-page OCR threshold and add missing vector-text signal
Two follow-up findings on the has_template_image gate added in the
previous commit:
1. insufficient_text used a hard-coded threshold of 10 text operators,
but Mixed-type per-page routing (the actual per-page decision this
function tries to agree with) uses config.min_text_ops_per_page
(default 3). The higher 10 threshold was borrowed from a *different*
classify_pdf code path — the effective_min_ops floor used only for
whole-document ImageBased/Scanned classification, a cross-page
aggregate this per-page function can't replicate anyway. Using the
lower per-page threshold removes a real disagreement window
(3-9 text ops with high alphanumeric diversity) without breaking
the #227 regression fixture (text_ops=1, still well under 3).
2. extract_pages_markdown_mem never checked has_vector_text at all,
even though Mixed-type per-page routing always sends
vector-outlined-text pages to OCR (outlined glyphs can't be
extracted as text). A page with massive path ops plus a short
genuine caption could extract that caption cleanly, slipping past
the existing empty/garbage-text checks. Added
page_has_vector_text() and wired it into needs_ocr the same way
has_template_image is.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* perf: compute template-image and vector-text OCR signals in one pass
page_template_image_needs_ocr and page_has_vector_text each called
analyze_page_content independently, so every requested page's content
streams (page + XObjects) and image coverage were decompressed and
scanned twice per page with one result discarded each time.
detect_from_document avoids this by caching its per-page PageAnalysis;
extract_pages_markdown_mem had no such cache.
Merged both into page_ocr_signals(), a single analyze_page_content
call returning both signals as a tuple.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Abimael Martell <1450169+abimaelmartell@users.noreply.github.com>
* fix(extractor): supply built-in metrics for non-embedded base-14 fonts
PDFs may legally omit /Widths for non-embedded standard fonts (Times,
Helvetica, Courier, Symbol, ZapfDingbats) — the spec requires the reader
to supply the metrics. We returned None, so every glyph advanced 0 and
each text item got width 0, silently breaking every gap-based heuristic
downstream: space synthesis, sub/superscript detection, table column
detection, heading merging.
- src/extractor/base14.rs: Adobe Core-14 AFM width tables keyed by
Unicode char, plus the standard Symbol/ZapfDingbats encoding vectors
(their glyphs sit at byte positions unrelated to Latin text, so widths
must resolve through the built-in encoding, not cp1252)
- Width resolution order: Differences -> built-in encoding -> the same
cp1252-style fallback the text decoder uses, so a code's advance always
matches the character we emit for it
- Type3 visual sizing: PK bitmap fonts (dvips) use FontMatrix
[1 0 0 -1 0 0] with nominal sizes like 0.12pt; scale by FontBBox height
x |matrix_y|. Applied in the page-stream and Form XObject paths.
Indirect numeric array elements are resolved before use.
Effect on Shannon's 'A Mathematical Theory of Communication' (1998
dvips/Distiller, the reported case): glued sentences 95 -> 5. Corpus
impact: 12 of 184 eval documents, e.g. Data-Processing-Agreement
recovers a paragraph that a phantom table had shredded into cells.
Layout heuristics tuned on the same document (indent-based paragraph
breaks, heading reclassification, table script filtering) are held back
for a separate PR — they change ~98 further documents and need to be
justified against the corpus, not against one PDF.
* review: narrow Type3 rescaling to self-inconsistent fonts; dedup + test all width tables
Addresses cubic review on #241, plus a follow-up from a local cubic run.
- Type3 visual scaling was applied to every Type3 font whose FontBBox
height x |matrix_y| deviated >5% from 1.0. FontBBox is the glyph box,
not the em box, so a conventional 1/1000-matrix font with a
descender..ascender bbox (~700 units) computed 0.7 and had every
reported size shrunk by 30% — corrupting the drop-cap, heading-tier,
sub/superscript and table heuristics this is meant to fix.
First attempt gated on the matrix being unit-scale, but a local cubic
run pointed out that wrongly excludes valid non-standard matrices (a
0.005 matrix with a full-em bbox legitimately needs a 5x scale). The
product is the right discriminator, not the matrix: a self-consistent
font lands near 1.0 because the matrix is the reciprocal of the
glyph-space em, so only a wildly inconsistent one (dvips/PK bitmap
fonts sit at ~159) is renormalized. Band widened to [0.25, 4.0].
Corpus effect: 12 -> 7 documents change. The 5 that drop out were
being wrongly rescaled — including Data-Processing-Agreement, whose
phantom-table fix turned out to come from this bug rather than from
the width fallback, so it is correctly given up.
- base14: all 14 width tables now covered by the sort-invariant test via
an ALL_TABLES registry, not a hand-picked subset.
- base14: identical tables share one static (all four Courier variants
are monospace 600; the oblique Helvetica variants match their upright
forms), removing 5 duplicate copies.
* test: refresh Shannon snapshot after merging main
CI checks out a merge of the PR head with main, and main advanced 8
commits since this branch was cut — including #201 (contextual digit
runs), #240 and #253 (markdown fixes). Those change extraction output,
so a snapshot generated on the unmerged branch could not match; the
Test job failed on the merge commit while passing on the branch itself.
The merged behaviour is better: the footnote marker '2' before
'Hartley, R. V. L.' is now recovered instead of dropped.
950 tests pass on the merged tree, clippy clean.
AGENTS.md was stale (179+ PDFs, missing the semantic-quality bullet).
Both files now match: ~200-PDF corpus, and iteration guidance to prefer
subset runs (bench.py test -q / -s <name>) with the full suite as the
final pre-commit check.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
strip_pdf_comments tracked parenthesis nesting to protect string
literals, but ignored backslash escapes. An escaped \) desynced the
depth counter, after which a % glyph inside a string was stripped as a
top-level comment, corrupting the stream for Content::decode and
silently truncating the page's text.
Treat \ inside a string literal as escaping the next byte, so \(,
\), and \\ never touch the nesting depth.
## Bug 1: Python type stubs missing 5 fields/classes
The pdf_inspector.pyi file was out of sync with the actual Python
bindings exposed via #[pyo3(get)] in src/python.rs. This breaks
IDE autocompletion and type checking (PyCharm, VS Code/Pylance, mypy)
for all Python users.
Added:
- PdfResult.ocr_reasons_by_page (python.rs:35)
- PageOcrReasons class with page and
easons fields (python.rs:67-86)
- RegionText.ocr_reason (python.rs:136)
- PageMarkdown.ocr_reason (python.rs:193)
- PagesExtractionResult.ocr_reasons_by_page (python.rs:226)
## Bug 2: PdfResult.pages_needing_ocr indexing undocumented
PdfResult.pages_needing_ocr is 1-indexed (per python.rs:30) but
neither the .pyi stubs nor docs/python.md annotated this, while the
same field on PdfClassification was annotated as 0-indexed. Users
mixing both APIs would get wrong page numbers.
## Bug 3: README.md duplicate bullet character
The Markdown features table listed * twice in bullet prefixes.
The first should be ullet (U+2022), matching the actual source code in
src/markdown/mod.rs:34 which uses: ullet, dash, sterisk, circle, illed-circle, open-circle.
## Bug 4: docs/python.md missing fields in type reference
The Types section was missing PageOcrReasons, RegionText class
definition, ocr_reason fields, and ocr_reasons_by_page fields.
## Evidence
Cross-referenced every #[pyo3(get)] attribute in src/python.rs
against the .pyi declarations and docs/python.md type reference.
* chore(ci): bump GitHub Actions to current majors
Node 20 action runtimes are deprecated on GitHub runners; bump every
first-party action to its latest major across all workflows:
- actions/checkout v4/v6 -> v7
- actions/cache v4 -> v6
- actions/upload-artifact v4 -> v7, download-artifact v4 -> v8
- actions/setup-node v6 -> v7, setup-python v5 -> v7
- actions/upload-pages-artifact v3 -> v5, deploy-pages v4 -> v5
Third-party pins (dtolnay/rust-toolchain, Swatinem/rust-cache,
setup-zig, setup-bun, taiki-e/install-action, maturin-action) are
already on their latest majors.
* chore(ci): pin all actions to full commit SHAs
Mutable @vN tags can be retagged; in the publish workflows that code
runs with OIDC credentials before npm/PyPI/crates.io publishes. Pin
every action (first- and third-party) to its release commit SHA with
the version in a trailing comment.
dtolnay/rust-toolchain infers the toolchain from its ref name, so the
SHA-pinned invocations pass an explicit toolchain: stable input.
Adds x86_64-unknown-linux-musl, aarch64-unknown-linux-gnu, and
aarch64-unknown-linux-musl to the napi build targets so
@firecrawl/pdf-inspector works on Alpine and ARM64 Linux deployments.
- gnu arm64 cross-compiles with --use-napi-cross (old-glibc sysroot),
musl targets with -x (zig + cargo-zigbuild), per the napi-rs template
- new platform packages carry npm libc metadata (glibc/musl)
- smoke-test job runs napi/test.mjs on all six targets before publish
(Alpine containers for musl, ubuntu-24.04-arm runners for ARM64)
- bump to 1.12.0 to trigger publishing of all platform packages
Closes#216
* fix(extractor): don't flag gid Differences names covered by ToUnicode
Pages were marked as having unresolvable gid-encoded fonts whenever any
font's /Differences array used gidNNNN glyph names, and when every page
carried such a font the whole document's markdown was suppressed.
LibreOffice exports do exactly this: subset fonts get /gidNNNN names in
Differences alongside a complete ToUnicode CMap that decodes them, so
ordinary text documents lost their entire markdown output even though
extraction decoded every glyph.
Track the character codes behind the gid names and only flag the font
when its ToUnicode CMap addresses none of them. Partially mapped codes
stay unflagged: an emoji ZWJ sequence maps whole on its first code, and
the remaining component-glyph codes are subset leftovers, not damage.
Fonts without ToUnicode, or whose CMap ignores the gid codes, are
flagged as before, and the downstream garbage/encoding checks still
catch partial breakage.
* fix(extractor): require a usable ToUnicode mapping to clear the gid flag
A mapping to U+FFFD (or an empty string) is rejected by extraction as
an invalid CMap result, so it must not count as decodable when deciding
whether gid-named Differences codes are resolvable.
