051253329f55474e407de197e16c60ec0a885a59
449
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Abimael Martell
|
051253329f |
fix(markdown): skip zero-span pages when collecting furniture keys
Review finding on #374: a page whose extracted text has no vertical span (a single line) produced a zero edge band, which classified its every item as page-edge furniture — a repeated one-row strip on three such pages could be vetoed. A zero-span page gives no evidence of where its edges are, so it now contributes no furniture keys. Also exposed that running_furniture_requires_three_pages_at_same_position was silently relying on the quirk (its pages contained only footer items); the tests now give every page real body extent, and a new test pins the corrected semantics. Corpus A/B is byte-identical to the previous revision on all 6 differing files — zero-span pages cannot produce multi-row table candidates in practice, so the guard is purely defensive. |
||
|
Abimael Martell
|
2c24e4c62c |
fix(markdown): restrict running-furniture veto to page-edge bands
Review finding on #374: repetition alone can misclassify a repeated form template — identical labels at identical mid-page coordinates on every page are real table cells, not furniture. What makes a header/footer is repetition *at the page edge*. Furniture keys are now restricted to the top/bottom 20% of each page's vertical content extent (computed from the items, so the bands adapt to real margins rather than assuming a media box). Mid-page repetition never qualifies, however many pages it spans — unit-tested with the repeated-form scenario from the review. Corpus effect is essentially unchanged (the three remaining header-fragment files are continued tables whose wrapped headers restart at the top of each page, inside the band): 6 of 203 still differ, five byte-identical to the previous revision, MTU 52 bytes smaller as the band spares a few mid-page candidates. |
||
|
Abimael Martell
|
993d2a865d |
fix(markdown): veto heuristic tables made of running headers/footers
Running headers and footers repeat verbatim at the same position on many pages. When such a block wraps a long title or navigation strip over aligned lines, the heuristic table detector reads it as a grid — the PA_PVEM running footer regression documented on #371, where the small-caps merge made the footer lines contiguous enough to detect. Page furniture is a document-wide property, so the veto is computed once in to_markdown_from_items_with_rects_and_lines rather than inside the per-page detector: an item is running furniture when its trimmed text appears at the same position (quantized to 0.5pt) on >= 3 distinct pages, and a heuristic table candidate is rejected when >= 80% of its items are furniture. Items are not deleted — the text flows as prose, exactly as it did before the candidate was accepted. Rect- and line-based tables are untouched: ruled structure is stronger evidence than repetition. Real tables keep their per-page content under the threshold even when their header row repeats on every page (continued tables), because the body rows differ. Header-only fragments of such tables *can* be vetoed when the body was claimed elsewhere — in the corpus that trades one garbled rendering for another (see below). A/B over all 203 eval PDFs, 197 byte-identical, 6 differ: PA_PVEM the #371 regression: running footer is bold prose again instead of a 4-column table Remiss-2026-00331 Swedish agency letterhead repeated per page — was a 2x2 table on every page, now prose MTUAeroEngines navigation strip (01|02|03|04|05 + section labels) repeated on ~250 pages — was a 3-row table every page, now compact prose (-71.5KB of repeated markup) KIT-ENGLISH repeated wrapped column-header fragment of a cross-page creditor table; was already a garbled header-only table, now flows as prose (-420 bytes) anthropic-mythos same pattern on a system-card table header (-11 bytes) ytec-agm same pattern on a director-shareholding table header (-561 bytes) The last three were garbled either way; the first three are the point. 199AD3d.pdf (the volume that motivated this series) is byte-identical: its running headers never formed heuristic tables and its real tables carry per-page content. Adds unit tests for the key builder (3-page minimum, position drift, short documents) and the veto (footer-only candidate rejected, real table with a repeating header row kept). |
||
|
Abimael Martell
|
ec6e54afb8 |
fix(extractor): merge small-caps runs so they stop reading as table columns (#371)
* fix(xobjects): track text line matrix and handle T*/TL/'/"/Tc/Tw in Form XObjects The Form XObject text extractor in xobjects.rs is a separate hand-rolled implementation of the operator state machine in content_stream.rs, and it had drifted well out of parity: - No text line matrix (TLM). `Td`/`TD` were applied to the text matrix already advanced by `Tj`/`TJ`, so every line began where the previous line *ended* instead of at the line start. Lines marched off the right edge and were dropped as off-page. - `T*` was not handled at all, so it never advanced to the next line. - `TL`, `'` and `"` were missing, and `TD` never set the leading as a side effect. - `Tc`/`Tw` were hardcoded to 0.0 when computing advance widths, drifting positions and inserting spurious spaces. - Text state (Tc/Tw/TL/Tf) is part of the graphics state but was not saved or restored by `q`/`Q`. This matters well beyond an edge case: producers that emit a page stream of just `q /X Do Q` and put all content in a Form XObject are common in print-to-PDF and typesetting workflows, so this parser is on the hot path for whole classes of real documents. Measured on nycourts.gov 199AD3d.pdf (1370 pages, PDFlib producer, every page wrapped in a Form XObject, 1331 pages using T*), word recall against a pdftotext reference goes from 19.2% to 97.7% — 116k extracted words to 582k against a 578k-word reference. On a 10-page subset, sequence similarity goes from 27.3% to 97.7%, against 99.8% for Mistral OCR. pdf-evals: 195 passed / 7 failed, byte-identical to the origin/main baseline with the same failure list — no regressions. Adds 7 unit tests covering the line-matrix-relative `Td`, `T*`, `TD` setting leading, `'`, `"`, `Tc` advance widths, and `q`/`Q` text-state restore, all driven through a page whose content is only `q /X1 Do Q`. * fix(xobjects): restore fill colour across q/Q in Form XObjects A white fill set inside a q/Q pair leaked past the Q, so all subsequent text was treated as invisible and dropped. Save and restore fill_is_white with the rest of the graphics state. This was the cause of several long-standing extraction failures where whole passages went missing or degraded into per-character garbage: cambridge_excerpt (+8.5KB of recovered text), MTUAeroEngines (+7.4KB), 2025_findings-acl_668 (+1.4KB), HTM_02-01_Part_A (+1.3KB), and HuttoISDWorkPerks / ebgt7isj04ophcq, which both went from exploded per-character tables to clean prose. Adds a regression test that fails without the restore (the text after Q is dropped entirely). Reported by cubic on #369. * fix(extractor): merge small-caps runs so they stop reading as table columns Typesetters render small caps as a full-size capital immediately followed by shrunken capitals in the same font — `(R) Tj` at 9.98pt, then `(OLANDO) Tj` at 6.74pt, touching. The 20% font-size band in merge_text_items split those into separate items, and since table column boundaries cluster on item *start* positions (find_column_boundaries never consults widths), each fragment started far enough from the last to become its own column. The result was garbled pseudo-tables. From 199AD3d.pdf p.5: |OLANDO|COSTA|NIL|INGH|| |---|---|---|---|---| |IANNE|ENWICK|ETER|OULTON|| |R|T. A|, P.J.|A|C. S| |---|---|---|---|---| now: |ROLANDO T. ACOSTA, P.J.|ANIL C. SINGH| |---|---| |DIANNE T. RENWICK|PETER H. MOULTON| Adds is_small_caps_continuation, gated tightly enough to exclude the other reasons a smaller run follows a larger one: superscripts and footnote markers (requires an uppercase *letter*, so digits never qualify), drop caps (the following body text is mixed case), and adjacent cells or separate words (requires the runs to be visually contiguous). A small-caps junction is mid-word, so it also suppresses the space that would otherwise be inserted ("T. A" + "COSTA" -> "T. ACOSTA", not "T. A COSTA"). On 199AD3d.pdf this drops false-positive tables from 65 to 52 and lifts word recall against a pdftotext reference from 97.7% to 97.8%. Verified by running both binaries over all 203 eval PDFs and diffing outputs: 19 differ, 184 byte-identical. Beyond the reporter volume the merge also fixes: Waters-Edge two more garbled heading-tables — "##### B. C R S" plus "||OMPLIANCE|EPORTING YSTEM|" became "##### B. COMPLIANCE REPORTING SYSTEM" cn-student-handbook six TOC entries had collapsed to their initials; "U M S H..." is now "UNIVERSITY MISSION AND STUDENT HANDBOOK..." 546403 "(FAA)" + "ADVISORY CIRCULARS ( )" + a stray "CONT" became "(FAA) ADVISORY CIRCULARS (CONT)" ERP-2025 "T ABLE B-1" -> "TABLE B-1" DMP-Keypad "THINLINE" + stray "TM KEYPADS" -> "THINLINETM KEYPADS" zhaw / Stijn subscripted math variables: "*T* *G*" -> "*TG*" One known regression, called out rather than hidden: PA_PVEM_Sen_Waldo_Fernández (+1689 bytes). Its running footer genuinely is small caps, so the merge correctly assembles the text (400 items -> 275), but the now-contiguous footer lines align well enough that the heuristic detector turns the wrapped document title into a 4-column table where it previously rendered as bold prose. Attempting to fix that in the detector was a dead end and is not included here: gating the `num_cols >= 3` bypass in has_table_like_content on "no cell has >= 12 words" removed 143 tables across 44 files, including correct ones — MCF5235RM went 505 -> 483 and its register glossary degraded into a fused column plus a ~100-word cell, because rejecting a good candidate lets a worse fallback win. No wordiness threshold separates the two cases: PA_PVEM's longest cell is 14 words while MCF5235RM's legitimate cells are <= 10. A positional signal (running-header/footer bands, or cross-page repetition) is the way in, and belongs in its own change. Adds 7 unit tests: the two-column small-caps row from the reporter volume, and rejection of superscript digits, drop caps, separate words, lowercase continuations, and out-of-band size ratios. * fix(extractor): tighten small-caps gate to cross-band junctions only Three findings from cubic on #371, all valid. 1. Space suppression was too broad. `is_small_caps_continuation` accepted size ratios up to 0.92, which is *inside* the 20% band that merge_text_items already treats as the same size. Two similarly-sized uppercase words with a small real word gap therefore had their space suppressed even though the normal path would have merged them correctly with a space ("SEE" + "ALSO" -> "SEEALSO"). The helper now requires the junction to *cross* the band, since rescuing junctions the band would break is its only purpose; within-band pairs keep the normal word-spacing logic. The band is now a shared MERGE_FONT_SIZE_BAND constant so the helper and its caller cannot drift. 2. A trailing digit was skipped. The backward search for "the capital we are continuing" skipped non-alphabetic characters, so text ending in a footnote marker ("ANGELA M. MAZZARELLI1") found the earlier uppercase letter and accepted the join. It now checks the actual trailing character. Rejecting every trailing digit outright turned out to cost real quality, so this keeps one narrow exception. In meetings_in_mass_july_2023 the source reads "TUESDAY, JULY 4TH" with the ordinal suffix set as a smaller run; a blanket digit rejection reverted that to "TUESDAY, JULY 4" plus a stray "TH" leaking onto the next line, which is what main produces and what pdftotext shows is wrong. Only the four English ordinal suffixes (TH/ST/ND/RD) may follow a digit; anything else after one is treated as a footnote marker and rejected, so the case cubic raised stays blocked. 3. A test's name did not match its data. `small_caps_merge_does_not_swallow_a_ second_column` claimed to exercise the 72pt column gap but contained only the second column's items, so it just re-tested the happy-path merge. It now holds the full nine-item row and asserts exactly two merged results, "ROLANDO T. ACOSTA, P.J." and "ANIL C. SINGH", which genuinely exercises the gap. Both new guards were verified to be load-bearing: removing either one makes its test fail. The document that motivated the change is unaffected — small caps there run at a 0.675 ratio, far outside the band — and 199AD3d.pdf p.5 still produces the correct two-column justices table. 900 unit tests pass (10 covering small caps), fmt and clippy clean. |
||
|
Abimael Martell
|
89dd20d02c |
fix(xobjects): track text line matrix and handle T*/TL/'/"/Tc/Tw in Form XObjects (#369)
* fix(xobjects): track text line matrix and handle T*/TL/'/"/Tc/Tw in Form XObjects The Form XObject text extractor in xobjects.rs is a separate hand-rolled implementation of the operator state machine in content_stream.rs, and it had drifted well out of parity: - No text line matrix (TLM). `Td`/`TD` were applied to the text matrix already advanced by `Tj`/`TJ`, so every line began where the previous line *ended* instead of at the line start. Lines marched off the right edge and were dropped as off-page. - `T*` was not handled at all, so it never advanced to the next line. - `TL`, `'` and `"` were missing, and `TD` never set the leading as a side effect. - `Tc`/`Tw` were hardcoded to 0.0 when computing advance widths, drifting positions and inserting spurious spaces. - Text state (Tc/Tw/TL/Tf) is part of the graphics state but was not saved or restored by `q`/`Q`. This matters well beyond an edge case: producers that emit a page stream of just `q /X Do Q` and put all content in a Form XObject are common in print-to-PDF and typesetting workflows, so this parser is on the hot path for whole classes of real documents. Measured on nycourts.gov 199AD3d.pdf (1370 pages, PDFlib producer, every page wrapped in a Form XObject, 1331 pages using T*), word recall against a pdftotext reference goes from 19.2% to 97.7% — 116k extracted words to 582k against a 578k-word reference. On a 10-page subset, sequence similarity goes from 27.3% to 97.7%, against 99.8% for Mistral OCR. pdf-evals: 195 passed / 7 failed, byte-identical to the origin/main baseline with the same failure list — no regressions. Adds 7 unit tests covering the line-matrix-relative `Td`, `T*`, `TD` setting leading, `'`, `"`, `Tc` advance widths, and `q`/`Q` text-state restore, all driven through a page whose content is only `q /X1 Do Q`. * fix(xobjects): restore fill colour across q/Q in Form XObjects A white fill set inside a q/Q pair leaked past the Q, so all subsequent text was treated as invisible and dropped. Save and restore fill_is_white with the rest of the graphics state. This was the cause of several long-standing extraction failures where whole passages went missing or degraded into per-character garbage: cambridge_excerpt (+8.5KB of recovered text), MTUAeroEngines (+7.4KB), 2025_findings-acl_668 (+1.4KB), HTM_02-01_Part_A (+1.3KB), and HuttoISDWorkPerks / ebgt7isj04ophcq, which both went from exploded per-character tables to clean prose. Adds a regression test that fails without the restore (the text after Q is dropped entirely). Reported by cubic on #369. |
||
|
Abimael MartellandCursor
|
3d33ff3dbd |
fix(extractor): bound CID /W range expansion (#372)
Type0 /W parsing and the Unicode-CID heuristic expanded every CID in every range. Repeating a full-width [0 65535 w] entry therefore re-materialized the same 65,536-key domain on every copy, growing a temporary vector and HashMap work without bound. Cap expansion at the 16-bit CID domain, collect unique CIDs for the median heuristic, and stop width assignment once that many entries have been written. Legitimate compact /W arrays are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Abimael MartellandCursor
|
75e9b09593 |
fix(extractor): bound Form XObject expansion per page (#370)
* fix(extractor): bound Form XObject expansion with invocation and operation budgets Nested Form XObjects were only limited by recursion depth (5). An acyclic graph where each form invokes the next N times still expands to N^depth work, so a small PDF can force millions of nested /Do evaluations. Share a per-page FormWalkBudget that caps 10,000 Form invocations and 1,000,000 operations walked across those expansions. Extraction stops when either cap is hit. Legitimate shallow nesting is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(extractor): share Form XObject budget across invisible-layer retry extract_page_text_items created a fresh FormWalkBudget on every call, so the invisible-text retry could consume a second full expansion budget for the same page. Own the budget at the call site and pass it into both passes. Also charge form operations independently of the invocation cap so a form that was already admitted can finish its stream. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Abimael MartellandClaude Fable 5
|
f4aab3b36f |
chore(release): bump package versions to 1.14.1 (#354)
Releases fix(regions) #351 — invisible (Tr 3) OCR text layers served from the region extractor instead of falling back to GPU OCR. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Abimael MartellandClaude Fable 5
|
f65b25906c |
fix(regions): serve invisible (Tr 3) OCR text layers instead of needs_ocr (#351)
* fix(regions): serve invisible (Tr 3) OCR text layers instead of needs_ocr Scanned pages (archive.org-style digitizations) carry their text as an invisible render-mode-3 layer behind the page raster. extract_text_in_regions extracted only visible items, so such pages yielded nothing but '[Image: ...]' placeholders and every region fell back to OCR — while the markdown path already includes the invisible layer for Mixed PDFs. The two extractors disagreed about the same page. Mirror the markdown path's gate, page-scoped: when the visible pass is effectively textless (<40 non-placeholder alphanumerics), retry with invisible text included and adopt the retry only when it contributes real, non-garbage text. Pages with real visible text never retry, so double-layer PDFs (visible text plus an invisible accessibility copy) are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: strict zero-visible adoption gate, whole-layer garbage check, doc placement, discriminating tests - Adoption now requires ZERO visible text on the page: the invisible pass returns visible items too, so adopting alongside any visible text would duplicate it. Strict gate instead of fuzzy dedupe; the dead inv_alnum > visible_alnum condition goes with it. - Garbage check judges the whole recovered layer, not the first 200 items. - Constant/helper moved above the doc block so rustdoc stays attached to extract_text_in_regions_mem. - Tests: any-visible-text-blocks-adoption case (single short visible line) with exactly-once assertions; mode-0 guard asserts occurrence count. - Re-verified against real scanned-book pages: all recover (9.2-11.7K chars, needs_ocr=false) — pure OCR-layer scans carry no visible text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: gate adoption on visible-item presence, not alphanumeric mass Punctuation-only visible text (zero alphanumerics) could still adopt the invisible pass; its OCR twin in the layer would duplicate the glyphs. The gate is now item-presence: any non-image item with non-whitespace text blocks adoption (whitespace-only artifacts still tolerated). Pinned by a punctuation-only fixture test; real scanned-book pages re-verified — all recover unchanged (they carry no visible items at all). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: punctuation-gate test pins preservation, comment states fixture scope The fixture's visible punctuation is a separate block, not mirrored in the invisible layer — so it pins the gate, not the duplication scenario. The doc comment now says so, and a positive assertion checks the punctuation survives exactly once. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: retry only when invisible text was actually skipped; negative gate tests - extract_page_text_items now reports skipped_invisible (4th return): the Tr-3 suppression sites set it, so the region extractor retries only when a recoverable layer exists. Blank pages and image-only scans without an OCR layer — the common scanned case — no longer pay a second content-stream parse. - Negative tests: below-floor watermark layer and symbol-garbage layer are both rejected (region keeps only the raster placeholder). needs_ocr semantics for placeholder-only regions deliberately unchanged — that contract predates this PR and downstream pipelines handle it. - Real scanned-book pages re-verified: recovery unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: flag the ' show-text suppression path; require nonempty strings - P1: invisible layers shown via the ' operator never set skipped_invisible, so those pages kept the placeholder and fell back to GPU OCR. The ' suppression path now flags too — pinned by a fixture whose layer is shown entirely via ' (nothing through Tj/TJ). - P3: all three flag sites (Tj, TJ, ') require a nonempty string operand — numeric-only TJ kerning arrays and empty shows no longer trigger the invisible reparse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Abimael Martell
|
9947485a92 | docs: document structure-element extraction and TextItem.mcid (#349) | ||
|
Abimael Martell
|
7054d6aa69 |
chore(release): unify package versions (#344)
* chore(release): unify package versions * fix(release): harden version synchronization |
||
|
Abimael MartellandClaude Fable 5
|
a67ee03269 |
feat(bindings): expose TextItem.mcid and structure-tree element extraction (#346)
Tagged PDFs carry a structure tree with real heading roles (H1..H6), and the core already parses it (structure_tree::StructTree) and threads MCIDs onto TextItem — but neither surfaced through the bindings. - Expose TextItem.mcid (Option<i64>) through the napi and pyo3 bindings, matching the core field added with the marked-content extractor. - Add StructRole::name(), the inverse of from_name, so roles have a stable string form. - Add extract_structure_elements / extract_structure_elements_mem to the core: one (page, mcid, role) entry per marked-content reference, sorted by (page, mcid), empty for untagged PDFs. Pages are 1-indexed to match TextItem.page, so results join directly against extract_text_with_positions output. - Bind it as extractStructureElements (napi) and extract_structure_elements / extract_structure_elements_bytes (pyo3), with type-stub updates in pdf_inspector.pyi. - Cover the join in Rust integration tests, napi test.mjs, and pytest, using the existing firecrawl_docs_tagged.pdf fixture (tagged) and thermo-freon12.pdf (untagged). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Abimael Martell
|
965dc65f1b | chore(release): bump package versions (#343) packages-2026-08-10 | ||
|
|
36dd5fa426 |
fix(structure-tree): bound recursive /K parsing with cycle detection and a node budget (#322)
* fix(structure-tree): bound tagged /K parsing against alias/cycle DoS A struct element that references itself (or an ancestor) through /K — e.g. /K [n 0 R n 0 R] — made parse_struct_element_dict branch exponentially: the depth cap (64) alone still permits 2^depth materialized nodes, so a ~830-byte PDF exhausts memory (OOM, exit 134). Add a StructWalk carrying (1) an active-path set of object IDs so a node that references itself/an ancestor is not re-expanded (breaks self- and mutual-reference cycles cheaply), and (2) a global node budget (MAX_STRUCT_NODES) that caps total materialization for aliased/DAG-shaped graphs of distinct objects the path guard cannot catch. Adds regression tests for self-alias, mutual-alias, and the aliased-DAG budget cap. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): charge /K content refs against the node budget The per-node budget only covered materialized struct elements and child recursion; bare MCIDs and MCR dicts in a /K array append to content_refs without charging it, so one element with a very wide /K array could still allocate content_refs without bound. Charge every /K array item before handling it, and stop the top-level /K loop once the budget is spent, so content refs and loop work are bounded too. Adds a wide-MCID-array test. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): charge /K budget per materialized item, not per array entry Charging every /K array item double-counted structural children (charged here and again at their node entry) and charged cycle-skipped references that materialize nothing, draining the budget up to ~2x faster than the per-node semantics and risking early truncation of large legitimate trees. Charge only the unbounded content-ref items (bare MCIDs and MCR dicts); structural children remain charged once at their node entry. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * refactor(structure-tree): charge every content ref uniformly via helper Route all budget charges through StructWalk::charge() so every marked-content reference is charged once, including the single-value /K branches (bare integer and MCR dict) that previously appended without charging. charge() also guards against underflow, so charging after the node-entry charge (which can leave the budget at 0) is safe. Makes the documented per-item budget contract hold uniformly across all branches. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * feat(structure-tree): log once when the node budget truncates parsing Add a one-shot truncation flag on StructWalk, set the first time the budget is exhausted, and emit a single warn! after parsing so an operator can tell when a (very large or malformed) tagged tree was cut off. Avoids per-item log spam; negligible overhead on the normal path. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): flag truncation at budget guards, not just in charge The truncation flag was only set inside charge() on the budget==0 branch, but the dominant skip paths use budget==0 guards that break/return before charge() is ever called with an empty budget, so the flag (and the warn!) almost never fired. Route those guards through a new exhausted() that sets the flag when it skips remaining work. Adds a parser-level test that would have caught the missed warning. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): flag cycle/depth skips and charge bare MCIDs fully Two review follow-ups: - Cycle-broken and depth-capped /K skips dropped tagged content without setting the truncation flag, so the one-shot warning never fired for malformed/over-deep trees. Mark those skips via note_skipped() and broaden the warning to cover non-budget truncation. - A bare /K MCID materializes a wrapper node AND a content reference but charged only one budget unit, allowing ~2x the advertised budget for such content; charge both. Adds tests: cycle-skip flags truncation, and bare MCID charges two units. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): charge MCR-dict wrappers the same two units as bare MCIDs A top-level MCR /K dict flows through parse_kid -> parse_struct_element_dict and materializes a Span node + one content ref (two items) but was charged only one unit at node entry, while the bare-MCID path charges two. Charge the content reference in the MCR branch too so the per-item budget is uniform across both wrapper paths. Adds a symmetric test. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): reserve leaf-wrapper budget units atomically A leaf MCID wrapper (bare MCID or MCR dict) materializes a node + one content ref and charged the two units via separate charge() calls. At the last unit the first charge succeeded and the second failed, consuming a unit without emitting the wrapper and denying it to a later element that would have fit. Add charge_n() to reserve both units atomically (or neither), and detect MCR before the node charge so it reserves both up front. Adds a boundary test asserting the leftover unit is preserved. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): stop scanning wide /K once a leaf reservation stalls The atomic charge_n(2) left budget nonzero (==1) when it failed, so exhausted() (budget==0) never broke the root /K loop and a crafted wide array of leaf wrappers was scanned in full after no leaf could fit. Add a stalled flag set on an insufficient reservation and fold it into exhausted(); charge()-based (one-unit) loops are unaffected since they reach budget 0 exactly. Adds a test that a one-unit budget still allows a one-unit item but a failed two-unit reservation stops the scan. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): add traversal budget and stop charging non-materializing dicts Two review follow-ups on budget accounting: - Wide /K arrays of non-materializing items (unsupported value types, OBJR dicts, cycle back-edges) consumed no node budget, so the loop scanned the whole array. Add a separate work budget charged per examined /K item and break the loops when it is spent, bounding traversal even when nothing materializes. - OBJR dicts and dicts without a valid /S were charged the node budget before being recognized and skipped, draining the shared budget and truncating real content later. Hoist the OBJR check and /S validation above the node charge so only materializing nodes consume it (matching the MCR hoisting). Adds tests for the work-budget bound, wide unsupported /K, and non-materializing dicts not charging the node budget. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * docs(structure-tree): mention traversal budget in truncation warning The one-shot truncation warning listed the node budget, cycle, and depth as causes but not the new traversal (work) budget, so a work-budget truncation printed a misleading message. Include MAX_STRUCT_WORK so malformed-PDF debugging identifies the actual limit hit. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |
||
|
|
fabec0aec3 |
feat(napi): add async variants that keep the Node event loop free (#337)
* feat(napi): add processPdfAsync, classifyPdfAsync, extractPagesMarkdownAsync The Node bindings are synchronous, so every call parses on the event loop thread — up to hundreds of milliseconds of dead loop per document in a server. Add additive AsyncTask-based variants that run the same shared implementations on the libuv thread pool and return promises. The existing synchronous exports keep their names, signatures, and behaviour; each sync/async pair shares one implementation. Panics in compute() are caught and surfaced as rejections, matching the sync error contract. Closes #336 Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(napi): read async task buffers in place instead of copying Review feedback on #337: buffer.to_vec() copied the whole PDF on the event loop before the task was queued, so large inputs still stalled the loop and doubled peak memory. The tasks now hold the napi Buffer itself — its ref pins the JS allocation for the task's lifetime and the backing store is stable, so compute() reads it directly from the worker thread. Callers must not mutate the buffer until the promise settles (same contract as Node's async fs APIs); documented on each export and in the README. The suggested removal of ts_return_type was checked and rejected: without it napi-rs generates Promise<unknown> for AsyncTask returns. A comment now records that finding. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(napi): copy async task input on the JS thread for soundness Review feedback on #337: holding the napi Buffer and reading it from the libuv worker was unsound. Buffer derefs straight to the JS-side allocation, so a caller mutating it before the promise settled would race the worker's reads — undefined behavior, not a recoverable error, and the documented don't-mutate contract was unenforceable. Deferring the copy to compute() would not help: any off-thread read races the same way. The JS thread is the only race-free place to take the copy, because JS is single-threaded and nothing can mutate the buffer during the synchronous part of the call. Revert to an owned Vec<u8> copied at call time. The cost is one memcpy, negligible next to the parse the async variants exist to unblock. Docs now state the buffer may be reused or mutated immediately, and a test locks in the copy semantics by mutating the input while a parse is in flight. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |
||
|
|
1f28c00a13 |
Bound column-detection histogram and harden coordinate handling (#328)
* Bound column-detection histogram size Derive the projection histogram from a clamped bin count and skip non-finite page widths. Extreme or malformed text-item coordinates (from the content-stream text matrix) could otherwise drive a very large allocation. 65,536 bins is ~9x the largest legal page, so real layouts are unaffected. Adds regression tests. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Exclude non-finite coordinates from page bounds Items at NaN/inf positions are now skipped when folding the page bounds, so a malformed coordinate can no longer escape as a ColumnRegion boundary, and an all-non-finite page returns no columns. Bad items are dropped individually rather than failing the page, so one stray glyph does not disable column detection. Addresses review feedback on the finite-width guard. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Trim far-outlier coordinates from page bounds Gutter margins, spanning-item width and the XY-cut margin are all fractions of page_width, so a single far-but-finite item (x=50_000 is enough) set the scale for the whole page: real gutters fell inside the rejected margin band and a genuine two-column page collapsed to one region. When the span exceeds one legal page (14_400 units), re-derive the bounds from items clustered around the median x. Outliers keep their text because column assignment buckets by nearest overlap. The MAX_BINS ceiling stays as an allocation bound that does not depend on this heuristic. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Harden bounds trimming against widths and wide layouts Check both item edges when trimming: a malformed width at an ordinary position poisoned x_max just as a malformed position poisoned x_min, so a huge width still collapsed a two-column page to one region. Only trim when the far items are a small minority (<=10%). A genuinely large-format page has content spread across its full width, so it now keeps its true bounds instead of being reduced to the median cluster. Correct the MAX_PAGE_EXTENT comment: 14_400 units is the traditional Acrobat architectural limit, not a format cap. PDF 2.0 sets no page-size limit and UserUnit scales physical size, so this is a heuristic. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Scale bin width so the histogram spans the whole page Clamping the bin count alone left anything past MAX_BINS * BIN_WIDTH (~131k points) outside the histogram, folded into the final bin. A page wide enough to hit that lost real gutters: with a visible gutter inside the covered range the XY-cut fallback never runs, so a three-column layout silently reported two. Derive bin_width from page_width instead, keeping the same allocation ceiling and degrading only resolution. Also anchor the trimming median on the same finite left/right items that bounds() accepts, so a malformed width cannot shift which items count as strays. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Require geometric evidence before detaching far content The median-window trim narrowed any content more than one page from the centre, so a valid large page with a sparse far sidebar lost the sidebar from its bounds and its text fell into column 0. An item-count minority rule cannot tell that layout from malformed coordinates. Group content into clusters separated by more than a whole page of continuous emptiness, and only drop a cluster that is both detached by such a void and a small minority of items. Real content does not leave a gap that large; a stray coordinate sits alone beyond one. A single run wider than one page is treated as a malformed width, which also covers the huge-width case the cluster sweep cannot see (such an item spans everything and leaves no gap). Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Judge run width against page content, not a fixed extent Treating any run wider than 14_400 units as malformed penalised valid large pages: one made entirely of such runs reported no columns at all, and a mixed page lost the right edge of every long run. Judge width relative to the page's own content instead. Positions cannot be inflated by a bogus width, so the spread of the core cluster is a sound scale: a run wider than that spread plus one page is malformed. A genuinely large page keeps its genuinely long runs, while a 1e12-wide run beside ordinary text is still rejected. Cluster on positions rather than filled intervals, so a bogus width can no longer merge everything into one cluster, and keep ordinary pages on an O(n) fast path that skips the sort entirely. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |
||
|
|
f4b8c9e854 |
Clarify SECURITY.md reporting channels (#329)
* Update SECURITY.md reporting channels Clarify that email is the only required channel and point the alternative at Firecrawl's Bugcrowd disclosure engagement instead of the private-advisory link, which is not enabled on this repo. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Make Bugcrowd the preferred reporting channel Bugcrowd's disclosure engagement is the primary channel; email to help@firecrawl.dev is offered as the alternative. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |
||
|
|
69039f2728 |
Fix char-boundary panic in hex_to_unicode_string (#320)
Use hex.get(i..i+2) instead of &hex[i..i+2] so a non-hex, non-ASCII destination in a /ToUnicode CMap can no longer trigger a UTF-8 char-boundary panic. An even byte length does not guarantee the byte offset falls on a char boundary; get() returns None on a non-boundary or out-of-range index, folding cleanly into the existing flow. Add regression tests covering a multi-byte destination char and a replacement-char byte. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |
||
|
|
3cca6446bd |
fix(glyph_names): handle non-ASCII input in uniXXXX glyph name parsing (#321)
Use str::get instead of a byte-length check plus slice when parsing the uniXXXX glyph-name form. The byte-length guard only proved the index was in bounds, not on a UTF-8 char boundary, so a glyph name containing non-ASCII bytes could cause a slice on a non-boundary index. Switch to a checked slice that folds into the existing Option flow, and add tests. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |
||
|
|
493fed498e |
fix(links): prevent stack-overflow DoS from AcroForm /Kids self-cycle (#314)
* fix(links): guard AcroForm /Kids traversal against cycles and huge trees A crafted PDF whose AcroForm field lists itself (or another ancestor) in /Kids caused walk_form_fields to recurse indefinitely, overflowing the stack and aborting pdf2md (exit 134) — an application-level DoS from a ~730-byte input. Track visited field object IDs to break /Kids cycles, and cap total field-node traversal at 100k nodes to bound pathologically large trees. Adds regression tests for self-cycle and mutual-cycle field graphs. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(links): cap AcroForm /Kids recursion depth to stop deep-chain overflow The visited-set guard stops cyclic /Kids graphs, but a long *acyclic* chain of distinct fields still recurses to the chain length and overflows the stack (a ~1.6MB PDF with 20k linked fields aborts pdf2md, exit 134) before the 100k node budget is reached. Add an explicit recursion depth cap (100 levels — far above any legitimate form hierarchy) so stack usage is bounded independently of node count. Adds a deep-acyclic-chain regression test. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(links): enforce form-field node budget before insertion The node-budget guard inserted each field ID into the visited set before checking the budget, so the check triggered an early return but never actually capped the set. A field with a huge /Kids array kept inserting post-budget IDs, letting visited (memory and work) grow with the crafted input rather than stopping at MAX_FORM_FIELD_NODES. Check depth and budget before inserting, so visited can never exceed the cap. Adds a wide-tree regression test. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(links): stop /Fields and /Kids iteration once node budget is spent Checking the budget before insertion capped the visited set, but callers still iterated every remaining entry of a wide /Fields or /Kids array after the budget was exhausted — each walk returned immediately, yet the O(N) sibling iteration let a single multi-million-entry array burn extraction CPU unbounded. Break out of both the top-level and recursive loops once visited reaches the cap, making the budget a true traversal-work cap. Adds a top-level wide-/Fields regression test. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(links): charge examined entries against the field-node budget The budget counted only distinct visited nodes, so /Fields or /Kids arrays full of invalid (non-reference) or duplicate entries never grew visited and ran to completion regardless of size — the node budget did not actually cap traversal work. Introduce FieldWalkBudget tracking both visited nodes and total entries examined; charge every array entry (valid, invalid, or duplicate) and stop once either hits MAX_FORM_FIELD_NODES. Adds a regression test with a huge /Kids array of duplicate + null entries. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(links): iterate /Fields and /Kids arrays by borrow, not clone Both arrays were cloned in full before the budget check, so a crafted oversized /Fields or /Kids array forced an O(n) allocation and copy regardless of the cap. resolve_array already returns a borrow tied to the document and the walker only needs a shared &Document, so iterate the borrowed arrays directly — the early break now bounds how many entries are even touched, before any per-array allocation. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * docs(links): correct wide-array test comments to match range assertions The two wide-array tests assert item counts within a range near the budget, not an exact value (charging entries in the entry guard shifts the boundary by one or two). Fix the stale comments that claimed exact counts. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |
||
|
Abimael Martell
|
436af97038 |
fix(tables): exclude script attachments and tiny numeric fragments from detection (#264)
* fix(tables): exclude script attachments and tiny numeric fragments from detection
Split out of #242 (draft) so it can be reviewed on its own evidence.
Display equations with sub/superscripts form phantom small-font table
regions: the subscripts cluster with nearby small text (footnotes, axis
labels) into fake multi-column grids. Two guards:
- Script attachment: a small-font item horizontally adjacent to a
larger-font item at a genuine baseline offset is a sub/superscript,
not a table cell, and is excluded from candidates. A real baseline
offset is required so a small cell beside a larger same-baseline
label is never filtered. Attachment targets are indexed by Y and
scanned through a bounded window rather than a full-page sweep.
The body-font pass applies the same exclusion but only for
heading-sized anchors (>= 1.15x base), so body-size table cells
beside slightly larger labels are untouched.
- Tiny numeric fragments: a <=2-row grid whose every cell is a bare
1-2 digit number carries no tabular information. Restricted to the
small-font pass, where the pattern is overwhelmingly exponent
clusters; body-font numeric grids are unaffected.
Corpus impact: 20 of 186 documents, measured against a control build of
main so main's own drift is excluded. Table rows fall in 17 of 18
inspected documents and no content is lost — 2103_07786 drops all 21
rows, every one a math fragment ('|X 42 43|1|||'); Stijn_SB_doc drops
173 rows of footnote text that had been shredded into cells, with word
count slightly UP and footnote markers intact. M2019_mordeste gains 11
rows from a 3-column grid re-detected as 2-column, neither clearly
better nor worse.
Note the 20 documents is far more than the 3 that per-change ablation
suggested: that figure measured sole-cause attribution inside the
original combined PR, where other heuristics changed the same files and
masked this one. Reach and sole-cause are different measurements.
805 unit + 148 integration tests pass, clippy clean, in-repo snapshots
unchanged.
* fix(tables): suppress script column evidence instead of dropping candidates
Reworked after reviewing the corpus diffs: the first approach removed
sub/superscripts from table candidates entirely, which had two failure
modes beyond the intended fix.
- Legitimate cell content was displaced. citizen-sr-282 is a calculator
manual whose engineering-notation table lists M = 10^6, k = 10^3.
Those exponents are superscripts, so they were dropped from the table
and resurfaced elsewhere in the reading order ('9 mega 6 kilo = 10 3
milli').
- Removing items changed the candidate geometry, so different spurious
structure could form from what remained.
Scripts are now kept as candidates and excluded only from the geometry:
they cannot create a column (find_column_boundaries), cannot qualify a
region on their own (find_table_regions / _strict), but are still
assigned to cells. Column alignment is validated against ALL items
including scripts — validating only the non-script subset would let a
region manufacture alignment by ignoring its awkward items, which is
what block-diagram pages did.
Corpus: 20 of 186 documents, net -359 table rows, no content lost.
Token-level comparison shows the only text changes are merges in the
right direction: 'X' + '10' becomes 'X10', 'L' + 'g' becomes 'Lg' —
subscripts joining their base instead of floating free.
Remaining artifact: MCF5235RM (and its _nxp duplicate) gains a small
spurious table from a block-diagram label line, and M2019_mordeste
gains 11 rows from a 3-column grid re-detected as 2-column. Both are
borderline regions where the previous output was also wrong; documented
rather than tuned away.
805 unit + 148 integration tests pass, clippy clean.
* fix(tables): use a heading-anchored script mask in the body-font pass
Cubic review of #264: a single script mask computed with a 0.0 anchor
was applied to both passes, including body-font region qualification and
geometry. The body pass is supposed to require a heading-sized anchor —
that distinction existed before the geometry rework and was lost in it.
Why it matters: body-pass candidates are themselves body-sized
(0.85..1.05x base). A cell at the low end of that band, say 8.5pt,
sitting beside a 10.5pt label clears the inherent 'anchor >= 1.2x cell'
rule (10.2) and so was flagged as a script attachment. At body sizes a
slightly larger neighbour is a bold label or column header, not the base
of a superscript, so flagging it stripped real cells out of the region
evidence and column geometry and could lose the table entirely.
Two masks now: the small-font pass keeps the 0.0 anchor, the body pass
requires >= 1.15x base. Note the threshold only bites below base size —
for a cell at base, 1.2x-of-cell already exceeds 1.15x-of-base — which
is exactly the 0.85..1.0x band cubic identified.
Corpus: 20 documents, net -367 table rows (was -359 with the single
mask), so the body pass now keeps 8 rows of real table it had been
discarding. 958 tests pass, clippy clean.
* fix(markdown): reject headings that end on a relational verb
A heading candidate ending in 'equals', 'denotes', 'implies' and the
like is the first half of a sentence, not a title. This shows up when a
block dissolves and strands its lead-in ahead of the formula it
introduced — opendataloader 01030000000144 produced
## Note that the exact error equals
M - Q(h) = e - 2.7525... = -0.0342....
Deliberately a very short list. Broader variants were tried and
measured, then rejected:
- Function words (of/and/for/the): a heading that WRAPS across lines
ends on exactly those. Destroyed real IRS Publication 17 headings —
'Casualty and' -> 'Casualty and Theft Losses', 'Rule 10. You Must Be
at' -> '... At Least Age 25'. 52 documents affected, -619 headings.
- Copulas and auxiliaries (is/are/be/have): same failure. 'Rule 15.
Your AGI Must Be', 'What Medical Expenses Are' and 'When Can a Roth
IRA Be' are real wrapped headings, while 'the tax burden should be'
is a genuine fragment. The trailing word cannot separate them; that
needs the next line's context, which this text-only predicate lacks.
The verbs kept never end a heading in any register, so they are safe
without context. Standalone the guard is a no-op on both benchmarks
(0 documents on opendataloader, 4 on pdf-evals with no net heading
change) — its value is as a companion to the table filter in this PR,
which is what strands these lead-ins.
Combined effect on opendataloader (200 docs, vs a control build of
main), where the table filter alone regressed:
table filter + this guard
overall -0.0003 +0.0003
mhs -0.0019 +0.0003
doc ...144 -0.063 +0.053
doc ...144 mhs -0.203 +0.028
* review: gate the dangling-verb veto on sentence case, drop 'yields'
Cubic review of
|
||
|
Abimael Martell
|
f731e1191c | fix(site): refresh benchmark results (#289) | ||
|
Andrew Barnes
|
54a1e9ab74 | Preserve page-prefixed Markdown content (#284) | ||
|
m-naoki-mandClaude Opus 5
|
fabbb63521 |
fix(tounicode): skip subset GID remap for CIDFontType0 (CFF) descendants (#209)
* fix(tounicode): skip subset GID remap for CIDFontType0 (CFF) descendants The sequential-GID repair in try_remap_subset_cmap assumes CIDs are glyph indices that a subsetter can renumber. That holds for CIDFontType2 (TrueType) but not for CIDFontType0 (CFF), where CIDs are resolved through the CFF charset, so a valid ToUnicode CMap stays valid after subsetting. For CFF fonts the corrupting path was unavoidable: CIDToGIDMap is CIDFontType2-only (PDF 32000-1:2008, 9.7.4.2), so the branch that repairs the CMap correctly can never be taken, and any CFF font whose /W array starts at a low CID fell through into remap_to_sequential. Japanese Adobe-Japan1 documents extracted as long runs of a single unrelated kanji. Guard both repair paths on a CIDFontType2 descendant, placed before the CIDToGIDMap branch so a CIDToGIDMap wrongly attached to a CFF font by a malformed producer is ignored too. On a National Diet Library proceedings PDF: 1233 U+FFFD in 69099 chars before, 0 in 68331 after; character 3-gram recall against a hand-written ground truth 0.354 -> 0.605. The PDF from #118 (CIDFontType2) extracts byte-identically before and after. Two existing tests build descendant dicts without a /Subtype and set CIDToGIDMap, which is CIDFontType2-only, so the fixtures now say what they already meant. Without that, test_try_remap_skipped_when_w_covers_cmap would keep passing while no longer exercising the W-coverage logic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tounicode): resolve indirect /Subtype, skip only explicit non-CIDFontType2 Addresses review feedback on the guard added in the previous commit. - /Subtype may be an indirect reference, and as_name() does not dereference it. A genuine CIDFontType2 font storing /Subtype indirectly would have been read as "not CIDFontType2", returning early and losing the repair it needs — reintroducing the corruption this PR fixes, for those fonts. Resolve the reference through the document before comparing. - Bail out only when /Subtype is explicitly a non-CIDFontType2 name. A missing or unresolvable /Subtype now keeps the pre-existing behaviour instead of silently disabling the repair. As a result the two existing tests no longer need fixture changes, and this commit reverts those; the diff against main is now additive only. - The CFF regression test now attaches a real CIDToGIDMap stream rather than /Identity, which get_cid_to_gid_map treats as "no map". With the stream, the test also fails if the guard is moved back below the CIDToGIDMap branch — verified by moving it and watching it fail. - Added test_try_remap_resolves_indirect_subtype. cargo fmt --check, cargo clippy -- -D warnings and cargo test (862 tests) pass. The Diet PDF still extracts with 0 U+FFFD and the #118 PDF is still unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
585d36e6a6 |
fix: recover from a corrupted startxref pointer (#230)
* fix: recover from a corrupted startxref pointer Fixes #228. A PDF whose startxref pointer has been corrupted to point at the wrong byte offset — a single flipped digit, which is what damaged writers emit in the wild — was entirely unprocessable: every entry point (classify_pdf, extract_pages_markdown, process_pdf) raised "Invalid PDF structure", even though the file's object data, real xref table, and trailer were all completely intact just past the wrong pointer. Both pypdf and pdfium recover from this by locating the real table directly instead of trusting the pointer; lopdf doesn't. Added a new repair candidate (alongside the existing missing-%%EOF-marker and stripped-leading-bytes repairs in repair_pdf_container_candidates): scan the buffer for the real, standalone `xref` keyword and append a corrected trailing `startxref`/`%%EOF` block. lopdf's own get_xref_start always reads the *last* `%%EOF` in the final 512 bytes of the buffer and the `startxref` value immediately before it, so the appended block transparently supersedes the corrupted one already in the file — no in-place byte surgery on content the original writer produced. Scoped to classic (non-stream) xref tables, matching the reported repro and the common case; a corrupted pointer into a cross-reference *stream* (`N 0 obj << /Type /XRef ...>>`, some PDF 1.5+ writers) would need the containing object's number, not just a byte offset — out of scope here. Verified against the issue's exact repro (a valid one-page PDF with a single corrupted byte in its startxref offset): before this fix, process_pdf/classify_pdf/extract_pages_markdown all raised "Invalid PDF structure"; after, both the page count and the real extracted text ("Order Detail Report by Account", "WIDGET ASSEMBLY", the dollar amount) come back correctly. New regression test added. Full suite (859 tests, 1 new) passes; cargo clippy --all-targets -- -D warnings unchanged at 28 pre-existing/unrelated errors. * fix: validate xref table shape and scan in a single reverse pass Addresses cubic-dev-ai's review of #230. - P2 (correctness/safety): the recovery candidate trusted the last standalone "xref" token unconditionally, without confirming it's actually a cross-reference table. A coincidental "xref" substring inside unrelated content — a stream, a string, uncompressed metadata — could get "repaired" against a bogus offset, letting lopdf load successfully against garbage instead of returning a clean error: a real failure turned into silent data corruption on the fallback path. Added looks_like_xref_subsection_header, which confirms a plausible classic xref subsection header (`<start-id> <count>`, e.g. "0 6" — the shape every real classic table starts with) actually follows the candidate token before accepting it. find_last_valid_xref_table_start now walks backward from the end of the buffer until it finds a token that both stands alone *and* validates, rather than accepting the first (rightmost) standalone match unconditionally. - P2 (performance): the old scan re-invoked `buf[..search_end].windows(4).rposition(...)` on a shrinking prefix every time a candidate token failed the boundary check, which is quadratic on a pathological buffer with many non-standalone "xref" occurrences. Rewrote as a single reverse byte-index walk — O(n) regardless of how many false candidates it has to reject along the way. Added direct unit tests on the byte-level scan (more precise than constructing adversarial full PDFs, and the coincidental-match scenario can't be represented in an integration-test fixture anyway since reportlab compresses page content by default): a coincidental standalone "xref" with no subsection header is rejected; a real classic table is found; a coincidental match positioned *after* the real table in the buffer doesn't shadow it; "xref" as a substring of "startxref" still doesn't match. The original #228 repro (corrupted startxref pointer, real table otherwise intact) is unaffected — verified manually in addition to the existing integration test. Full suite (863 tests, 5 new) passes; cargo clippy --all-targets -- -D warnings unchanged at 28 pre-existing/unrelated errors. * fix: reject xref subsection count runs with trailing garbage looks_like_xref_subsection_header validated that a count run of digits followed the whitespace separator, but never checked what came after it. A coincidental "xref\n0 6garbage" in stream/literal content would still validate as a real subsection header shape and get repaired against a bogus offset. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Abimael Martell <1450169+abimaelmartell@users.noreply.github.com> |
||
|
|
371de80b14 |
fix: extract_pages_markdown's needs_ocr now agrees with classify_pdf (#231)
* fix: extract_pages_markdown's needs_ocr now agrees with classify_pdf Fixes #227. extract_pages_markdown_mem computed its per-page needs_ocr entirely from text-quality signals: decoding/garble issues, empty markdown, GID fonts, garbage-text ratio. It had no awareness of the page's image content at all — so a page that is fundamentally a full-page scan with a little genuine native text drawn over it (a header, a stamp, a cover-sheet annotation) extracts that text cleanly, trips none of the text-quality checks, and reports needs_ocr=false — while classify_pdf/detect_pdf_type correctly see the dominant background image and flag the same page as needing OCR. Two public APIs answering the same question, silently disagreeing, in the unsafe direction (skipping OCR on a page that needs it). Exposed detector::analyze_page_images at crate visibility (was private) and call it per page in extract_pages_markdown_mem's loop — the same "large background image" signal (>50% page coverage) that already powers has_template_image in classify_pdf/detect_pdf_type, rather than reimplementing image-area detection a second time with its own thresholds that could drift out of sync again. When it's true, the page is flagged needs_ocr (with OCR_REASON_SCANNED added to ocr_reasons_by_page, matching how the same signal is already reported elsewhere) and its markdown is blanked, exactly like the existing text-quality-triggered needs_ocr paths already do — no special-casing added for "cleanly-extracted-but-still-a-scan" text. Verified against the issue's exact repro (a full-page raster with one native text line drawn over it, built via reportlab/pillow): before this fix, extract_pages_markdown_bytes reported page 0 needs_ocr=False with the header line as markdown while classify_pdf_bytes correctly flagged pages_needing_ocr=[0]; after, both agree needs_ocr=True and the page's markdown is empty. Confirmed no regression on a normal text-based fixture (nexo-price-en.pdf: needs_ocr stays False, full markdown returned). New Rust regression test added exercising both APIs against the same fixture. Full suite (860 tests, 1 new) passes; cargo clippy --all-targets -- -D warnings unchanged at 28 pre-existing/unrelated errors. * fix: gate has_template_image behind the same OCR signals classify_pdf uses extract_pages_markdown_mem was treating has_template_image alone as sufficient to force needs_ocr=true and discard the page's markdown, but classify_pdf/detect_pdf_type never treats that raw signal alone as needing OCR. A text page with a full-bleed watermark, letterhead, or large figure would get its clean markdown wrongly blanked and routed to OCR. Added page_template_image_needs_ocr(), mirroring the two distinct signals classify_pdf actually uses to decide a template-image page needs OCR: the looks_like_scan gate (image_count <= 1, few text ops, low alphanumeric diversity) used for Mixed-type routing, and the insufficient-text-volume signal (text_operator_count < 10) that routes a page with a dominant background image and only a couple of native text calls to PdfType::ImageBased independent of looks_like_scan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix: match per-page OCR threshold and add missing vector-text signal Two follow-up findings on the has_template_image gate added in the previous commit: 1. insufficient_text used a hard-coded threshold of 10 text operators, but Mixed-type per-page routing (the actual per-page decision this function tries to agree with) uses config.min_text_ops_per_page (default 3). The higher 10 threshold was borrowed from a *different* classify_pdf code path — the effective_min_ops floor used only for whole-document ImageBased/Scanned classification, a cross-page aggregate this per-page function can't replicate anyway. Using the lower per-page threshold removes a real disagreement window (3-9 text ops with high alphanumeric diversity) without breaking the #227 regression fixture (text_ops=1, still well under 3). 2. extract_pages_markdown_mem never checked has_vector_text at all, even though Mixed-type per-page routing always sends vector-outlined-text pages to OCR (outlined glyphs can't be extracted as text). A page with massive path ops plus a short genuine caption could extract that caption cleanly, slipping past the existing empty/garbage-text checks. Added page_has_vector_text() and wired it into needs_ocr the same way has_template_image is. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * perf: compute template-image and vector-text OCR signals in one pass page_template_image_needs_ocr and page_has_vector_text each called analyze_page_content independently, so every requested page's content streams (page + XObjects) and image coverage were decompressed and scanned twice per page with one result discarded each time. detect_from_document avoids this by caching its per-page PageAnalysis; extract_pages_markdown_mem had no such cache. Merged both into page_ocr_signals(), a single analyze_page_content call returning both signals as a tuple. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Abimael Martell <1450169+abimaelmartell@users.noreply.github.com> |
||
|
Abimael Martell
|
ede48099c0 |
fix(layout): preserve ruled tables and chart prose order (#262)
* fix(layout): preserve ruled tables and chart prose order * fix(layout): harden chart region detection * fix(layout): tighten chart geometry guards * fix(layout): bound chart inference * fix(layout): tighten chart claim bounds * fix(layout): tighten chart evidence * fix(layout): preserve edge-adjacent chart labels * fix(layout): require external chart label overlap |
||
|
Abimael Martell
|
12e9a655e3 |
fix(extractor): supply built-in metrics for non-embedded base-14 fonts (#241)
* fix(extractor): supply built-in metrics for non-embedded base-14 fonts PDFs may legally omit /Widths for non-embedded standard fonts (Times, Helvetica, Courier, Symbol, ZapfDingbats) — the spec requires the reader to supply the metrics. We returned None, so every glyph advanced 0 and each text item got width 0, silently breaking every gap-based heuristic downstream: space synthesis, sub/superscript detection, table column detection, heading merging. - src/extractor/base14.rs: Adobe Core-14 AFM width tables keyed by Unicode char, plus the standard Symbol/ZapfDingbats encoding vectors (their glyphs sit at byte positions unrelated to Latin text, so widths must resolve through the built-in encoding, not cp1252) - Width resolution order: Differences -> built-in encoding -> the same cp1252-style fallback the text decoder uses, so a code's advance always matches the character we emit for it - Type3 visual sizing: PK bitmap fonts (dvips) use FontMatrix [1 0 0 -1 0 0] with nominal sizes like 0.12pt; scale by FontBBox height x |matrix_y|. Applied in the page-stream and Form XObject paths. Indirect numeric array elements are resolved before use. Effect on Shannon's 'A Mathematical Theory of Communication' (1998 dvips/Distiller, the reported case): glued sentences 95 -> 5. Corpus impact: 12 of 184 eval documents, e.g. Data-Processing-Agreement recovers a paragraph that a phantom table had shredded into cells. Layout heuristics tuned on the same document (indent-based paragraph breaks, heading reclassification, table script filtering) are held back for a separate PR — they change ~98 further documents and need to be justified against the corpus, not against one PDF. * review: narrow Type3 rescaling to self-inconsistent fonts; dedup + test all width tables Addresses cubic review on #241, plus a follow-up from a local cubic run. - Type3 visual scaling was applied to every Type3 font whose FontBBox height x |matrix_y| deviated >5% from 1.0. FontBBox is the glyph box, not the em box, so a conventional 1/1000-matrix font with a descender..ascender bbox (~700 units) computed 0.7 and had every reported size shrunk by 30% — corrupting the drop-cap, heading-tier, sub/superscript and table heuristics this is meant to fix. First attempt gated on the matrix being unit-scale, but a local cubic run pointed out that wrongly excludes valid non-standard matrices (a 0.005 matrix with a full-em bbox legitimately needs a 5x scale). The product is the right discriminator, not the matrix: a self-consistent font lands near 1.0 because the matrix is the reciprocal of the glyph-space em, so only a wildly inconsistent one (dvips/PK bitmap fonts sit at ~159) is renormalized. Band widened to [0.25, 4.0]. Corpus effect: 12 -> 7 documents change. The 5 that drop out were being wrongly rescaled — including Data-Processing-Agreement, whose phantom-table fix turned out to come from this bug rather than from the width fallback, so it is correctly given up. - base14: all 14 width tables now covered by the sort-invariant test via an ALL_TABLES registry, not a hand-picked subset. - base14: identical tables share one static (all four Courier variants are monospace 600; the oblique Helvetica variants match their upright forms), removing 5 duplicate copies. * test: refresh Shannon snapshot after merging main CI checks out a merge of the PR head with main, and main advanced 8 commits since this branch was cut — including #201 (contextual digit runs), #240 and #253 (markdown fixes). Those change extraction output, so a snapshot generated on the unmerged branch could not match; the Test job failed on the merge commit while passing on the branch itself. The merged behaviour is better: the footnote marker '2' before 'Hartley, R. V. L.' is now recovered instead of dropped. 950 tests pass on the merged tree, clippy clean. |
||
|
Abimael MartellandClaude Fable 5
|
1d134e26aa |
docs: sync AGENTS.md with CLAUDE.md, refresh eval workflow guidance (#243)
AGENTS.md was stale (179+ PDFs, missing the semantic-quality bullet). Both files now match: ~200-PDF corpus, and iteration guidance to prefer subset runs (bench.py test -q / -s <name>) with the full suite as the final pre-commit check. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Shubham Mathur
|
bfd6c3eabb |
fix(extractor): make the comment stripper escape-aware (#259)
strip_pdf_comments tracked parenthesis nesting to protect string literals, but ignored backslash escapes. An escaped \) desynced the depth counter, after which a % glyph inside a string was stripped as a top-level comment, corrupting the stream for Content::decode and silently truncating the page's text. Treat \ inside a string literal as escaping the next byte, so \(, \), and \\ never touch the nesting depth. |
||
|
Sheroy Cooper
|
04abab951f | Support password-protected PDF item JSON extraction (#245) | ||
|
Sunil
|
a410d5aa08 |
fix: sync Python type stubs (.pyi) with actual bindings and fix doc bugs (#250)
## Bug 1: Python type stubs missing 5 fields/classes The pdf_inspector.pyi file was out of sync with the actual Python bindings exposed via #[pyo3(get)] in src/python.rs. This breaks IDE autocompletion and type checking (PyCharm, VS Code/Pylance, mypy) for all Python users. Added: - PdfResult.ocr_reasons_by_page (python.rs:35) - PageOcrReasons class with page and easons fields (python.rs:67-86) - RegionText.ocr_reason (python.rs:136) - PageMarkdown.ocr_reason (python.rs:193) - PagesExtractionResult.ocr_reasons_by_page (python.rs:226) ## Bug 2: PdfResult.pages_needing_ocr indexing undocumented PdfResult.pages_needing_ocr is 1-indexed (per python.rs:30) but neither the .pyi stubs nor docs/python.md annotated this, while the same field on PdfClassification was annotated as 0-indexed. Users mixing both APIs would get wrong page numbers. ## Bug 3: README.md duplicate bullet character The Markdown features table listed * twice in bullet prefixes. The first should be ullet (U+2022), matching the actual source code in src/markdown/mod.rs:34 which uses: ullet, dash, sterisk, circle, illed-circle, open-circle. ## Bug 4: docs/python.md missing fields in type reference The Types section was missing PageOcrReasons, RegionText class definition, ocr_reason fields, and ocr_reasons_by_page fields. ## Evidence Cross-referenced every #[pyo3(get)] attribute in src/python.rs against the .pyi declarations and docs/python.md type reference. |
||
|
Abimael Martell
|
7747b3a086 |
fix(markdown): reject non-text strikeout rules (#253)
* fix(markdown): reject non-text strikeout rules * fix(markdown): address strikeout ownership edge cases * fix(markdown): reject connected filled strike rules * fix(markdown): group drifted strikeout runs |
||
|
|
ae6246ba0c |
fix(markdown): preserve redline edits in prose (#240)
* fix(markdown): preserve redline edits in prose * fix(tables): preserve live evidence near redlines * fix(tables): scope redline table suppression * fix(tables): preserve revised cell evidence * fix(tables): retain page context for redlines * fix(tables): propagate revised column evidence * fix(tables): preserve redline boundaries in headers * fix(tables): separate redline suppression spans * fix(tables): tighten revised cell evidence --------- Co-authored-by: Bryan Nathan <bryan@users.noreply.github.com> Co-authored-by: Abimael Martell <1450169+abimaelmartell@users.noreply.github.com> |
||
|
Abimael Martell
|
3fb545284b |
fix(layout): preserve contextual digit runs (#201)
* fix(layout): preserve contextual digit runs * fix(layout): harden page folio filtering * fix(markdown): distinguish folios from contextual numbers * fix(layout): distinguish running folios from contextual digits * fix(layout): tighten running folio evidence * test(layout): guard the folio evidence floor * fix(layout): preserve page-number decisions across partitions * fix(layout): harden document-level folio filtering * fix(markdown): carry folio context across public APIs * fix(layout): isolate folios from layout metadata * fix(markdown): preserve table cells in per-page extraction * fix(layout): preserve folio context with page filters * fix(layout): handle contextual folio sequences * fix(layout): tighten adjacent folio evidence * fix(layout): constrain folio context inference * fix(forms): resolve widget pages from annotations |
||
|
Jase-Omeileo West
|
1c32e4bd69 |
fix(deps): bump lopdf to 0.42.0 for nesting-depth DoS (#198)
Bumps lopdf from 0.41.0 to 0.42.0 to fix RUSTSEC-2026-0187, preventing deeply nested PDFs from causing an unrecoverable stack-overflow process abort. |
||
|
Abimael Martell
|
8121ae97ce |
chore(ci): bump GitHub Actions to current majors (#237)
* chore(ci): bump GitHub Actions to current majors Node 20 action runtimes are deprecated on GitHub runners; bump every first-party action to its latest major across all workflows: - actions/checkout v4/v6 -> v7 - actions/cache v4 -> v6 - actions/upload-artifact v4 -> v7, download-artifact v4 -> v8 - actions/setup-node v6 -> v7, setup-python v5 -> v7 - actions/upload-pages-artifact v3 -> v5, deploy-pages v4 -> v5 Third-party pins (dtolnay/rust-toolchain, Swatinem/rust-cache, setup-zig, setup-bun, taiki-e/install-action, maturin-action) are already on their latest majors. * chore(ci): pin all actions to full commit SHAs Mutable @vN tags can be retagged; in the publish workflows that code runs with OIDC credentials before npm/PyPI/crates.io publishes. Pin every action (first- and third-party) to its release commit SHA with the version in a trailing comment. dtolnay/rust-toolchain infers the toolchain from its ref name, so the SHA-pinned invocations pass an explicit toolchain: stable input. |
||
|
Abimael Martell
|
98990cc550 |
feat(npm): add Linux musl and ARM64 N-API binaries (#233)
Adds x86_64-unknown-linux-musl, aarch64-unknown-linux-gnu, and aarch64-unknown-linux-musl to the napi build targets so @firecrawl/pdf-inspector works on Alpine and ARM64 Linux deployments. - gnu arm64 cross-compiles with --use-napi-cross (old-glibc sysroot), musl targets with -x (zig + cargo-zigbuild), per the napi-rs template - new platform packages carry npm libc metadata (glibc/musl) - smoke-test job runs napi/test.mjs on all six targets before publish (Alpine containers for musl, ubuntu-24.04-arm runners for ARM64) - bump to 1.12.0 to trigger publishing of all platform packages Closes #216 |
||
|
Abimael Martell
|
a15ec2d68d |
docs(readme): refresh local parser benchmark results (#192)
* docs(readme): refresh parser benchmark results * docs(readme): include stored parser results * docs(readme): refresh remaining parser results * docs: sync benchmark references |
||
|
Abimael Martell
|
7b7960ee73 | chore(pypi): bump pdf-inspector to 0.2.6 (#188) | ||
|
Abimael Martell
|
7188667045 | chore(wasm): bump @firecrawl/pdf-inspector-wasm to 0.1.3 (#190) | ||
|
Abimael Martell
|
6ff104409a | chore(npm): bump @firecrawl/pdf-inspector to 1.11.2 (#189) | ||
|
Abimael Martell
|
5e8f1570f6 | chore(crate): bump pdf-inspector to 0.1.7 (#187) | ||
|
tomsideguide
|
b31e4b1727 |
fix(extractor): don't flag gid Differences names covered by ToUnicode (#186)
* fix(extractor): don't flag gid Differences names covered by ToUnicode Pages were marked as having unresolvable gid-encoded fonts whenever any font's /Differences array used gidNNNN glyph names, and when every page carried such a font the whole document's markdown was suppressed. LibreOffice exports do exactly this: subset fonts get /gidNNNN names in Differences alongside a complete ToUnicode CMap that decodes them, so ordinary text documents lost their entire markdown output even though extraction decoded every glyph. Track the character codes behind the gid names and only flag the font when its ToUnicode CMap addresses none of them. Partially mapped codes stay unflagged: an emoji ZWJ sequence maps whole on its first code, and the remaining component-glyph codes are subset leftovers, not damage. Fonts without ToUnicode, or whose CMap ignores the gid codes, are flagged as before, and the downstream garbage/encoding checks still catch partial breakage. * fix(extractor): require a usable ToUnicode mapping to clear the gid flag A mapping to U+FFFD (or an empty string) is rejected by extraction as an invalid CMap result, so it must not count as decodable when deciding whether gid-named Differences codes are resolvable. |
||
|
Abimael Martell
|
3c6eb8bf6b |
feat(site): add local WebAssembly demo (#181)
* feat(site): add local WebAssembly demo * feat(site): parse PDFs on selection |
||
|
Abimael Martell
|
5b287341a0 |
feat(wasm): add browser bindings (#180)
* feat(wasm): add browser bindings * fix(wasm): address review feedback * fix(wasm): preserve numeric plain text * chore(wasm): prepare 0.1.2 release |
||
|
Abimael Martell
|
55d50ad1c4 |
docs(site): redesign open-source project page (#179)
* docs(site): redesign open-source project page
* docs(site): lead with node and cli
* docs(site): emphasize package registries
* docs(site): align footer wordmark
* docs(site): remove hero terminal scrollbar
* docs(site): keep hero install command inline
* docs(site): feature rust core in hero
* docs(site): add hero language playground
* Revert "docs(site): add hero language playground"
This reverts commit
|
||
|
Abimael Martell
|
a910b7df1d |
docs(benchmark): refresh parser comparison (#178)
* docs: refresh benchmark comparison * docs(site): refresh benchmark section * docs: reframe benchmark positioning * docs: focus benchmark positioning on best fit |
||
|
Abimael Martell
|
15c0b22093 |
test(bench): probe optional backend evidence (#176)
* test(bench): probe optional backend evidence * fix(bench): accept native stext pages |
||
|
Abimael Martell
|
0c06dac976 |
test(bench): compare OpenDataLoader builds (#175)
* test(bench): compare OpenDataLoader builds * docs(bench): keep reference comparisons generic * fix(bench): keep regression gates complete * fix(bench): clarify missing reference gates * fix(bench): validate nonnegative limits * fix(bench): isolate prediction runs * chore(bench): refresh review |