main
481
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Abimael Martell
|
841513d3fb |
fix(extract): pass page lines to per-page markdown conversion (#434)
CI / Test (push) Has been cancelled
CI / Format (push) Has been cancelled
CI / Clippy (push) Has been cancelled
CI / Build (macos-latest) (push) Has been cancelled
CI / Build (ubuntu-latest) (push) Has been cancelled
CI / OCR (macos-latest) (push) Has been cancelled
CI / OCR (ubuntu-latest) (push) Has been cancelled
CI / OCR (windows-latest) (push) Has been cancelled
CI / OCR runtime smoke (push) Has been cancelled
CI / WebAssembly (push) Has been cancelled
* fix(layout): keep relative-valley column detection on table pages The page_has_table guard predates item claiming: by grouping time the detected table's items have already left the flow, so a table cannot fake a gutter in the histogram this guard protects, and the prose gate inside relative-valley acceptance rejects any residual table-shaped split. Without the fallback, the prose remainder of a table-bearing two-column page fell to single-column Y-sorting and its columns interleaved line by line. * fix(layout): lower the relative-valley floor to sparse pages The 100-item floor guarded against shallow histogram dips on sparse pages, but OCR'd multi-column pages produce few long line-runs (a two-column French academic page arrives as ~60 items) and were falling to single-column Y-sorting, weaving their columns line by line. The prose gate inside relative-valley acceptance is the real defense against spurious dips; 30 items is enough for it to judge. * fix(layout): re-guard the ungated XY-cut fallthrough on table pages Removing page_has_table from the relative-valley condition also unlocked the XY-cut fallback inside that block, which has no prose gate — a table page whose valley candidate was just rejected could take an unvalidated split. The guard is restored on that call specifically; the relative-valley path keeps its prose-gated access. Also rewrite the stale dense-page comment for the 30-item floor. Bench and corpus unchanged (462/884, corpus byte-identical). * fix(extract): pass page lines to per-page markdown conversion The per-page extraction path (extract_pages_markdown_mem_impl, used by the pages API and every --ocr auto run) partitioned each page's rects but passed an empty slice for PDF lines to markdown conversion, while the extracted all_lines sat unused. Every table only the line-based detector finds (text-anchor rule tables, ruled grids) was silently dropped in that mode, even though the whole-document path emitted it fine. Partition all_lines per page like rects and pass them through. Regression test: a rule-anchored table fixture must survive the pages API. |
||
|
Abimael Martell
|
bac056e801 |
fix(layout): restore column detection on table pages and sparse pages (#430)
* fix(layout): keep relative-valley column detection on table pages The page_has_table guard predates item claiming: by grouping time the detected table's items have already left the flow, so a table cannot fake a gutter in the histogram this guard protects, and the prose gate inside relative-valley acceptance rejects any residual table-shaped split. Without the fallback, the prose remainder of a table-bearing two-column page fell to single-column Y-sorting and its columns interleaved line by line. * fix(layout): lower the relative-valley floor to sparse pages The 100-item floor guarded against shallow histogram dips on sparse pages, but OCR'd multi-column pages produce few long line-runs (a two-column French academic page arrives as ~60 items) and were falling to single-column Y-sorting, weaving their columns line by line. The prose gate inside relative-valley acceptance is the real defense against spurious dips; 30 items is enough for it to judge. * fix(layout): re-guard the ungated XY-cut fallthrough on table pages Removing page_has_table from the relative-valley condition also unlocked the XY-cut fallback inside that block, which has no prose gate — a table page whose valley candidate was just rejected could take an unvalidated split. The guard is restored on that call specifically; the relative-valley path keeps its prose-gated access. Also rewrite the stale dense-page comment for the 30-item floor. Bench and corpus unchanged (462/884, corpus byte-identical). |
||
|
Abimael Martell
|
0027b048ce |
fix(tables): reject parallel-prose grids on all unsplit pages (#429)
* fix(tables): reject parallel-prose grids on all unsplit pages The body-font heuristic pass projects multi-column text pages onto table grids: on a two-column reference section, every line pair across the gutter looks like a row with two X-clusters, and the page is emitted as a woven table. The parallel-prose rejector — which requires transition evidence (unterminated cells flowing into lowercase starts in the same column), not mere cell length — was gated to chart pages; it now runs for every unsplit page. A compact header row still blocks the rejection, except when cross-row prose continuations outnumber the rows: no genuine table produces a continuation on average in every row, so the 'header' there is just short line fragments atop parallel prose columns. Band-split retries stay exempt: they exist for tables that only assemble after recombining bands. * fix(tables): header bypass requires continuations to strictly outnumber rows Align the code with its stated rule (the comparison allowed the bypass at exact equality) and add the dedicated positive-path test: a compact header atop parallel prose columns whose cross-row continuations outnumber the rows is flagged as parallel prose. Bench unchanged. |
||
|
Abimael Martell
|
fb45d37dfe |
feat(markdown): strip page-edge furniture on short documents (#427)
* feat(markdown): strip page-edge furniture on short documents
Running headers, footer credits, and page indicators are proven by
cross-page repetition, which needs three-plus pages; on one- and
two-page documents they survived untouched. Add a positional and
typographic classifier for that gap, dispatched explicitly by document
length (strip_header_footer_lines), and consolidate all header/footer
policy — repetition classifier, edge classifier, shared predicates —
into a new markdown::furniture module.
The edge classifier only strips where evidence is strong: the outermost
block (<= 2 lines) per page edge, vertically isolated from the body by
1.8x the median leading, set strictly smaller than body text, short,
and non-structural. Guards, each earned by a measured false positive:
- same-size blocks stay (section headings, affiliations, continuation
paragraphs merely sit at the margin)
- digit- or (digit)-led blocks stay (footnotes, legal enumerations)
- three-plus item clusters on a baseline stay (figure label rows,
numeric rows of undetected borderless tables)
- number-only blocks set above body size stay (cover years); at or
below body size they strip on isolation alone (page indicators)
- pages under 8 lines and blocks over 90 chars are never touched
* fix(furniture): address review findings on edge and repetition guards
- Number-only blocks strip only on the full page-indicator shape: one
text run, <= 20 chars, set no larger than body text. Multi-cell
numeric rows are form or table data and stay.
- Marker-led guard extends to symbol footnotes ('† ...') and single
lowercase-letter markers ('a See ...'); capitalized one-letter words
('A Publication ...') remain strippable prose.
- The repetition classifier now checks structural shape on the original
text: normalization strips leading digits, so numbered headings like
'1. Introduction' were losing their protection along with the number.
- Restore the repetition classifier's doc comment to strip_repeated_lines;
the module move had left it fused onto the dispatcher.
* fix(furniture): extend structural protection to removal passes, accept (letter) markers
- The repetition classifier's structural check ran only at candidate
collection; a structural line sharing a normalized key with a
non-structural candidate (normalization strips the heading's leading
digit) was still stripped by the removal passes. All three passes —
individual, Y-band, and sibling propagation — now recheck the
original text.
- The marker-led guard accepts parenthesized-letter enumerations
('(a) See ...'), common in legal and scholarly documents.
* fix(furniture): count characters in the repetition floor, protect (digit) enumerations
- The ten-character candidate minimum counted UTF-8 bytes, so short
non-ASCII headings (a six-character CJK section head is 18 bytes)
slipped past the floor and were stripped as repeated furniture. Count
characters in both frequency loops; a Japanese corpus doc recovers a
repeated section heading.
- is_structural_line recognizes parenthesized numbered markers
('(1) Sign and date ...'), protecting repeated enumerations in the
repetition classifier the same way bare numbered lines already were.
* fix(furniture): per-member band protection, close parenthesized markers
- The Y-band removal pass inserted whole bands after checking only the
coalesced text, so a structural sibling banded with a plain fragment
was stripped on every page after the first. Each member is now checked
individually, matching the sibling-propagation pass.
- A parenthesized marker must be a short digit/lowercase token closed by
')' — '(1)', '(a)', '(iv)'. Parenthetical prose footers
('(all amounts in thousands)') strip again.
* fix(furniture): accept Unicode lowercase in parenthesized markers
'(α)'-style footnote markers are lowercase too; the paren-marker check
was ASCII-only while the bare single-letter guard already accepted
Unicode. Token extraction and validation now use Unicode alphanumeric
and lowercase classes, keeping the three-character and closing-paren
bounds.
* fix(furniture): whole-row structural protection, Unicode digits, page floor
- A Y-band holding a structural member is one physical row of content:
a precomputed protected-band set now guards ALL removal paths
(individual, band, propagation), so a heading's row-mates are never
half-removed. Replaces the per-member filter, which mangled rows.
- Every digit check in the marker and structural guards uses Unicode
is_numeric(): '(١)' earns the same protection as '(1)'.
- The edge classifier requires strictly more than 8 lines per page —
at the floor, the two edge blocks could remove half the content.
* fix(furniture): Unicode page-number normalization, shared marker predicate, boundary-safe row protection
- normalize_for_comparison trims Unicode numerics, so headers whose page
numbers use non-ASCII digits normalize to one key across pages and the
repetition classifier can see them.
- The parenthesized-marker rule is one shared predicate
(starts_with_paren_marker) used by both the edge classifier and
is_structural_line: repeated '(a) See ...' annotations survive on
three-plus-page documents too.
- Whole-row structural protection extends to neighboring quantization
buckets: same-row fragments 0.02pt apart can straddle a rounding
boundary, and the boundary must never split a row's protection.
* refactor(furniture): single source for Y-band coalescing
The sort-join-normalize sequence was triplicated across the band
frequency build and both band passes, with each copy free to drift.
One helper (coalesced_band) now produces the sorted members, the
coalesced row text, and its normalized comparison key; each site keeps
only its own guards. Behavior-neutral: full suite unchanged.
|
||
|
Abimael Martell
|
5eb6a13860 |
feat(layout): banded region segmentation for vertically-changing column layouts (#426)
* feat(layout): banded region segmentation for vertically-changing column layouts Pages whose column structure changes down the page (newsletter bands, figure-split flows, a three-column strip inside a two-column page) cannot be represented by one full-height column set: the projection profile either finds nothing and Y-interleaves the columns, or weaves the odd band's text into the wrong buckets. - Split pages into horizontal bands at full-width whitespace gaps (wide spanning items are excluded from occupancy — separators sit inside the very gaps being sought), detect columns independently per band, and re-merge consecutive bands with matching gutters across empty gaps so figure floats keep flowing down their columns while headline-separated bands stay independent. - Engage only on contradicting evidence: a prose-validated band whose column count differs from the page-level structure. Pages the flat column model already explains keep their current ordering. - Read short prose columns (5-14 lines, >=60% width fill on >=60% of lines in every column) as newspaper instead of Y-interleaving them as tabular. Kept as a standalone reading-order refinement so the table pipeline's is_newspaper_layout veto is unaffected. - Split ordering entry points: order_multi_column_region keeps every page-level defense; order_validated_band (banded planner only) trusts validated bands, skipping line-count minimums and straggler splitting that would misfire on Y-cohesive bands. * docs(layout): record why the band wide-item test is per-item Assembling same-baseline fragments into runs before the wide test was implemented and measured against the reading-order benchmark: word-gap and gutter-gap distributions overlap in real documents, so assembled runs fused narrow-guttered column pairs into page-wide lines, emptied the occupancy, and disengaged banding on pages it rescues — a measured regression with no measured win. Keep the per-item test (a fragmented separator can suppress a cut, which only misses an engagement) and document the boundary for future attempts. * fix(layout): keep figure placeholders out of band whitespace probes An image placeholder sitting between two matching column bands is the very figure float whose flow-through the band merge exists for, yet it read as content twice: its glyph box filled the occupancy gap (blocking the cut) and the merge probe counted it as separator content (blocking the merge). Both probes now see text layout items only. * fix(layout): anchor band-merge matching on the founding band's columns The merge comparison ran against the widened union, whose gutter is the intersection of its constituents' gutters. Across a chain of one-directionally drifting bands that intersection can walk past GUTTER_TOLERANCE and reject a band identical to the run's own first member. Compare candidates against the founding band's raw columns instead: the run's column system is defined by its founder, so drift can no longer accumulate in either direction. Union widening is kept for item bucketing only. * test(layout): differential coverage for the band-merge anchor rule - banded_layout_rejects_creeping_drift: a band within tolerance of the moving union but 36pt from the founder must not join the run — the case the anchor rule exists for; the pre-anchor union admitted it. - Reword the founder-anchor chain test as the invariant lock it is. - Note at the merge site why a reject-overlapping-unions guard is unimplementable: detect_columns returns contiguous partitions whose adjacent regions share boundary coordinates, so the check degenerates to exact-equality matching and rejects every legitimate merge; the boundary-disagreement zone is bounded by GUTTER_TOLERANCE and split proportionally by greatest-overlap bucketing. |
||
|
Abimael Martell
|
74ebce430c |
fix(layout): accept figure-diluted prose columns via a sustained full-line run (#417)
* fix(layout): accept figure-diluted prose columns via a sustained full-line run The columns_have_prose gate (added against table/TOC/checklist false-splits in relative-valley detection) requires >=40% of a column's lines to span >=45% of its width. A genuine prose column hosting a figure and caption dilutes that global ratio below the bar — measured 0.38 on a two-column research page — so the valley is rejected, the page falls back to single-column, and same-baseline items from both columns merge across the gutter into woven sentences. Accept a column that contains a sustained paragraph block instead: at least 6 consecutive full-width lines. The scattered layouts the gate exists to reject cannot produce an unbroken run of full lines, and the other guards (column width, minimum lines, items-per-line) still apply. Tests pin both directions; regression corpus is unchanged. * docs(layout): restore columns_have_prose rustdoc displaced by hoisted const The hoisted LINE_FILL_THRESHOLD landed between the gate's rustdoc block and the items that followed, absorbing the function's documentation onto the constant. The docs return to the function, extended to cover the sustained-run acceptance path. |
||
|
Abimael Martell
|
cf4e42b91c |
perf(ocr): adaptive detection escalation and parallel staged engine (#413)
* perf(ocr): adaptive detection escalation and parallel staged engine Three structural limits in the OCR engine, found benchmarking against other parsers, each addressed here: 1. Detection resolution. The backend downscales every page so its longest side fits 960px before text detection. A broadsheet render loses over 80% of its resolution, making body print ~2px tall — invisible to the detection model regardless of render DPI, since recognition crops come from the full-resolution render but detection never sees the text. Detection now runs at 960 and escalates to 2560 only when the standard pass proves the page is dense fine print (page was downscaled, >=80 regions, median region height under 13px at detection scale — measured margins separate gaining pages at 9.9-12.0px from non-gaining pages at 12.2px+). Renders more than twice the standard limit skip the doomed standard pass entirely. 2. Session serialization. oar-ocr guards each ONNX session behind a mutex (pool size is fixed at one), so concurrent predict calls serialize and added intra-op threads go to waste — measured, 12 threads run 2.4x slower than 4. The engine now builds one worker (detector + recognizer sessions) per cores/4 capped at 3, each with 2 intra-op threads, and fans pages across them via a sized rayon pool. A new OcrEngine::preferred_page_concurrency hint lets the pipeline size page batches at three waves per worker, replacing the fixed 4-page chunks that starved workers at batch barriers. 3. Escalation double-cost. The engine is restaged onto oar-ocr's own public components — detect, sort (sort_quad_boxes), crop (TextCroppingProcessor, rotation-aware), recognize — as separate calls instead of the combined predict, with output verified byte-identical across an 8-document corpus. An escalated page now reruns only detection; recognition runs once, on the final region set. Recognition stays one crop per call: batching pads every crop to the widest member, and even width-sorted batches measured 2-3x slower on CPU (ONNX Runtime re-plans kernels per input shape). Escalation lifts ground-truth word recall on dense fine print from 0.678 to 0.895 with no change on ordinary scans, which keep first-pass speed and quality; a 15-page scan drops from 9.4s to 8.0s end to end. * fix(ocr): address review — even-count median, escalation fallback, orientation note - median_detection_height averages the two middle values for even-count region lists instead of taking the upper one, so borderline pages near the 13px escalation threshold are not skewed away from it. - Direct escalated detection (renders over twice the standard limit) falls back to the standard pass on error, matching the adaptive branch, instead of failing the page outright. - Document why OcrSpan::orientation_degrees is None: the combined pipeline's angle came from the text-line-orientation classifier, a model this engine never loads — it was structurally None before the staged split too; region rotation is carried by the polygon. * fix(ocr): return standard boxes directly after direct-escalation failure Falling through to the adaptive branch would re-invoke the escalated detector that just failed — repeating an OOM on a large dense page — before settling on the standard boxes anyway. * fix(ocr): restore unclip_ratio 2.0 and recalibrate escalation for it Supplying an explicit TextDetectionConfig suppresses OAROCR's general-text-type overrides, so the staged detector was silently running unclip 1.5 (the struct default) where the pre-split pipeline ran 2.0 — tighter box expansion that risks clipping edge glyphs. Pin unclip 2.0 explicitly and document that every override-set field must be pinned when passing an explicit config. The escalation threshold is coupled to unclip (expansion inflates measured region heights ~15-20%), so it is recalibrated from fresh measurements at 2.0: gain pages sit at 12.0-14.2px with 144+ regions, the nearest non-gaining page above the region gate at 15.7px — threshold moves 13 -> 15. A trace-level log records each page's standard-pass median to keep future recalibration cheap. Trigger tests updated to the measured values; quality re-verified end to end (GT scores match pre-split main exactly on non-escalated docs; escalation gains intact). |
||
|
Abimael Martell
|
d390d402a5 |
fix(detector): route masthead-over-scan pages to OCR (#412)
A full-page scan carrying a few text ops of genuine chrome (a newspaper masthead, stamp, or date line) defeated the alphanum_low scan heuristic — the chrome is real, diverse, decodable text — and cleared the bare min_text_ops_per_page floor (3), so page_ocr_signals called the page native while whole-document classification, whose pages_with_text floor for image-bearing pages is max(10), correctly called it scanned. The per-page and document-level paths silently disagreed (the drift #227 exists to prevent), and OCR auto mode returned almost nothing for a third of a fully scanned document. Align both per-page sites on classification's own floor: a template- image page below min_text_ops_per_page.max(10) text ops routes to OCR regardless of byte diversity, in page_ocr_signals and in the Mixed-type routing clause of detect_from_document. Tests pin the masthead case (form-wrapped full-page scan, ~4 diverse chrome ops must route) and the letterhead counter-case (a real text page over a background image must stay native). |
||
|
Abimael Martell
|
84789459b1 |
feat(extractor): stamp items with the font family name, not the resource tag (#415)
* feat(extractor): stamp items with the font family name, not the resource tag
TextItem::font carried the page's font resource name ("F2", "T22") —
an arbitrary per-page tag — even though both content-stream parsers
already resolve the /BaseFont family name for bold/italic detection at
every item-creation site. Stamp that resolved family name instead
("ABCDEF+CMMI10", "Courier"), from a single item_font_name helper so
the two parsers cannot drift.
One deliberate carve-out, documented on the helper: resource names
using Distiller's CID convention (C2_0, C0_1) are kept as-is, because
text_utils::is_cid_font keys on that prefix for micro-gap joining and
the family name carries no CID marker to replace it.
Consumers that match on font names start working against real names:
- Code detection (is_monospace_font) previously never fired against
opaque resource tags. It now does — so line classification also moves
from any-item matching to a majority-by-characters rule
(line_is_monospace): code lines are wholly monospace, while a lone
URL or identifier styled in a mono face inside a prose line must not
fence the surrounding sentence.
- Heading/body font grouping now merges resource aliases of the same
family instead of treating them as distinct fonts.
- Positioned-item output (--items-json and the bindings) reports real
face names.
Regression corpus: code-heavy manuals improve substantially (assembly
and C snippets previously emitted as prose now fence with line
structure preserved); remaining churn reviewed as improvements.
* fix(markdown): address review of font-name consumers
- Monotype is a foundry prefix on proportional faces (Monotype Corsiva,
Monotype Garamond); it must not satisfy is_monospace_font's generic
"mono" token. Regression tests pin both directions.
- Flush the pending code block before inserting a positioned table or
image, so a block that falls between two code lines cannot be emitted
ahead of code that precedes it in reading order; a code line after
the block reopens a new fence naturally.
* fix(markdown): emit sub-3-char mono fragments as plain text, not fences
A lone registered-trademark glyph or stray bullet set in a mono face is
not code; a fenced block containing one character reads as noise.
* fix(markdown): font-based code blocks open only at paragraph boundaries
HTML-to-PDF producers smear an inline code literal's mono style across
whole wrapped lines, so a prose paragraph can alternate body and mono
fonts line by line. Fencing those lines cut sentences in three: prose
head, fenced middle, prose tail. A mono-set line that continues an open
prose paragraph now stays prose; font-based blocks open at paragraph
boundaries (or continue an open block), and struct-tree Code roles are
honored unconditionally.
* refactor(markdown): drop paragraph-flush branch made unreachable by the boundary gate
The enclosing guard proves in_paragraph is false, so the nested flush
could never run; the guard and mono check collapse into one condition.
|
||
|
Abimael Martell
|
06a9bab6b3 |
fix(release): repair ARM64 package builds (#411)
Define the ARMv8 assembler macro for legacy AArch64 cross-compilers used by the npm and PyPI release matrices, and keep independent targets running when one fails.v1.15.0 |
||
|
Abimael Martell
|
ca6d667146 |
chore(release): bump package versions to 1.15.0 (#410)
Synchronize the Rust, Python, Node, platform, and WebAssembly package versions for the OCR release. |
||
|
Abimael Martell
|
a4b1c714e8 |
chore(ocr): polish launch readiness (#409)
Polish the selective OCR runtime documentation, packaging guidance, and cross-language launch smoke coverage. |
||
|
Abimael Martell
|
0f9b5fa1c6 |
feat(bindings): expose OCR in Node and Python (#405)
* feat(bindings): expose selective OCR * fix(bindings): address review feedback |
||
|
Abimael Martell
|
dba1eadf4d |
ci(ocr): validate optional feature (#404)
* ci(ocr): validate optional feature * ci(ocr): preserve test fixture line endings * ci(ocr): verify ONNX Runtime archive |
||
|
Abimael Martell
|
264a1c8372 | refactor(vision): finalize OCR API (#403) | ||
|
Abimael Martell
|
72003730ee |
fix(vision): harden OCR recovery (#402)
* fix(vision): harden OCR recovery * fix(vision): address OCR hardening review |
||
|
Abimael Martell
|
828a68c03b |
feat(vision): adaptively fuse native and OCR text (#394)
* feat(vision): adaptively fuse native and OCR text * fix(vision): preserve adaptive OCR fallbacks |
||
|
Abimael Martell
|
99069ce3d9 |
feat(vision): recover credible PDFium text layers (#393)
* feat(vision): recover credible PDFium text layers * fix(vision): validate native recovery coverage |
||
|
Abimael Martell
|
e2d2bc33d9 | fix(tables): preserve numbered reference notes (#392) | ||
|
Abimael Martell
|
926720f8ff |
perf(vision): reuse OCR runtime sessions (#391)
* perf(vision): reuse OCR runtime sessions * fix(vision): harden OCR engine caching |
||
|
Abimael Martell
|
aa3ad2e6e0 |
fix(vision): preserve OCR row boundaries (#390)
* fix(vision): preserve OCR row boundaries * fix(vision): harden OCR line recovery * fix(vision): normalize white bullet rows |
||
|
Abimael Martell
|
2cebb3c95f |
feat(vision): expose OCR pipeline (#360)
* feat(vision): expose OCR pipeline * fix(vision): harden OCR API * refactor(vision): expose OCR API |
||
|
Abimael Martell
|
7c63a00242 |
feat(vision): fuse OCR output (#359)
* feat(vision): fuse OCR output * fix(vision): make OCR fusion conservative * refactor(vision): use OCR fusion terminology |
||
|
Abimael Martell
|
e460a45f73 |
feat(vision): add OCR routing (#358)
* feat(vision): add OCR routing * fix(vision): serialize model acquisition * refactor(vision): name routed OCR results |
||
|
Abimael Martell
|
12d30b43b0 |
feat(vision): add OAR OCR engine (#357)
* feat(vision): add OAR OCR engine * fix(vision): harden OAR runtime loading * refactor(vision): use OCR engine terminology |
||
|
Abimael Martell
|
dd467dd78d |
feat(vision): add OCR contracts (#355)
* feat(vision): add OCR contracts * fix(vision): harden OCR contracts * refactor(vision): use OCR terminology |
||
|
Abimael Martell
|
d9b83993df |
feat(render): add optional PDFium page rendering (#348)
* feat(render): add optional PDFium page rendering * fix(render): honor PDFium row stride * docs(render): use OCR terminology |
||
|
Abimael Martell
|
2543abe371 |
feat(markdown): rejoin words hyphenated at line breaks (#388)
Justified print breaks words at syllables; after paragraph lines are joined
with spaces those breaks survive as "de- fendant" — thousands of them in a
long document — and, when an emphasis span was split with the word, as
"Bap-** **tist".
Whether the hyphen belongs in the word cannot be decided locally ("de-
fendant" is one word, "Third- Party" is a hyphenated compound), so the
document is used as its own dictionary. For each break:
1. fragments appear joined elsewhere ("defendant") -> join plain
2. appear hyphenated elsewhere ("six-month"), or the
continuation is capitalized ("Hinds- Radix") -> keep the hyphen
3. both fragments are words the document uses and the
continuation has 4+ letters ("commercial- type") -> keep the hyphen
4. no evidence -> leave untouched
The policy contains zero hard-coded words: vocabulary evidence,
capitalization, and two length invariants. The 4-letter floor keeps
suspended hyphens intact in any language ("mid- and long-term", "klein- und
mittelgroß", "kuva- tai video") because conjunctions are near-universally
1-3 letters. Fragments over 40 combined characters are fused reading-order
noise and are never joined. The vocabulary is collected after scrubbing the
break pairs themselves and excludes fenced code blocks; table rows and code
blocks are never rewritten. Split emphasis spans rejoin inside their
markers. Runs under the existing fix_hyphenation option (default on).
On a 1,370-page justified legal reporter this rejoins ~8,000 broken words
(98.8% of breaks; evidence-less ones stay visibly intact); word recall
against a reference extraction rises from 97.8% to 99.1%. No "six-month" ->
"sixmonth" class errors, and no fused-column corruption by construction:
no rule joins without evidence.
Regression-checked against a ~200-document corpus with semantic scoring
against an OCR baseline: zero regressions. Three in-repo fixture snapshots
regenerated with each diff inspected. 17 unit tests cover every rule, the
vocabulary scrubbing and code-block exclusion, the length gates, chained
breaks, mismatched emphasis markers, accented and Cyrillic words, German
and Finnish suspended hyphens, and the table/code skips.
|
||
|
Abimael Martell
|
7f982d2094 |
fix(markdown): veto heuristic tables made of running headers/footers (#374)
Running headers and footers repeat verbatim at the same position on many pages. When such a block wraps a long title or a navigation strip over aligned lines, the heuristic table detector reads it as a grid and emits the same pseudo-table on every page. Follow-up to #371, which noted this as a known limitation. Page furniture is a document-wide property, so the veto is computed once in to_markdown_from_items_with_rects_and_lines rather than inside the per-page detector: - an item is running furniture when its trimmed text appears at the same position (quantized to 0.5pt) on >= 3 distinct pages AND it sits in the top/bottom 20% of its page's vertical content extent — repetition alone is not enough, since a form template repeated per record carries identical labels at identical mid-page coordinates, and those are real table cells - pages whose text has no vertical span contribute no furniture keys - a heuristic table candidate is vetoed when >= 80% of its items are furniture Items are never deleted — the text flows as prose. Rect- and line-based tables are untouched: ruled structure is stronger evidence than repetition. Real tables keep per-page content under the threshold even when their header row repeats on every page, because their body rows differ. 905 unit tests pass (5 covering the furniture logic). Regression-checked against a ~200-document corpus: the overwhelming majority of outputs are byte-identical; the handful that change lose repeated header/footer pseudo-tables. Semantic scoring against an OCR baseline shows no regressions. |
||
|
Abimael MartellandCursor
|
4bee4f993b |
chore(release): bump package versions to 1.14.2 (#382)
Ship the extractor resource bounds and layout fixes that landed since 1.14.1. Co-authored-by: Cursor <cursoragent@cursor.com>v1.14.2 |
||
|
Abimael MartellandCursor
|
1719d24871 |
fix(tables): bound disjoint-rect clustering so overlap tests stay subquadratic (#381)
* fix(tables): bound disjoint-rect clustering so overlap tests stay subquadratic MAX_CLUSTER_RECTS only helped when a component actually merged. Pairwise-disjoint drawing rects never hit that cap, so the all-pairs loop stayed O(n²). Sweep by left edge and cap AABB tests at 1e6. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tables): cap clustering overlap tests per rect, not globally A page-wide AABB budget could be spent on a dense stack of disjoint drawings and never reach an independent table at a later X. Limit each rect to 256 later candidates so other X-ranges still cluster. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tables): cluster overlapping rects with a spatial grid A per-rect cap in X-sort order could skip a same-X neighbor after 256 junk candidates. Hash rects into 64-pt cells and pair only inside each cell so independent regions still cluster and disjoint drawings stay subquadratic. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tables): cluster oversized rects via a bounded fallback A span cap of 64 grid cells could omit the far end of a huge rect. Those rects now compare against every other rect (up to 32 oversized). Grid buckets are visited in sorted key order so union-find is deterministic. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tables): visit every oversized rect under a per-rect overlap budget Dropping .take(32) on the oversized-span list so later page-wide rules still union the cells they overlap. AABB tests stay capped per oversized rect. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tables): query overlapping grid cells for oversized cluster rects Index-order scans starved later overlaps once a per-rect check cap filled with disjoint drawings. Oversized spans now probe the cells they cover, with Y-banded oversized-to-oversized unions so stacked page-wide rules stay linear. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tables): range-query cluster grid cells for oversized rects Scan only occupied rows in the oversized rect's Y range, then X-partition those keys, so unrelated drawings are not visited. Band oversized-to-oversized unions on the short axis instead of a per-rect huge-Y fallback. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tables): union crossing oversized cluster rects across orientation bands Wide and tall page-spanning rules are indexed on different axes, so a crossing pair never shared a bucket. Query the tall X-index from each wide or dual-oversized rect, and insert dual-oversized spans into every coarse Y cell they cover. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tables): skip quadratic wide-by-tall clustering when the product is huge Cross-orientation union is only needed for a handful of page-spanning rules. When |wide|×|tall| exceeds the per-cell pair cap, skip that pass so mixed oversized drawings cannot go quadratic. Pair counts in a range query no longer reset per band. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tables): count unique oversized candidates when querying X/Y bands A tall rule occupying several X cells was charged once per cell against the pair budget, which could skip a later overlapping partner. Deduplicate `j` per query so the cap applies to distinct rects. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Abimael MartellandCursor
|
f114e79c8b |
fix(detector): bound Tj/TJ operand lookback to the previous operator (#380)
* fix(detector): bound Tj/TJ operand lookback to the previous operator A missing `[` before `TJ` walked the entire prefix for every operator, so a compact `] TJ` stream was quadratic. Stop each lookback at the previous text/font operator so total work stays linear. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(detector): skip strings and comments when scanning text operators A `Tj` token inside a literal string was treated as an operator and pinned the lookback floor, so the real `Tj` could not see its operand. Skip literals, hex strings, and comments before matching operators. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(detector): skip inline image data before string/hex scanning A `(` or `<` byte in `BI`/`ID` sample data could enter string or hex state and hide every later text operator. Jump from `BI` to `EI` before applying those delimiter states. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(detector): skip inline images by declared size, not the first EI Sample bytes can contain `EI` followed by a token-like character. When Width/Height are present and the image is uncompressed, jump that many bytes before looking for `EI`; DCT images use JPEG EOI, and the generic scan requires the following bytes to look like PDF content. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(detector): only trust inline-image length when the dict is complete Require Width, Height, bits-per-component, and a known color space before skipping by size; pad each row to a byte; treat image masks as 1-bit. Drop the post-EI binary heuristic so a following non-ASCII string does not hide later text operators. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(detector): keep a strict EI fallback for filtered inline images Exact-length skips still accept a following non-ASCII string. Fallback scans require printable PDF after `EI` unless the next token starts a string, name, or array. Boolean image-mask values must end at a token boundary. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(detector): stop the EI printable check at the next string token A fallback scan of `EI` then `BT (` plus high-byte text was rejected because the 16-byte window included the string payload. Count binary-ness only until `(`, `<`, `[`, or `/`. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(detector): treat Tj/TJ as operators only after a string/array closer Inline-image EI scanning cannot be made complete in this heuristic, and each attempt produced a new counterexample. Count Tj/TJ only when the previous token is `)`, `>`, or `]`: that keeps `] TJ` lookback linear and ignores `Tj` inside `(Hello Tj World)` without parsing BI/ID/EI. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Abimael MartellandCursor
|
544538b99f |
fix(extractor): bound ToUnicode bfrange expansion during subset remap (#379)
* fix(extractor): bound ToUnicode bfrange expansion during subset remap Repeated full-width beginbfrange entries were expanded into individual CID inserts on every copy. Stop after 65,536 assignments, matching the existing /W and Encoding caps. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(extractor): document bfrange remap truncation and assert visit count The 65,536 cap counts overwrites so repeated ranges cannot keep expanding. The test now checks the assignment count, not just HashMap size (u16 keys are always ≤ 65,536). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Abimael MartellandCursor
|
c8ba909407 |
fix(extractor): bound Encoding CMap cidrange expansion (#375)
* fix(extractor): bound Encoding CMap cidrange expansion Repeating full-width begincidrange declarations re-inserted the entire 16-bit domain on every copy. Stop after 65,536 assignments, matching the existing /W cap. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(extractor): clarify Encoding cidrange cap counts insert operations The bound includes overwrites so repeated full-width ranges cannot keep working after the map is full. Unique-key coverage alone would re-open the CPU blow-up. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(extractor): distinguish /W insert vs unique-key CID caps Encoding cidrange and /W width assignment count every insert; the /W unicode heuristic caps unique CIDs with the same 65,536 bound. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Abimael MartellandCursor
|
076183e2e4 |
fix(extractor): cap content-stream decode before allocating operators (#373)
* fix(extractor): cap content-stream decode before allocating operators The 1M operation limit ran after lopdf materialized the full vector, so a compact page of q/Q pairs could still abort under memory pressure. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(extractor): treat NUL and form-feed as PDF whitespace in op counting Names must stop on the full PDF whitespace set so a following operator is not absorbed into /Name, which would undercount and skip the decode cap. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(extractor): scan inline-image EI with the full PDF whitespace set A missed EI terminator used to consume the rest of the stream and drop later operators from the decode cap. If EI is absent, keep scanning. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Abimael Martell
|
ec6e54afb8 |
fix(extractor): merge small-caps runs so they stop reading as table columns (#371)
* fix(xobjects): track text line matrix and handle T*/TL/'/"/Tc/Tw in Form XObjects The Form XObject text extractor in xobjects.rs is a separate hand-rolled implementation of the operator state machine in content_stream.rs, and it had drifted well out of parity: - No text line matrix (TLM). `Td`/`TD` were applied to the text matrix already advanced by `Tj`/`TJ`, so every line began where the previous line *ended* instead of at the line start. Lines marched off the right edge and were dropped as off-page. - `T*` was not handled at all, so it never advanced to the next line. - `TL`, `'` and `"` were missing, and `TD` never set the leading as a side effect. - `Tc`/`Tw` were hardcoded to 0.0 when computing advance widths, drifting positions and inserting spurious spaces. - Text state (Tc/Tw/TL/Tf) is part of the graphics state but was not saved or restored by `q`/`Q`. This matters well beyond an edge case: producers that emit a page stream of just `q /X Do Q` and put all content in a Form XObject are common in print-to-PDF and typesetting workflows, so this parser is on the hot path for whole classes of real documents. Measured on nycourts.gov 199AD3d.pdf (1370 pages, PDFlib producer, every page wrapped in a Form XObject, 1331 pages using T*), word recall against a pdftotext reference goes from 19.2% to 97.7% — 116k extracted words to 582k against a 578k-word reference. On a 10-page subset, sequence similarity goes from 27.3% to 97.7%, against 99.8% for Mistral OCR. pdf-evals: 195 passed / 7 failed, byte-identical to the origin/main baseline with the same failure list — no regressions. Adds 7 unit tests covering the line-matrix-relative `Td`, `T*`, `TD` setting leading, `'`, `"`, `Tc` advance widths, and `q`/`Q` text-state restore, all driven through a page whose content is only `q /X1 Do Q`. * fix(xobjects): restore fill colour across q/Q in Form XObjects A white fill set inside a q/Q pair leaked past the Q, so all subsequent text was treated as invisible and dropped. Save and restore fill_is_white with the rest of the graphics state. This was the cause of several long-standing extraction failures where whole passages went missing or degraded into per-character garbage: cambridge_excerpt (+8.5KB of recovered text), MTUAeroEngines (+7.4KB), 2025_findings-acl_668 (+1.4KB), HTM_02-01_Part_A (+1.3KB), and HuttoISDWorkPerks / ebgt7isj04ophcq, which both went from exploded per-character tables to clean prose. Adds a regression test that fails without the restore (the text after Q is dropped entirely). Reported by cubic on #369. * fix(extractor): merge small-caps runs so they stop reading as table columns Typesetters render small caps as a full-size capital immediately followed by shrunken capitals in the same font — `(R) Tj` at 9.98pt, then `(OLANDO) Tj` at 6.74pt, touching. The 20% font-size band in merge_text_items split those into separate items, and since table column boundaries cluster on item *start* positions (find_column_boundaries never consults widths), each fragment started far enough from the last to become its own column. The result was garbled pseudo-tables. From 199AD3d.pdf p.5: |OLANDO|COSTA|NIL|INGH|| |---|---|---|---|---| |IANNE|ENWICK|ETER|OULTON|| |R|T. A|, P.J.|A|C. S| |---|---|---|---|---| now: |ROLANDO T. ACOSTA, P.J.|ANIL C. SINGH| |---|---| |DIANNE T. RENWICK|PETER H. MOULTON| Adds is_small_caps_continuation, gated tightly enough to exclude the other reasons a smaller run follows a larger one: superscripts and footnote markers (requires an uppercase *letter*, so digits never qualify), drop caps (the following body text is mixed case), and adjacent cells or separate words (requires the runs to be visually contiguous). A small-caps junction is mid-word, so it also suppresses the space that would otherwise be inserted ("T. A" + "COSTA" -> "T. ACOSTA", not "T. A COSTA"). On 199AD3d.pdf this drops false-positive tables from 65 to 52 and lifts word recall against a pdftotext reference from 97.7% to 97.8%. Verified by running both binaries over all 203 eval PDFs and diffing outputs: 19 differ, 184 byte-identical. Beyond the reporter volume the merge also fixes: Waters-Edge two more garbled heading-tables — "##### B. C R S" plus "||OMPLIANCE|EPORTING YSTEM|" became "##### B. COMPLIANCE REPORTING SYSTEM" cn-student-handbook six TOC entries had collapsed to their initials; "U M S H..." is now "UNIVERSITY MISSION AND STUDENT HANDBOOK..." 546403 "(FAA)" + "ADVISORY CIRCULARS ( )" + a stray "CONT" became "(FAA) ADVISORY CIRCULARS (CONT)" ERP-2025 "T ABLE B-1" -> "TABLE B-1" DMP-Keypad "THINLINE" + stray "TM KEYPADS" -> "THINLINETM KEYPADS" zhaw / Stijn subscripted math variables: "*T* *G*" -> "*TG*" One known regression, called out rather than hidden: PA_PVEM_Sen_Waldo_Fernández (+1689 bytes). Its running footer genuinely is small caps, so the merge correctly assembles the text (400 items -> 275), but the now-contiguous footer lines align well enough that the heuristic detector turns the wrapped document title into a 4-column table where it previously rendered as bold prose. Attempting to fix that in the detector was a dead end and is not included here: gating the `num_cols >= 3` bypass in has_table_like_content on "no cell has >= 12 words" removed 143 tables across 44 files, including correct ones — MCF5235RM went 505 -> 483 and its register glossary degraded into a fused column plus a ~100-word cell, because rejecting a good candidate lets a worse fallback win. No wordiness threshold separates the two cases: PA_PVEM's longest cell is 14 words while MCF5235RM's legitimate cells are <= 10. A positional signal (running-header/footer bands, or cross-page repetition) is the way in, and belongs in its own change. Adds 7 unit tests: the two-column small-caps row from the reporter volume, and rejection of superscript digits, drop caps, separate words, lowercase continuations, and out-of-band size ratios. * fix(extractor): tighten small-caps gate to cross-band junctions only Three findings from cubic on #371, all valid. 1. Space suppression was too broad. `is_small_caps_continuation` accepted size ratios up to 0.92, which is *inside* the 20% band that merge_text_items already treats as the same size. Two similarly-sized uppercase words with a small real word gap therefore had their space suppressed even though the normal path would have merged them correctly with a space ("SEE" + "ALSO" -> "SEEALSO"). The helper now requires the junction to *cross* the band, since rescuing junctions the band would break is its only purpose; within-band pairs keep the normal word-spacing logic. The band is now a shared MERGE_FONT_SIZE_BAND constant so the helper and its caller cannot drift. 2. A trailing digit was skipped. The backward search for "the capital we are continuing" skipped non-alphabetic characters, so text ending in a footnote marker ("ANGELA M. MAZZARELLI1") found the earlier uppercase letter and accepted the join. It now checks the actual trailing character. Rejecting every trailing digit outright turned out to cost real quality, so this keeps one narrow exception. In meetings_in_mass_july_2023 the source reads "TUESDAY, JULY 4TH" with the ordinal suffix set as a smaller run; a blanket digit rejection reverted that to "TUESDAY, JULY 4" plus a stray "TH" leaking onto the next line, which is what main produces and what pdftotext shows is wrong. Only the four English ordinal suffixes (TH/ST/ND/RD) may follow a digit; anything else after one is treated as a footnote marker and rejected, so the case cubic raised stays blocked. 3. A test's name did not match its data. `small_caps_merge_does_not_swallow_a_ second_column` claimed to exercise the 72pt column gap but contained only the second column's items, so it just re-tested the happy-path merge. It now holds the full nine-item row and asserts exactly two merged results, "ROLANDO T. ACOSTA, P.J." and "ANIL C. SINGH", which genuinely exercises the gap. Both new guards were verified to be load-bearing: removing either one makes its test fail. The document that motivated the change is unaffected — small caps there run at a 0.675 ratio, far outside the band — and 199AD3d.pdf p.5 still produces the correct two-column justices table. 900 unit tests pass (10 covering small caps), fmt and clippy clean. |
||
|
Abimael Martell
|
89dd20d02c |
fix(xobjects): track text line matrix and handle T*/TL/'/"/Tc/Tw in Form XObjects (#369)
* fix(xobjects): track text line matrix and handle T*/TL/'/"/Tc/Tw in Form XObjects The Form XObject text extractor in xobjects.rs is a separate hand-rolled implementation of the operator state machine in content_stream.rs, and it had drifted well out of parity: - No text line matrix (TLM). `Td`/`TD` were applied to the text matrix already advanced by `Tj`/`TJ`, so every line began where the previous line *ended* instead of at the line start. Lines marched off the right edge and were dropped as off-page. - `T*` was not handled at all, so it never advanced to the next line. - `TL`, `'` and `"` were missing, and `TD` never set the leading as a side effect. - `Tc`/`Tw` were hardcoded to 0.0 when computing advance widths, drifting positions and inserting spurious spaces. - Text state (Tc/Tw/TL/Tf) is part of the graphics state but was not saved or restored by `q`/`Q`. This matters well beyond an edge case: producers that emit a page stream of just `q /X Do Q` and put all content in a Form XObject are common in print-to-PDF and typesetting workflows, so this parser is on the hot path for whole classes of real documents. Measured on nycourts.gov 199AD3d.pdf (1370 pages, PDFlib producer, every page wrapped in a Form XObject, 1331 pages using T*), word recall against a pdftotext reference goes from 19.2% to 97.7% — 116k extracted words to 582k against a 578k-word reference. On a 10-page subset, sequence similarity goes from 27.3% to 97.7%, against 99.8% for Mistral OCR. pdf-evals: 195 passed / 7 failed, byte-identical to the origin/main baseline with the same failure list — no regressions. Adds 7 unit tests covering the line-matrix-relative `Td`, `T*`, `TD` setting leading, `'`, `"`, `Tc` advance widths, and `q`/`Q` text-state restore, all driven through a page whose content is only `q /X1 Do Q`. * fix(xobjects): restore fill colour across q/Q in Form XObjects A white fill set inside a q/Q pair leaked past the Q, so all subsequent text was treated as invisible and dropped. Save and restore fill_is_white with the rest of the graphics state. This was the cause of several long-standing extraction failures where whole passages went missing or degraded into per-character garbage: cambridge_excerpt (+8.5KB of recovered text), MTUAeroEngines (+7.4KB), 2025_findings-acl_668 (+1.4KB), HTM_02-01_Part_A (+1.3KB), and HuttoISDWorkPerks / ebgt7isj04ophcq, which both went from exploded per-character tables to clean prose. Adds a regression test that fails without the restore (the text after Q is dropped entirely). Reported by cubic on #369. |
||
|
Abimael MartellandCursor
|
3d33ff3dbd |
fix(extractor): bound CID /W range expansion (#372)
Type0 /W parsing and the Unicode-CID heuristic expanded every CID in every range. Repeating a full-width [0 65535 w] entry therefore re-materialized the same 65,536-key domain on every copy, growing a temporary vector and HashMap work without bound. Cap expansion at the 16-bit CID domain, collect unique CIDs for the median heuristic, and stop width assignment once that many entries have been written. Legitimate compact /W arrays are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Abimael MartellandCursor
|
75e9b09593 |
fix(extractor): bound Form XObject expansion per page (#370)
* fix(extractor): bound Form XObject expansion with invocation and operation budgets Nested Form XObjects were only limited by recursion depth (5). An acyclic graph where each form invokes the next N times still expands to N^depth work, so a small PDF can force millions of nested /Do evaluations. Share a per-page FormWalkBudget that caps 10,000 Form invocations and 1,000,000 operations walked across those expansions. Extraction stops when either cap is hit. Legitimate shallow nesting is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(extractor): share Form XObject budget across invisible-layer retry extract_page_text_items created a fresh FormWalkBudget on every call, so the invisible-text retry could consume a second full expansion budget for the same page. Own the budget at the call site and pass it into both passes. Also charge form operations independently of the invocation cap so a form that was already admitted can finish its stream. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Abimael MartellandClaude Fable 5
|
f4aab3b36f |
chore(release): bump package versions to 1.14.1 (#354)
Releases fix(regions) #351 — invisible (Tr 3) OCR text layers served from the region extractor instead of falling back to GPU OCR. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Abimael MartellandClaude Fable 5
|
f65b25906c |
fix(regions): serve invisible (Tr 3) OCR text layers instead of needs_ocr (#351)
* fix(regions): serve invisible (Tr 3) OCR text layers instead of needs_ocr Scanned pages (archive.org-style digitizations) carry their text as an invisible render-mode-3 layer behind the page raster. extract_text_in_regions extracted only visible items, so such pages yielded nothing but '[Image: ...]' placeholders and every region fell back to OCR — while the markdown path already includes the invisible layer for Mixed PDFs. The two extractors disagreed about the same page. Mirror the markdown path's gate, page-scoped: when the visible pass is effectively textless (<40 non-placeholder alphanumerics), retry with invisible text included and adopt the retry only when it contributes real, non-garbage text. Pages with real visible text never retry, so double-layer PDFs (visible text plus an invisible accessibility copy) are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: strict zero-visible adoption gate, whole-layer garbage check, doc placement, discriminating tests - Adoption now requires ZERO visible text on the page: the invisible pass returns visible items too, so adopting alongside any visible text would duplicate it. Strict gate instead of fuzzy dedupe; the dead inv_alnum > visible_alnum condition goes with it. - Garbage check judges the whole recovered layer, not the first 200 items. - Constant/helper moved above the doc block so rustdoc stays attached to extract_text_in_regions_mem. - Tests: any-visible-text-blocks-adoption case (single short visible line) with exactly-once assertions; mode-0 guard asserts occurrence count. - Re-verified against real scanned-book pages: all recover (9.2-11.7K chars, needs_ocr=false) — pure OCR-layer scans carry no visible text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: gate adoption on visible-item presence, not alphanumeric mass Punctuation-only visible text (zero alphanumerics) could still adopt the invisible pass; its OCR twin in the layer would duplicate the glyphs. The gate is now item-presence: any non-image item with non-whitespace text blocks adoption (whitespace-only artifacts still tolerated). Pinned by a punctuation-only fixture test; real scanned-book pages re-verified — all recover unchanged (they carry no visible items at all). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: punctuation-gate test pins preservation, comment states fixture scope The fixture's visible punctuation is a separate block, not mirrored in the invisible layer — so it pins the gate, not the duplication scenario. The doc comment now says so, and a positive assertion checks the punctuation survives exactly once. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: retry only when invisible text was actually skipped; negative gate tests - extract_page_text_items now reports skipped_invisible (4th return): the Tr-3 suppression sites set it, so the region extractor retries only when a recoverable layer exists. Blank pages and image-only scans without an OCR layer — the common scanned case — no longer pay a second content-stream parse. - Negative tests: below-floor watermark layer and symbol-garbage layer are both rejected (region keeps only the raster placeholder). needs_ocr semantics for placeholder-only regions deliberately unchanged — that contract predates this PR and downstream pipelines handle it. - Real scanned-book pages re-verified: recovery unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: flag the ' show-text suppression path; require nonempty strings - P1: invisible layers shown via the ' operator never set skipped_invisible, so those pages kept the placeholder and fell back to GPU OCR. The ' suppression path now flags too — pinned by a fixture whose layer is shown entirely via ' (nothing through Tj/TJ). - P3: all three flag sites (Tj, TJ, ') require a nonempty string operand — numeric-only TJ kerning arrays and empty shows no longer trigger the invisible reparse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Abimael Martell
|
9947485a92 | docs: document structure-element extraction and TextItem.mcid (#349) | ||
|
Abimael Martell
|
7054d6aa69 |
chore(release): unify package versions (#344)
* chore(release): unify package versions * fix(release): harden version synchronization |
||
|
Abimael MartellandClaude Fable 5
|
a67ee03269 |
feat(bindings): expose TextItem.mcid and structure-tree element extraction (#346)
Tagged PDFs carry a structure tree with real heading roles (H1..H6), and the core already parses it (structure_tree::StructTree) and threads MCIDs onto TextItem — but neither surfaced through the bindings. - Expose TextItem.mcid (Option<i64>) through the napi and pyo3 bindings, matching the core field added with the marked-content extractor. - Add StructRole::name(), the inverse of from_name, so roles have a stable string form. - Add extract_structure_elements / extract_structure_elements_mem to the core: one (page, mcid, role) entry per marked-content reference, sorted by (page, mcid), empty for untagged PDFs. Pages are 1-indexed to match TextItem.page, so results join directly against extract_text_with_positions output. - Bind it as extractStructureElements (napi) and extract_structure_elements / extract_structure_elements_bytes (pyo3), with type-stub updates in pdf_inspector.pyi. - Cover the join in Rust integration tests, napi test.mjs, and pytest, using the existing firecrawl_docs_tagged.pdf fixture (tagged) and thermo-freon12.pdf (untagged). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Abimael Martell
|
965dc65f1b | chore(release): bump package versions (#343) packages-2026-08-10 | ||
|
|
36dd5fa426 |
fix(structure-tree): bound recursive /K parsing with cycle detection and a node budget (#322)
* fix(structure-tree): bound tagged /K parsing against alias/cycle DoS A struct element that references itself (or an ancestor) through /K — e.g. /K [n 0 R n 0 R] — made parse_struct_element_dict branch exponentially: the depth cap (64) alone still permits 2^depth materialized nodes, so a ~830-byte PDF exhausts memory (OOM, exit 134). Add a StructWalk carrying (1) an active-path set of object IDs so a node that references itself/an ancestor is not re-expanded (breaks self- and mutual-reference cycles cheaply), and (2) a global node budget (MAX_STRUCT_NODES) that caps total materialization for aliased/DAG-shaped graphs of distinct objects the path guard cannot catch. Adds regression tests for self-alias, mutual-alias, and the aliased-DAG budget cap. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): charge /K content refs against the node budget The per-node budget only covered materialized struct elements and child recursion; bare MCIDs and MCR dicts in a /K array append to content_refs without charging it, so one element with a very wide /K array could still allocate content_refs without bound. Charge every /K array item before handling it, and stop the top-level /K loop once the budget is spent, so content refs and loop work are bounded too. Adds a wide-MCID-array test. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): charge /K budget per materialized item, not per array entry Charging every /K array item double-counted structural children (charged here and again at their node entry) and charged cycle-skipped references that materialize nothing, draining the budget up to ~2x faster than the per-node semantics and risking early truncation of large legitimate trees. Charge only the unbounded content-ref items (bare MCIDs and MCR dicts); structural children remain charged once at their node entry. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * refactor(structure-tree): charge every content ref uniformly via helper Route all budget charges through StructWalk::charge() so every marked-content reference is charged once, including the single-value /K branches (bare integer and MCR dict) that previously appended without charging. charge() also guards against underflow, so charging after the node-entry charge (which can leave the budget at 0) is safe. Makes the documented per-item budget contract hold uniformly across all branches. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * feat(structure-tree): log once when the node budget truncates parsing Add a one-shot truncation flag on StructWalk, set the first time the budget is exhausted, and emit a single warn! after parsing so an operator can tell when a (very large or malformed) tagged tree was cut off. Avoids per-item log spam; negligible overhead on the normal path. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): flag truncation at budget guards, not just in charge The truncation flag was only set inside charge() on the budget==0 branch, but the dominant skip paths use budget==0 guards that break/return before charge() is ever called with an empty budget, so the flag (and the warn!) almost never fired. Route those guards through a new exhausted() that sets the flag when it skips remaining work. Adds a parser-level test that would have caught the missed warning. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): flag cycle/depth skips and charge bare MCIDs fully Two review follow-ups: - Cycle-broken and depth-capped /K skips dropped tagged content without setting the truncation flag, so the one-shot warning never fired for malformed/over-deep trees. Mark those skips via note_skipped() and broaden the warning to cover non-budget truncation. - A bare /K MCID materializes a wrapper node AND a content reference but charged only one budget unit, allowing ~2x the advertised budget for such content; charge both. Adds tests: cycle-skip flags truncation, and bare MCID charges two units. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): charge MCR-dict wrappers the same two units as bare MCIDs A top-level MCR /K dict flows through parse_kid -> parse_struct_element_dict and materializes a Span node + one content ref (two items) but was charged only one unit at node entry, while the bare-MCID path charges two. Charge the content reference in the MCR branch too so the per-item budget is uniform across both wrapper paths. Adds a symmetric test. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): reserve leaf-wrapper budget units atomically A leaf MCID wrapper (bare MCID or MCR dict) materializes a node + one content ref and charged the two units via separate charge() calls. At the last unit the first charge succeeded and the second failed, consuming a unit without emitting the wrapper and denying it to a later element that would have fit. Add charge_n() to reserve both units atomically (or neither), and detect MCR before the node charge so it reserves both up front. Adds a boundary test asserting the leftover unit is preserved. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): stop scanning wide /K once a leaf reservation stalls The atomic charge_n(2) left budget nonzero (==1) when it failed, so exhausted() (budget==0) never broke the root /K loop and a crafted wide array of leaf wrappers was scanned in full after no leaf could fit. Add a stalled flag set on an insufficient reservation and fold it into exhausted(); charge()-based (one-unit) loops are unaffected since they reach budget 0 exactly. Adds a test that a one-unit budget still allows a one-unit item but a failed two-unit reservation stops the scan. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(structure-tree): add traversal budget and stop charging non-materializing dicts Two review follow-ups on budget accounting: - Wide /K arrays of non-materializing items (unsupported value types, OBJR dicts, cycle back-edges) consumed no node budget, so the loop scanned the whole array. Add a separate work budget charged per examined /K item and break the loops when it is spent, bounding traversal even when nothing materializes. - OBJR dicts and dicts without a valid /S were charged the node budget before being recognized and skipped, draining the shared budget and truncating real content later. Hoist the OBJR check and /S validation above the node charge so only materializing nodes consume it (matching the MCR hoisting). Adds tests for the work-budget bound, wide unsupported /K, and non-materializing dicts not charging the node budget. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * docs(structure-tree): mention traversal budget in truncation warning The one-shot truncation warning listed the node budget, cycle, and depth as causes but not the new traversal (work) budget, so a work-budget truncation printed a misleading message. Include MAX_STRUCT_WORK so malformed-PDF debugging identifies the actual limit hit. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |
||
|
|
fabec0aec3 |
feat(napi): add async variants that keep the Node event loop free (#337)
* feat(napi): add processPdfAsync, classifyPdfAsync, extractPagesMarkdownAsync The Node bindings are synchronous, so every call parses on the event loop thread — up to hundreds of milliseconds of dead loop per document in a server. Add additive AsyncTask-based variants that run the same shared implementations on the libuv thread pool and return promises. The existing synchronous exports keep their names, signatures, and behaviour; each sync/async pair shares one implementation. Panics in compute() are caught and surfaced as rejections, matching the sync error contract. Closes #336 Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(napi): read async task buffers in place instead of copying Review feedback on #337: buffer.to_vec() copied the whole PDF on the event loop before the task was queued, so large inputs still stalled the loop and doubled peak memory. The tasks now hold the napi Buffer itself — its ref pins the JS allocation for the task's lifetime and the backing store is stable, so compute() reads it directly from the worker thread. Callers must not mutate the buffer until the promise settles (same contract as Node's async fs APIs); documented on each export and in the README. The suggested removal of ts_return_type was checked and rejected: without it napi-rs generates Promise<unknown> for AsyncTask returns. A comment now records that finding. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(napi): copy async task input on the JS thread for soundness Review feedback on #337: holding the napi Buffer and reading it from the libuv worker was unsound. Buffer derefs straight to the JS-side allocation, so a caller mutating it before the promise settled would race the worker's reads — undefined behavior, not a recoverable error, and the documented don't-mutate contract was unenforceable. Deferring the copy to compute() would not help: any off-thread read races the same way. The JS thread is the only race-free place to take the copy, because JS is single-threaded and nothing can mutate the buffer during the synchronous part of the call. Revert to an owned Vec<u8> copied at call time. The cost is one memcpy, negligible next to the parse the async variants exist to unblock. Docs now state the buffer may be reused or mutated immediately, and a test locks in the copy semantics by mutating the input while a parse is in flight. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |
||
|
|
1f28c00a13 |
Bound column-detection histogram and harden coordinate handling (#328)
* Bound column-detection histogram size Derive the projection histogram from a clamped bin count and skip non-finite page widths. Extreme or malformed text-item coordinates (from the content-stream text matrix) could otherwise drive a very large allocation. 65,536 bins is ~9x the largest legal page, so real layouts are unaffected. Adds regression tests. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Exclude non-finite coordinates from page bounds Items at NaN/inf positions are now skipped when folding the page bounds, so a malformed coordinate can no longer escape as a ColumnRegion boundary, and an all-non-finite page returns no columns. Bad items are dropped individually rather than failing the page, so one stray glyph does not disable column detection. Addresses review feedback on the finite-width guard. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Trim far-outlier coordinates from page bounds Gutter margins, spanning-item width and the XY-cut margin are all fractions of page_width, so a single far-but-finite item (x=50_000 is enough) set the scale for the whole page: real gutters fell inside the rejected margin band and a genuine two-column page collapsed to one region. When the span exceeds one legal page (14_400 units), re-derive the bounds from items clustered around the median x. Outliers keep their text because column assignment buckets by nearest overlap. The MAX_BINS ceiling stays as an allocation bound that does not depend on this heuristic. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Harden bounds trimming against widths and wide layouts Check both item edges when trimming: a malformed width at an ordinary position poisoned x_max just as a malformed position poisoned x_min, so a huge width still collapsed a two-column page to one region. Only trim when the far items are a small minority (<=10%). A genuinely large-format page has content spread across its full width, so it now keeps its true bounds instead of being reduced to the median cluster. Correct the MAX_PAGE_EXTENT comment: 14_400 units is the traditional Acrobat architectural limit, not a format cap. PDF 2.0 sets no page-size limit and UserUnit scales physical size, so this is a heuristic. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Scale bin width so the histogram spans the whole page Clamping the bin count alone left anything past MAX_BINS * BIN_WIDTH (~131k points) outside the histogram, folded into the final bin. A page wide enough to hit that lost real gutters: with a visible gutter inside the covered range the XY-cut fallback never runs, so a three-column layout silently reported two. Derive bin_width from page_width instead, keeping the same allocation ceiling and degrading only resolution. Also anchor the trimming median on the same finite left/right items that bounds() accepts, so a malformed width cannot shift which items count as strays. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Require geometric evidence before detaching far content The median-window trim narrowed any content more than one page from the centre, so a valid large page with a sparse far sidebar lost the sidebar from its bounds and its text fell into column 0. An item-count minority rule cannot tell that layout from malformed coordinates. Group content into clusters separated by more than a whole page of continuous emptiness, and only drop a cluster that is both detached by such a void and a small minority of items. Real content does not leave a gap that large; a stray coordinate sits alone beyond one. A single run wider than one page is treated as a malformed width, which also covers the huge-width case the cluster sweep cannot see (such an item spans everything and leaves no gap). Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Judge run width against page content, not a fixed extent Treating any run wider than 14_400 units as malformed penalised valid large pages: one made entirely of such runs reported no columns at all, and a mixed page lost the right edge of every long run. Judge width relative to the page's own content instead. Positions cannot be inflated by a bogus width, so the spread of the core cluster is a sound scale: a run wider than that spread plus one page is malformed. A genuinely large page keeps its genuinely long runs, while a 1e12-wide run beside ordinary text is still rejected. Cluster on positions rather than filled intervals, so a bogus width can no longer merge everything into one cluster, and keep ordinary pages on an O(n) fast path that skips the sort entirely. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |
||
|
|
f4b8c9e854 |
Clarify SECURITY.md reporting channels (#329)
* Update SECURITY.md reporting channels Clarify that email is the only required channel and point the alternative at Firecrawl's Bugcrowd disclosure engagement instead of the private-advisory link, which is not enabled on this repo. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * Make Bugcrowd the preferred reporting channel Bugcrowd's disclosure engagement is the primary channel; email to help@firecrawl.dev is offered as the alternative. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |
||
|
|
69039f2728 |
Fix char-boundary panic in hex_to_unicode_string (#320)
Use hex.get(i..i+2) instead of &hex[i..i+2] so a non-hex, non-ASCII destination in a /ToUnicode CMap can no longer trigger a UTF-8 char-boundary panic. An even byte length does not guarantee the byte offset falls on a char boundary; get() returns None on a non-boundary or out-of-range index, folding cleanly into the existing flow. Add regression tests covering a multi-byte destination char and a replacement-char byte. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> |