* fix(extractor): cap content-stream decode before allocating operators
The 1M operation limit ran after lopdf materialized the full vector, so a
compact page of q/Q pairs could still abort under memory pressure.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): treat NUL and form-feed as PDF whitespace in op counting
Names must stop on the full PDF whitespace set so a following operator is
not absorbed into /Name, which would undercount and skip the decode cap.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): scan inline-image EI with the full PDF whitespace set
A missed EI terminator used to consume the rest of the stream and drop
later operators from the decode cap. If EI is absent, keep scanning.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(xobjects): track text line matrix and handle T*/TL/'/"/Tc/Tw in Form XObjects
The Form XObject text extractor in xobjects.rs is a separate hand-rolled
implementation of the operator state machine in content_stream.rs, and it had
drifted well out of parity:
- No text line matrix (TLM). `Td`/`TD` were applied to the text matrix already
advanced by `Tj`/`TJ`, so every line began where the previous line *ended*
instead of at the line start. Lines marched off the right edge and were
dropped as off-page.
- `T*` was not handled at all, so it never advanced to the next line.
- `TL`, `'` and `"` were missing, and `TD` never set the leading as a side
effect.
- `Tc`/`Tw` were hardcoded to 0.0 when computing advance widths, drifting
positions and inserting spurious spaces.
- Text state (Tc/Tw/TL/Tf) is part of the graphics state but was not saved or
restored by `q`/`Q`.
This matters well beyond an edge case: producers that emit a page stream of
just `q /X Do Q` and put all content in a Form XObject are common in
print-to-PDF and typesetting workflows, so this parser is on the hot path for
whole classes of real documents.
Measured on nycourts.gov 199AD3d.pdf (1370 pages, PDFlib producer, every page
wrapped in a Form XObject, 1331 pages using T*), word recall against a
pdftotext reference goes from 19.2% to 97.7% — 116k extracted words to 582k
against a 578k-word reference. On a 10-page subset, sequence similarity goes
from 27.3% to 97.7%, against 99.8% for Mistral OCR.
pdf-evals: 195 passed / 7 failed, byte-identical to the origin/main baseline
with the same failure list — no regressions.
Adds 7 unit tests covering the line-matrix-relative `Td`, `T*`, `TD` setting
leading, `'`, `"`, `Tc` advance widths, and `q`/`Q` text-state restore, all
driven through a page whose content is only `q /X1 Do Q`.
* fix(xobjects): restore fill colour across q/Q in Form XObjects
A white fill set inside a q/Q pair leaked past the Q, so all subsequent
text was treated as invisible and dropped. Save and restore fill_is_white
with the rest of the graphics state.
This was the cause of several long-standing extraction failures where
whole passages went missing or degraded into per-character garbage:
cambridge_excerpt (+8.5KB of recovered text), MTUAeroEngines (+7.4KB),
2025_findings-acl_668 (+1.4KB), HTM_02-01_Part_A (+1.3KB), and
HuttoISDWorkPerks / ebgt7isj04ophcq, which both went from exploded
per-character tables to clean prose.
Adds a regression test that fails without the restore (the text after Q
is dropped entirely).
Reported by cubic on #369.
* fix(extractor): merge small-caps runs so they stop reading as table columns
Typesetters render small caps as a full-size capital immediately followed by
shrunken capitals in the same font — `(R) Tj` at 9.98pt, then `(OLANDO) Tj`
at 6.74pt, touching. The 20% font-size band in merge_text_items split those
into separate items, and since table column boundaries cluster on item *start*
positions (find_column_boundaries never consults widths), each fragment
started far enough from the last to become its own column.
The result was garbled pseudo-tables. From 199AD3d.pdf p.5:
|OLANDO|COSTA|NIL|INGH||
|---|---|---|---|---|
|IANNE|ENWICK|ETER|OULTON||
|R|T. A|, P.J.|A|C. S|
|---|---|---|---|---|
now:
|ROLANDO T. ACOSTA, P.J.|ANIL C. SINGH|
|---|---|
|DIANNE T. RENWICK|PETER H. MOULTON|
Adds is_small_caps_continuation, gated tightly enough to exclude the other
reasons a smaller run follows a larger one: superscripts and footnote markers
(requires an uppercase *letter*, so digits never qualify), drop caps (the
following body text is mixed case), and adjacent cells or separate words
(requires the runs to be visually contiguous). A small-caps junction is
mid-word, so it also suppresses the space that would otherwise be inserted
("T. A" + "COSTA" -> "T. ACOSTA", not "T. A COSTA").
On 199AD3d.pdf this drops false-positive tables from 65 to 52 and lifts word
recall against a pdftotext reference from 97.7% to 97.8%.
Verified by running both binaries over all 203 eval PDFs and diffing outputs:
19 differ, 184 byte-identical. Beyond the reporter volume the merge also fixes:
Waters-Edge two more garbled heading-tables — "##### B. C R S" plus
"||OMPLIANCE|EPORTING YSTEM|" became
"##### B. COMPLIANCE REPORTING SYSTEM"
cn-student-handbook six TOC entries had collapsed to their initials;
"U M S H..." is now
"UNIVERSITY MISSION AND STUDENT HANDBOOK..."
546403 "(FAA)" + "ADVISORY CIRCULARS ( )" + a stray "CONT"
became "(FAA) ADVISORY CIRCULARS (CONT)"
ERP-2025 "T ABLE B-1" -> "TABLE B-1"
DMP-Keypad "THINLINE" + stray "TM KEYPADS" -> "THINLINETM KEYPADS"
zhaw / Stijn subscripted math variables: "*T* *G*" -> "*TG*"
One known regression, called out rather than hidden:
PA_PVEM_Sen_Waldo_Fernández (+1689 bytes). Its running footer genuinely is
small caps, so the merge correctly assembles the text (400 items -> 275), but
the now-contiguous footer lines align well enough that the heuristic detector
turns the wrapped document title into a 4-column table where it previously
rendered as bold prose.
Attempting to fix that in the detector was a dead end and is not included
here: gating the `num_cols >= 3` bypass in has_table_like_content on "no cell
has >= 12 words" removed 143 tables across 44 files, including correct ones —
MCF5235RM went 505 -> 483 and its register glossary degraded into a fused
column plus a ~100-word cell, because rejecting a good candidate lets a worse
fallback win. No wordiness threshold separates the two cases: PA_PVEM's
longest cell is 14 words while MCF5235RM's legitimate cells are <= 10. A
positional signal (running-header/footer bands, or cross-page repetition) is
the way in, and belongs in its own change.
Adds 7 unit tests: the two-column small-caps row from the reporter volume, and
rejection of superscript digits, drop caps, separate words, lowercase
continuations, and out-of-band size ratios.
* fix(extractor): tighten small-caps gate to cross-band junctions only
Three findings from cubic on #371, all valid.
1. Space suppression was too broad. `is_small_caps_continuation` accepted size
ratios up to 0.92, which is *inside* the 20% band that merge_text_items
already treats as the same size. Two similarly-sized uppercase words with a
small real word gap therefore had their space suppressed even though the
normal path would have merged them correctly with a space ("SEE" + "ALSO" ->
"SEEALSO"). The helper now requires the junction to *cross* the band, since
rescuing junctions the band would break is its only purpose; within-band
pairs keep the normal word-spacing logic. The band is now a shared
MERGE_FONT_SIZE_BAND constant so the helper and its caller cannot drift.
2. A trailing digit was skipped. The backward search for "the capital we are
continuing" skipped non-alphabetic characters, so text ending in a footnote
marker ("ANGELA M. MAZZARELLI1") found the earlier uppercase letter and
accepted the join. It now checks the actual trailing character.
Rejecting every trailing digit outright turned out to cost real quality, so
this keeps one narrow exception. In meetings_in_mass_july_2023 the source
reads "TUESDAY, JULY 4TH" with the ordinal suffix set as a smaller run; a
blanket digit rejection reverted that to "TUESDAY, JULY 4" plus a stray "TH"
leaking onto the next line, which is what main produces and what pdftotext
shows is wrong. Only the four English ordinal suffixes (TH/ST/ND/RD) may
follow a digit; anything else after one is treated as a footnote marker and
rejected, so the case cubic raised stays blocked.
3. A test's name did not match its data. `small_caps_merge_does_not_swallow_a_
second_column` claimed to exercise the 72pt column gap but contained only
the second column's items, so it just re-tested the happy-path merge. It now
holds the full nine-item row and asserts exactly two merged results,
"ROLANDO T. ACOSTA, P.J." and "ANIL C. SINGH", which genuinely exercises the
gap.
Both new guards were verified to be load-bearing: removing either one makes its
test fail.
The document that motivated the change is unaffected — small caps there run at
a 0.675 ratio, far outside the band — and 199AD3d.pdf p.5 still produces the
correct two-column justices table.
900 unit tests pass (10 covering small caps), fmt and clippy clean.
* fix(extractor): bound Form XObject expansion with invocation and operation budgets
Nested Form XObjects were only limited by recursion depth (5). An acyclic
graph where each form invokes the next N times still expands to N^depth
work, so a small PDF can force millions of nested /Do evaluations.
Share a per-page FormWalkBudget that caps 10,000 Form invocations and
1,000,000 operations walked across those expansions. Extraction stops
when either cap is hit. Legitimate shallow nesting is unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): share Form XObject budget across invisible-layer retry
extract_page_text_items created a fresh FormWalkBudget on every call, so
the invisible-text retry could consume a second full expansion budget for
the same page. Own the budget at the call site and pass it into both
passes.
Also charge form operations independently of the invocation cap so a form
that was already admitted can finish its stream.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(regions): serve invisible (Tr 3) OCR text layers instead of needs_ocr
Scanned pages (archive.org-style digitizations) carry their text as an
invisible render-mode-3 layer behind the page raster. extract_text_in_regions
extracted only visible items, so such pages yielded nothing but
'[Image: ...]' placeholders and every region fell back to OCR — while the
markdown path already includes the invisible layer for Mixed PDFs. The two
extractors disagreed about the same page.
Mirror the markdown path's gate, page-scoped: when the visible pass is
effectively textless (<40 non-placeholder alphanumerics), retry with
invisible text included and adopt the retry only when it contributes real,
non-garbage text. Pages with real visible text never retry, so double-layer
PDFs (visible text plus an invisible accessibility copy) are unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: strict zero-visible adoption gate, whole-layer garbage check, doc placement, discriminating tests
- Adoption now requires ZERO visible text on the page: the invisible pass
returns visible items too, so adopting alongside any visible text would
duplicate it. Strict gate instead of fuzzy dedupe; the dead
inv_alnum > visible_alnum condition goes with it.
- Garbage check judges the whole recovered layer, not the first 200 items.
- Constant/helper moved above the doc block so rustdoc stays attached to
extract_text_in_regions_mem.
- Tests: any-visible-text-blocks-adoption case (single short visible line)
with exactly-once assertions; mode-0 guard asserts occurrence count.
- Re-verified against real scanned-book pages: all recover (9.2-11.7K chars,
needs_ocr=false) — pure OCR-layer scans carry no visible text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: gate adoption on visible-item presence, not alphanumeric mass
Punctuation-only visible text (zero alphanumerics) could still adopt the
invisible pass; its OCR twin in the layer would duplicate the glyphs. The
gate is now item-presence: any non-image item with non-whitespace text
blocks adoption (whitespace-only artifacts still tolerated). Pinned by a
punctuation-only fixture test; real scanned-book pages re-verified — all
recover unchanged (they carry no visible items at all).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: punctuation-gate test pins preservation, comment states fixture scope
The fixture's visible punctuation is a separate block, not mirrored in the
invisible layer — so it pins the gate, not the duplication scenario. The
doc comment now says so, and a positive assertion checks the punctuation
survives exactly once.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: retry only when invisible text was actually skipped; negative gate tests
- extract_page_text_items now reports skipped_invisible (4th return): the
Tr-3 suppression sites set it, so the region extractor retries only when
a recoverable layer exists. Blank pages and image-only scans without an
OCR layer — the common scanned case — no longer pay a second
content-stream parse.
- Negative tests: below-floor watermark layer and symbol-garbage layer are
both rejected (region keeps only the raster placeholder). needs_ocr
semantics for placeholder-only regions deliberately unchanged — that
contract predates this PR and downstream pipelines handle it.
- Real scanned-book pages re-verified: recovery unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: flag the ' show-text suppression path; require nonempty strings
- P1: invisible layers shown via the ' operator never set skipped_invisible,
so those pages kept the placeholder and fell back to GPU OCR. The '
suppression path now flags too — pinned by a fixture whose layer is shown
entirely via ' (nothing through Tj/TJ).
- P3: all three flag sites (Tj, TJ, ') require a nonempty string operand —
numeric-only TJ kerning arrays and empty shows no longer trigger the
invisible reparse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(extractor): supply built-in metrics for non-embedded base-14 fonts
PDFs may legally omit /Widths for non-embedded standard fonts (Times,
Helvetica, Courier, Symbol, ZapfDingbats) — the spec requires the reader
to supply the metrics. We returned None, so every glyph advanced 0 and
each text item got width 0, silently breaking every gap-based heuristic
downstream: space synthesis, sub/superscript detection, table column
detection, heading merging.
- src/extractor/base14.rs: Adobe Core-14 AFM width tables keyed by
Unicode char, plus the standard Symbol/ZapfDingbats encoding vectors
(their glyphs sit at byte positions unrelated to Latin text, so widths
must resolve through the built-in encoding, not cp1252)
- Width resolution order: Differences -> built-in encoding -> the same
cp1252-style fallback the text decoder uses, so a code's advance always
matches the character we emit for it
- Type3 visual sizing: PK bitmap fonts (dvips) use FontMatrix
[1 0 0 -1 0 0] with nominal sizes like 0.12pt; scale by FontBBox height
x |matrix_y|. Applied in the page-stream and Form XObject paths.
Indirect numeric array elements are resolved before use.
Effect on Shannon's 'A Mathematical Theory of Communication' (1998
dvips/Distiller, the reported case): glued sentences 95 -> 5. Corpus
impact: 12 of 184 eval documents, e.g. Data-Processing-Agreement
recovers a paragraph that a phantom table had shredded into cells.
Layout heuristics tuned on the same document (indent-based paragraph
breaks, heading reclassification, table script filtering) are held back
for a separate PR — they change ~98 further documents and need to be
justified against the corpus, not against one PDF.
* review: narrow Type3 rescaling to self-inconsistent fonts; dedup + test all width tables
Addresses cubic review on #241, plus a follow-up from a local cubic run.
- Type3 visual scaling was applied to every Type3 font whose FontBBox
height x |matrix_y| deviated >5% from 1.0. FontBBox is the glyph box,
not the em box, so a conventional 1/1000-matrix font with a
descender..ascender bbox (~700 units) computed 0.7 and had every
reported size shrunk by 30% — corrupting the drop-cap, heading-tier,
sub/superscript and table heuristics this is meant to fix.
First attempt gated on the matrix being unit-scale, but a local cubic
run pointed out that wrongly excludes valid non-standard matrices (a
0.005 matrix with a full-em bbox legitimately needs a 5x scale). The
product is the right discriminator, not the matrix: a self-consistent
font lands near 1.0 because the matrix is the reciprocal of the
glyph-space em, so only a wildly inconsistent one (dvips/PK bitmap
fonts sit at ~159) is renormalized. Band widened to [0.25, 4.0].
Corpus effect: 12 -> 7 documents change. The 5 that drop out were
being wrongly rescaled — including Data-Processing-Agreement, whose
phantom-table fix turned out to come from this bug rather than from
the width fallback, so it is correctly given up.
- base14: all 14 width tables now covered by the sort-invariant test via
an ALL_TABLES registry, not a hand-picked subset.
- base14: identical tables share one static (all four Courier variants
are monospace 600; the oblique Helvetica variants match their upright
forms), removing 5 duplicate copies.
* test: refresh Shannon snapshot after merging main
CI checks out a merge of the PR head with main, and main advanced 8
commits since this branch was cut — including #201 (contextual digit
runs), #240 and #253 (markdown fixes). Those change extraction output,
so a snapshot generated on the unmerged branch could not match; the
Test job failed on the merge commit while passing on the branch itself.
The merged behaviour is better: the footnote marker '2' before
'Hartley, R. V. L.' is now recovered instead of dropped.
950 tests pass on the merged tree, clippy clean.
* fix(layout): unfuse independent column runs sharing a baseline
Two-column report pages with charts fused headings into the adjacent
column's body text: the columns' ~6pt gutter is below what histogram
valley detection can safely use, so the page grouped single-column and
same-baseline items from both columns joined into one line ('6.2.
Expectations for Re-Hiring Employees' + mid-sentence text, killing
MHS and NID on the whole survey-report doc family).
Three changes:
- Line grouping splits same-baseline runs separated by a wide void
(>3x font size, >=30pt) when the incoming run starts lowercase
(mid-sentence continuation from another column) and both sides are
multi-word prose. TOC page numbers, dot leaders, and table cells
(numbered/capitalized) stay joined.
- Column detection is blind to chart-region text (tight 2pt bounds —
wider padding ate rows adjacent to charts), via a chart-aware line
grouping variant wired from the markdown pipeline.
- validate_and_build_columns computes its vertical span from
histogram-eligible items only, so full-width captions no longer sink
the overlap ratio for partial-page column regions.
opendataloader-bench: overall 0.8532 -> 0.8554, MHS 0.761 -> 0.769;
doc 038 +0.434, no regressions. pdf-evals: 34 snapshots change,
semantic composite wash (0.5749 -> 0.5748), no per-doc mover >0.015.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(layout): review follow-ups — chart-aware band-split grouping, single chart scan per page
Band-split pages now route through the chart-aware grouping too, and
the band loop reuses the precomputed page_chart_map instead of
re-scanning the rect list per page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Link items carry an annotation rect, so y is a box edge — unlike text
items, where y is a baseline. Testing rect-bottom dropped partially
visible links whose bottom edge dipped past the tolerance. Follow-up
to a #160 review comment.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(extractor): clip page content to the visible page box
Single-page extracts and imposed spreads keep neighboring pages'
content in the stream, positioned outside the CropBox. Extracting it
appends invisible sections to the page, scrambles NID, and poisons
font statistics (heading tiers built from off-page text).
Clip items (by center), and — only when off-page text was actually
found — rects and lines (by overlap) to CropBox-else-MediaBox, walking
page-tree inheritance. Rotated pages are left unclipped: their item
coordinates are already transformed out of box space. Degenerate boxes
(<1 inch) are ignored.
opendataloader-bench: overall 0.8445 -> 0.8537, NID +0.008,
MHS +0.013; six docs up (best +0.426), none down.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(extractor): guard page-box clipping with coherence and straddle checks
Two real-document counterexamples: curved display text leaves short
glyph fragments with artifact coordinates outside the box (judge by
character mass, not item count), and some PDFs compute inflated
coordinates for visible body text (an off-page item continuing an
on-page baseline means our transform model is wrong there — skip).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(extractor): clip off-box link annotations when page text was clipped
Review follow-up: annotations from the neighboring page bypassed the
filter. Form fields are left as-is — they're document-scoped and rare
on imposed spreads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(underline): rescue snug-owned underlines from the table-ruling filters
Documents that underline many full-width lines (dense CJK business docs,
legal redlines, 10-K section links) produce span-similar rules at 3+
y-levels — exactly what the repeated-ruling filter treats as table
rulings, so every semantic underline on such pages was discarded. Three
changes fix detection without re-marking real tables:
1. Snug-owner rescue: a rule survives the repeated-ruling filter when
the union of touching text runs on its baseline row owns it (rule
contained within the union's span +0.75em, runs cover >=60% of the
rule, no column-sized gaps between runs). Table row separators fail
ownership: they overshoot their cells' text or match gapped items.
Same-row segmented rules (column-header separators) always stay
discarded, and a rule enclosed by a drawn cell-sized box (rect-grid
tables) is never rescued.
2. Vertical window widened 0.35em -> 0.72em below the baseline: CJK
layouts draw underlines under the full em box, measured at ~0.67em.
3. Prose-table guard in the positions-path suppressor: a detected
'table' whose cells hold flowing prose (>=30% of cells over 100
chars) is a detection artifact of boxed callouts + stacked rules,
not a real table — suppressing there erased every underline on the
page.
Also fixes cluster_x_positions fabricating phantom table columns from
style-split continuation runs (touching items, gap <2pt, now feed one
column start) — the fix that keeps rect-grid table shapes stable while
underlined links inside cells are correctly marked.
Snapshot updates are underline gains on regulation/form fixtures and one
empty spacer-column change in a subscripted header.
Corpus (508-doc public bench sweep): text output byte-identical on all
docs; underlined items +224/-0; strikeout now fires on redline docs.
Item-level GT coverage: is_underline 86->151/405, is_strikeout 0->10/44.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(underline): fraction-bar guard + subscript merge across underline marks
Found by a 202-doc real-world corpus diff (pdf-evals) that exercises the
full markdown pipeline, which the bench-corpus item sweep does not:
1. Math fraction bars and lattice grid lines are underline geometry —
short horizontal rules under digits. Guard: a narrow rule (<=60pt)
with bar-sized text hanging just below it (denominator) never marks.
The below-text width bound matters: tightly-leaded REAL underlines
have a full-width next line below, which must not trip the guard.
2. merge_subscript_items refused to merge when the parent was underlined
but the tiny digit was not (the drawn rule easily misses the digit's
own overlap window) — losing the merge broke subscript tokens inside
table cells (b+2 no longer became b₂). Strikeout boundaries still
block the merge in both directions; only parent-underlined/digit-bare
merges, absorbing with the parent's flags.
Corpus after refinement: underlined items +220/-2 (the 2 are fraction
bars the old code wrongly marked), GT rule-text coverage 149/405
underline + 10/44 strikeout, text output identical on all 508 bench
docs and word-count-identical on the 202 pdf-evals docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(underline): address review — grid-evidence veto, strikeout-safe fraction guard, bounded gaps
- Cell-box veto now requires GRID EVIDENCE (a vertically abutting
neighbor rect with x-overlap) instead of a height window: multiline
table cells taller than the old 90pt ceiling veto again, and isolated
filled callout panels (which legitimately contain underlines) no
longer veto at all.
- The fraction guard gates only UNDERLINE marking; rule_strikes_item
still evaluates, so short strikeouts near lower text survive.
- Fraction hug distance tightened to 0.3em so a short last-line at
normal leading is not mistaken for a denominator.
- Continuation-run suppression bounds the negative gap (-4pt): text
overhanging from an adjacent cell keeps its own column start.
Corpus after review fixes: underlined items +222/-2, GT coverage
150/405 underline + 10/44 strikeout, bench text output identical on
all 508 docs, pdf-evals word loss bounded at equation-reflow noise.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* chore: appease clippy (redundant closure)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(extractor): run-local space floor for tracked (letter-spaced) glyph runs
Display type set with tracking renders one glyph per show op; the merge
loop's fixed space thresholds (0.08-0.13 em) then read every letter gap
as a word boundary and emit "H O W" / "F U R T H E R" instead of
"HOW" / "FURTHER". The page-level Canva fixer can't help: it requires
>=50% of the page's items to be letter-spaced, and these docs track
only their display headings.
merge_text_items now pre-scans each run of consecutive single-glyph
items (same size band, same style, mergeable gaps — the loop's own
break conditions) and, when the run is tracked, derives the space floor
from the run's own gap distribution:
- runs with >=4 gaps qualify when the median gap clears the fixed
threshold; word gaps, if present, form a second mode — split at the
largest relative jump (>=1.4x), else the run is a single word
("I T I S I M P O R T A N T" -> "IT IS IMPORTANT")
- short runs (2-3 gaps: "H O W") additionally demand uniform gaps and
ALL-CAPS or CJK — a genuine spaced sequence of single letters
("x y z" variables) has the same gap count, and display tracking is
a caps convention; CJK never wants inter-glyph spaces
Corpus sweep (708 opendataloader + ParseBench text PDFs) vs main: 9
docs change — the tracked display titles ("HOW CAN YOU HELP?",
"LUNCHTIME MENU", a tracked email address), and CJK glyph-per-item
docs whose spurious inter-glyph spaces now collapse (GT for those docs
is unspaced CJK; should_join_items already treats no-space CJK as
correct on its path). No other doc moves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): convention gate on both tiers; Han/Kana floor always infinite (PR #133 review)
- Long lowercase spaced-single runs ("a b c d e") had the tracked gap
shape in the >=4-gap tier with no convention guard — word boundaries
lost. The caps/CJK/title-case gate now applies to BOTH tiers; a
title-case single word ("B u f f a l o") also qualifies.
- Han/Kana runs skipped straight to the bimodal split, so a nonuniform
gap distribution (justification, punctuation spacing) could
manufacture a word boundary. Han/Kana now always floors at infinity;
Hangul deliberately keeps word-boundary handling — Korean spaces
between words (is_spaceless_cjk excludes it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(extractor): preserve mixed-case glyph boundaries
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(extractor): descriptor/embedded-font style flags + geometric strikeout detection
Two style-recall gaps, both invisible to the existing name-based
heuristics:
1. Subset fonts with opaque BaseFont names ("Tc1", "AAAAAB+Amplitude")
defeat is_italic_font/is_bold_font. New descriptor_style_flags reads
the FontDescriptor (ItalicAngle beyond 4 degrees, Flags bit 7 Italic,
bit 19 ForceBold) and, when the descriptor claims upright, falls back
to the embedded font file: ttf-parser's OS/2 fsSelection + post
italicAngle for sfnt fonts, and the CFF Name INDEX PostScript name
for bare-CFF FontFile3 (descriptor rewritten to ItalicAngle 0 while
embedding "Amplitude-LightItalic" was observed in the wild).
ORed into is_bold/is_italic at item creation (content streams and
form XObjects).
2. No strikeout signal existed. New is_strikeout on TextItem, detected
in the same pass as underline: same rules pipeline (stroked lines /
thin filled rects, table-ruling suppression), different vertical
window — a rule crossing the glyphs at 12-55% of the em above the
baseline instead of sitting at it. Exposed through napi and python
bindings and pdf2md --items-json.
Verified on public ParseBench corpus docs: previously-missed italic
council titles and bold CJK itinerary headings now flagged (render-
checked); 24/508 docs gain flags, none lose any; 35 strikeout items
detected corpus-wide, disjoint from underline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): quote-op advance width, Ts text rise, doc-level font style cache (PR #125 review)
Address three valid findings from review:
- The ' (move-to-next-line-and-show-text) operator emitted zero-width
items and never advanced the text matrix, so geometric underline/
strikeout detection (which requires width > 0) could never mark its
text, and following show ops overlapped it. Reuse Tj's advance-width
computation and matrix advance.
- Ts (text rise) was dropped entirely: raised/lowered runs kept the
unshifted baseline, so rules drawn at the risen glyph position missed
the strike/underline windows. Track rise in the text state (saved and
restored with q/Q) and shift the rendering position through the text
matrix's y column; advances stay on the unshifted matrix per spec.
- descriptor_style_flags re-decompressed and re-parsed the same embedded
font program on every page whenever the descriptor left a style flag
unset (the common case). Add a document-scoped FontStyleCache keyed by
the FontFile2/FontFile3 object id, threaded through page and form
extraction alongside the existing CMapDecisionCache.
The fourth finding (Form XObject rules never reach geometric detection)
is real but pre-existing for underline and needs the form walker to grow
path/paint tracking plus a new return type; deferred as a follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): ActualText items render at their glyphs' text rise (PR #125 review)
The EMC-built ActualText item used the captured text matrix without the
rise adjustment the ordinary Tj/TJ/' emission sites apply, so a tagged
run shown with Ts landed on the unshifted baseline — off the strikeout/
underline windows and inconsistent with untagged runs. The rise is
captured together with the first-glyph matrix (and at BDC for the
entry-position fallback): the item must render at the rise of its
GLYPHS, not whatever rise is set by EMC time.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): capture ActualText glyph position after the quote op's line move (PR #125 review)
The `'` handler skipped the entire suppressed-extraction block, so a
tagged span whose show op is `'` never captured its glyph matrix/rise —
the EMC item fell back to the BDC-entry matrix, which sits on the
PREVIOUS line (the `'` line move happens after BDC) with no rise. The
capture now happens right after the line move, matching the Tj/TJ
paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
* fix(review): style-boundary gate on subscript merge + strikeout suppression coverage (PR #125 review)
merge_subscript_items absorbed a script digit into its parent
regardless of underline/strikeout flags — dropping the digit's own mark
or widening the parent's over it. The merged item carries one flag, so
differing marks now break the merge, mirroring merge_text_items'
style-boundary rule (pre-existing for underline as well).
Also extends the table-suppression test to assert is_strikeout is
cleared alongside is_underline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(markdown): underline emission, Unicode scripts, style-preserving merges (ENG-5015 2b)
Three formatting losses in the direct-extraction markdown path:
1. text_with_formatting gains <u> run emission (detect_underline option,
default on) using the geometric is_underline flag from 1.9.9.
Underline runs stay free of nested bold/italic markers — consumers
match tag content literally. Heading lines keep plain text for
bold/italic but preserve <u>: the tag carries meaning `#` doesn't.
2. merge_subscript_items now maps absorbed digit scripts to Unicode
sub/superscript forms with direction from the baseline offset
("H"+"2" -> "H₂", "word"+raised "2" -> "word²", "m"+"3" -> "m³").
NFKC/NFKD folds these back to plain digits so text matching
downstream is unaffected; renderers keep the script semantics.
3. merge_text_items no longer merges across bold/italic boundaries —
absorbing a styled run into a plain neighbor erased the styling
before markdown emission ever saw it. On eval docs this recovers
20-82 italic runs per document that previously emitted as plain.
Snapshots regenerated (diffs are the features: CCl₂F₂, m³, underlined
legal section headings, finer bold runs). pdf-evals regression suite:
202/202 real PDFs pass. napi 1.9.9 -> 1.9.10.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): break merges at underline boundaries too (review)
OR-merging underline stretched the eventual <u> span over neighboring
plain fragments. Merge runs now break on any style-flag change, the
redundant accumulator is gone, and format_list_item learned to move
bullet markers outside <u> wrappers so fully-underlined bullet lines
still render as markdown lists. td9264 snapshot regenerated — spans are
tighter (trailing periods correctly outside the tag).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(markdown): strip stray spaces before sentence punctuation (review)
Style-boundary item splits can strand a trailing period in its own
fragment, and multiple assembly paths join fragments with spaces,
yielding "word ." artifacts. Rather than chasing every join site, a
postprocess pass removes a space before `.`/`,`/`;` when the mark ends
its token (whitespace, cell boundary `|`, or end of text follows).
Dot leaders/ellipses and mid-token periods are untouched.
Fixes the td9264 "companies ." artifacts and two pre-existing
"armoring ," artifacts in the 2013-app2 snapshot. pdf-evals: zero
markdown diffs across all 203 corpus PDFs vs committed baselines.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tables): trim spaces inside parenthetical cell fragments
* fix(tables): reject sparse prose row-stripe tables
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(extractor): geometric underline detection on TextItem (ENG-5015)
PDFs carry no underline font flag — underlines are stroked horizontal
lines or thin filled rects drawn under the baseline. Correlate those
graphics (already parsed from the content stream) with text items in a
post-pass: a rule within ~0.35em below the baseline covering >=60% of
an item's width marks is_underline.
Exposed through the napi and python bindings. Verified on real docs:
4/4 underlined sentences flagged on a Japanese report, links/headings
flagged on 8 of 10 underline-bearing eval docs, zero flags on docs
without underlines. Known FP source (table cell borders) documented —
downstream applies inline styling only to plain-text regions.
napi 1.9.8 -> 1.9.9.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): underline rules only from painted rects, normalized extents (review)
Two review fixes: (1) normalize rect extents before the thickness/width
checks — `re` operands pass through the CTM so width/height can be
negative, which missed negative-width rules and let negative-height
bands pass as thin; (2) only feed painted rects to underline detection —
`re` rects now wait in a pending list until a paint operator (S/s, f/F/
f*, B/B*/b/b*) confirms them, and `re W n` clip-only paths are discarded
at `n`, so invisible clip boundaries no longer underline nearby text.
Marking moved into content_stream where paint state lives (pre-rotation,
consistent device space).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(extractor): harden underline detection
* feat(cli): export positioned text item json
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* emit ItemType::Image bboxes for Image XObjects (was: silently dropped)
Background. ItemType::Image, MarkdownOptions::include_images, and the
markdown emitter's image-collection path have all been in the tree
for a while, but no producer ever populated them — content_stream.rs
explicitly `// Skip images — text extraction only` at the Do
operator, and the nested Form-XObject walker in xobjects.rs only
matched XObjectType::Form, silently dropping Image entries. The
declared types were dead code.
This PR lights them up. At every Do that resolves to an Image
XObject (both top-level and nested inside Form XObjects), we now
compute the page-space bbox from the current CTM via a new
`image_bbox_from_ctm` helper — handling both axis-aligned and
rotated/sheared placements via 4-corner AABB — and emit a TextItem
with `item_type: ItemType::Image` and the legacy `[Image: <name>]`
text payload that the markdown emitter already knows how to render.
Callers can now find raster figures via `extract_text_with_positions`
(and the `_mem` variant, newly re-exported at the crate root) without
needing to re-parse the PDF or run a vision/layout model. The intended
consumer is layout-aware text pipelines that want to crop figures and
caption them out-of-band.
Two backstops to avoid silent breakage for existing callers:
1. `MarkdownOptions::include_images` default flipped `true → false`.
If it stayed at `true`, every existing user of
`extract_pages_markdown` would suddenly see ``
placeholders inserted throughout their output the moment they
upgraded. Image data is still available structurally via
`extract_text_with_positions`; rendering it into markdown is now
an opt-in. New regression test asserts `extract_pages_markdown`
output is unchanged for the image-bearing fixture.
2. Image items now also skip the layout heuristics
(`detect_columns`, `detect_tables_from_rects`) via a new
`is_text_layout_item` predicate. Without this filter, an image's
left edge would land in the column-projection profile and skew
table column detection — surfaced by
`vector_grid_tests::upstage_key_functions_four_cols` going from 4
detected columns to 5 in CI before the filter was added.
Re-exporting `extract_text_with_positions_mem` at the crate root —
strictly additive; mirrors how `extract_pages_markdown_mem` is already
available there.
Tests:
- test_extract_text_with_positions_emits_image_bboxes — minimal PDF
with one 200×100 image at (50, 600); asserts one Image item with
correct bbox + page + text.
- test_image_xobject_bbox_handles_rotated_ctm — 90° rotated image
via shear-component CTM; asserts AABB is correct (handles non-
axis-aligned placements via 4-corner clamp).
- test_image_emission_does_not_change_default_markdown — asserts no
`Image:` token leaks into default markdown output, regression
guard for the include_images flip.
- test_markdown_options_default_has_include_images_false — explicit
sentinel so anyone flipping it back catches it in CI.
* Bump version from 1.8.15 to 1.9.0
Align region filtering with rotated-page coordinate rewrites, switch region text assembly to the shared line-grouping pipeline, and retain edge-overlap text to avoid false empty regions that incorrectly trigger OCR fallback. Also make Python region inputs fail fast with clear ValueError messages for malformed boxes.
Made-with: Cursor
* fix sort panics on NaN values from bogus PDF font metrics
Replace all `partial_cmp(...).unwrap_or(Ordering::Equal)` and bare
`partial_cmp(...).unwrap()` with `total_cmp()` across the codebase.
`partial_cmp` returns `None` for NaN, and mapping that to `Equal`
violates total ordering: `a == NaN` and `NaN == b` but `a != b`.
Rust 1.81+ detects this and panics in sort_by. `total_cmp` handles
NaN deterministically (sorts to end) and guarantees total ordering.
The critical crash was in `extract_text_in_regions` (lib.rs:478)
where PDFs with bogus font ascent/descent values produced NaN in
text item coordinates, causing process abort via NAPI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix missed partial_cmp in layout.rs and restore napi exports
- Convert two remaining b.y.partial_cmp(&a.y) calls to total_cmp
in group_single_column and column layout sorting
- Restore missing napi exports: detectPdf, extractText,
extractTextWithPositions, processPdf
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add `extract_text_in_regions_mem` that takes layout-detected bounding
boxes (top-left origin, PDF points) and returns text within each region
in reading order. Designed for pipelines where a layout model detects
regions and text-based pages can skip GPU OCR by extracting text from
the PDF structure directly.
Also add `classify_pdf_mem` for lightweight PDF type classification
returning 0-indexed pages_needing_ocr.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(layout): relative valley column detection for justified text
Add fallback column detection using relative valley analysis for PDFs with
justified text where item widths extend past gutter boundaries. The absolute
valley detector fails on these layouts because gutter bins are at ~40% of
peak (well above the 15% noise threshold).
The relative valley detector smooths the histogram with a 5-bin moving
average, finds local minima where contrast < 0.60 of surrounding peaks,
and validates with peak balance >= 0.40. Limited to single best valley
(max 2 columns) and requires >= 100 items per page.
Tested on IRS Publication 17 (2002), a 289-page 2-column justified text
document: column detection went from ~40 pages to 165 pages.
190 passed, 0 regressions across 191 eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(layout): tighten relative valley thresholds to reduce false positives
Reduce PEAK_WINDOW from 40 to 25 bins (50pt) so valleys are only validated
against nearby peaks, not distant ones. Add MIN_PEAK_HEIGHT of 20 (smoothed)
to reject sparse pages where histogram peaks are too low to indicate dense
two-column text.
Previous thresholds caused 13 regressions across the eval suite by splitting
tables, TOCs, checklists, and forms. Now: 188 passed, 0 regressions (2 minor
metadata-only diffs on IRS P17 and 9978293).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(layout): skip relative valley detection on pages with tables
Table column gaps in the histogram look identical to text column gutters
but the table pipeline already handles reading order for those pages.
Pass page_has_table flag through detect_columns to suppress the relative
valley fallback on pages where tables were detected.
This eliminates all remaining regressions from relative valley detection:
190 passed, 0 regressions across 191 eval PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(layout): prose density validation for relative valley detection
Add columns_have_prose() to validate relative valley column splits.
Checks that both sides of a proposed split contain paragraph-like
content (fill ratio >= 40%, avg items/line <= 3.5) before committing
to a column split. Combined with the table-page guard, this prevents
false column splits on financial statements, forms, and tabular
layouts where long labels or dot leaders fill the column width.
Also tightens find_relative_valleys() thresholds (PEAK_WINDOW 40->25,
MIN_PEAK_HEIGHT 5->20) to reduce false positive valley candidates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(text): handle Tc/Tw character and word spacing in text width computation
PDFs using Tc (character spacing) and Tw (word spacing) operators for text
justification had words incorrectly split across TextItems. The computed
advance width didn't account for these spacing parameters, causing spurious
spaces mid-word (e.g. "deve lopers" instead of "developers").
- Add Tc/Tw operator handling and graphics state save/restore
- Incorporate char_spacing and word_spacing into compute_string_width_ts
- Add adaptive merge threshold: tighter for lowercase→lowercase junctions,
wider before joining punctuation
- Add unit tests for Tc/Tw width computation and merge behavior
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(fonts): add large Tc width computation test
Verifies that large character spacing values are applied in full
without any artificial cap, matching PDF spec behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(text): guard against Tc/Tw-inflated widths in merge and join paths
Two targeted fixes to prevent character-spacing (Tc) and word-spacing
(Tw) inflation from causing data quality regressions:
1. should_join_items: reject large negative gaps (< -font_size) that
arise when Tc/Tw inflate item widths past adjacent items. Fixes
FY_2015 merged numbers (e.g. "239.696.0" → "239.69 6.0").
2. merge_text_items: cap effective width for gap computation when Tw
inflates space-containing items beyond 0.85× font_size per char.
Prevents column-level gaps from collapsing into merge range,
recovering table detection for Baldwin-Edwards and similar PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(fonts): CID-as-Unicode passthrough and subscript/superscript merging
Add smart CID-as-Unicode passthrough for Identity-H fonts without
ToUnicode maps. Uses /W array median CID heuristic to distinguish
Unicode-CID PDFs (Chromium-generated) from GID-based subsets.
Add merge_subscript_items() pass that merges small-font items (<75%
of dominant font size, ≤4 chars, tightly adjacent) into parent items.
Fixes chemical formulas (NH3, H2O, KClO3), footnote references, and
subscript notation (vf, Hfg, m3/kg) that were previously orphaned
as separate text items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(subscripts): restrict merge to purely numeric text only
Tighten subscript merging to only merge items containing ASCII digits
(0-9). This avoids false positives with ordinal indicators (º), letter
subscripts (sol, vf), and small bullet characters (▶) that caused
table restructuring regressions.
Numeric-only keeps the primary wins: chemical formulas (NH3, H2O),
footnote references, and unit notation (m2, m3).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(subscripts): restrict merge to parent text ending with a letter
Only merge numeric subscripts when the parent item's text ends with
an alphabetic character. Prevents false merges like "33" + "1" in
fractions (33 1/3%), table credit numbers after spaces, and footnote
refs after punctuation (land.1 → land. 1). Chemical formulas (NH3,
H2O, KClO3) still merge correctly since parent ends with a letter.
Reduces pdf-eval regressions from 13 to 2 (both are correct reversions
of over-aggressive footnote merging from the prior commit).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When rect-based, line-based, and heuristic detection all fail to find
tables on a page, try building a table directly from the layout
engine's column boundaries. Handles borderless tabular layouts like
exam/reference grids where columns are defined purely by text
alignment.
Includes header-row column refinement: when a detected column
contains multiple header items, it gets split at the gap between
them to recover the correct number of columns.
Guards against false positives: requires ≥4 columns, ≤40 rows,
>50% multi-column rows, short cells (avg ≤40 chars), no prose
content, no dominant single column, and no structural elements
(≥6 rects or ≥4 lines) on the page.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fonts using raw glyph ID names (gidNNNNN) in their Differences
encoding cannot be decoded to Unicode without the original font's
cmap table. Detect this pattern during font parsing and add
affected pages to pages_needing_ocr so downstream consumers
know to use OCR instead.
Fixes text extraction on PDFs like Tezukuri_Food-Menu.pdf where
the main body font (AcuminVariableConcept) uses gid-encoded
glyphs — even PyMuPDF and ODL fail on these.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scanned PDFs with OCR text layers use rendering mode 3 (invisible text)
positioned behind page images. Previously we skipped all Tr=3 text.
Now for Mixed/template PDFs, if normal extraction produces garbage or
empty output, we retry with invisible text included. This unlocks text
from OCR-generated PDFs without requiring external OCR.
Also adds Windows Unicode BMP (3,1) subtable support to the TrueType
cmap fallback, and allows TrueType fonts with explicit encoding to
extract their embedded cmap (OCR fonts often lie about encoding).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: tagged PDF structure tree support for semantic markdown generation
Parse /StructTreeRoot from tagged PDFs and use semantic roles (H1-H6, P,
LI, BlockQuote, Code, Caption) to improve markdown output. Structure tree
headings add to font-size heuristics without suppressing them. Coverage
threshold (≥50%) ensures only properly tagged PDFs activate this path.
Phase 1: Parse structure tree with role maps, MCID collection, flattening
Phase 2: Capture MCIDs from BMC/BDC operators, tag TextItems
Phase 3: Structure-aware markdown generation in convert loop
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: accumulate consecutive code lines into single fenced block
Per-line code fencing produced broken markdown for multi-line code
blocks (separate ``` open/close per line). Unify struct-tree Code
role and font-based monospace detection into a single is_code_line
check with in_code_block state for proper accumulation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add tagged PDF fixture with Firecrawl docs content
Synthetic 7-page PDF with rich structure tree exercising H1, H2, H3,
P, Code, LI, Caption, TH, TD roles. Generated via fpdf2 script.
Integration test verifies struct tree parsing and code fence output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: remove python PDF generator script from repo
Keep the generated fixture PDF but don't track the generator script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle malformed bare-name struct types in tagged PDFs
Some PDF generators (e.g. fpdf2) write /S Code instead of /S /Code
in structure elements. lopdf silently drops these objects since bare
tokens are invalid PDF syntax. Add a pre-processor that scans for
known bare struct type names and prepends / before loading.
Unifies path and memory loading through the same fix pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: update lopdf dependency to main branch
The firecrawl/zlib-checksum-encrypted branch was merged and deleted.
Point to main which includes all previously merged fixes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: switch lopdf to upstream repo pinned at 845cd3d
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Narrow annotation columns beside wide body columns were being
Y-interleaved with body text, producing garbled reading order.
Add sidebar detection with width ratio, line balance, and sparse
density guards to correctly trigger sequential column reading.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Canva-generated PDFs render text character-by-character with CSS-style
letter-spacing (~0.5-0.9× font_size). The hardcoded 0.10 threshold
caused every character to get a space inserted ("K a r i b i b").
Detect Canva pages via fix_letterspaced_items (≥50% items match "a b c"
pattern), compute an IQR-based threshold (median × 1.55) on the gap
distribution BEFORE space removal, then propagate per-page thresholds
through PageThresholds → group_into_lines_with_thresholds → TextLine
→ should_join_items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Many IRS forms and government PDFs draw table gridlines using path
operators (m/l/S) instead of rectangle (re) operators. This adds
line-based table detection to capture these tables.
- Add PdfLine type for line segments from path operators
- Capture m/l/h/S/s/B/b/f/n path operators in content_stream.rs
- Thread Vec<PdfLine> through extraction pipeline
- New detect_lines.rs: classify lines, snap to grid, validate and
assign items with extensive false-positive filters
- Integrate in markdown pipeline: rects first, then lines as fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove `process_mode` from `MarkdownOptions` (it controlled the
pipeline, not markdown formatting)
- Remove dead `text` field from `PdfProcessResult`
- Add `PdfOptions` builder consolidating mode, detection, markdown,
and page filter configuration
- Add convenience functions: `detect_pdf()`, `detect_pdf_mem()`,
`process_pdf_with_options()`, `process_pdf_mem_with_options()`
- Eliminate double document parsing: load once, share between
detection and extraction via internal `pub(crate)` helpers
- Deprecate old `process_pdf_with_config*` functions (kept as shims)
- Update binaries to use new API
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Map lopdf's Unimplemented("encrypted...") error to PdfError::Encrypted
instead of falling through to PdfError::Parse. Retry all Document::load
calls with an empty password when encryption is detected, so
owner-password-only PDFs can be opened. If the retry also fails, the
user now sees a clear "PDF is encrypted" message.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces a ProcessMode enum that controls how far the PDF pipeline
runs, enabling fast document triage without paying extraction or
markdown conversion costs. Exposed via --detect-only and --analyze
flags in both pdf2md and detect-pdf CLIs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add LayoutComplexity struct to PdfProcessResult so callers can detect
when a PDF has complex layout (tables or multi-column text) and decide
whether to use the extracted markdown or fall back to OCR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates double-parsing of PDFs by using the lopdf document model exclusively
for ToUnicode CMap extraction. The old raw byte scanner parsed PDFs separately
from lopdf and only handled FlateDecode, while lopdf handles FlateDecode + LZW +
ASCII85. The new approach walks page fonts and Form XObject fonts via the document
API, yielding ~7x speedup on text-heavy PDFs and ~1.2x overall.
- Add FontCMaps::from_doc() with recursive Form XObject font walking
- Remove ~270 lines of raw byte scanning code (from_pdf_bytes, extract_stream_from_raw_pdf, etc.)
- Remove flate2 dependency (lopdf handles decompression internally)
- Remove dead CMap lookup fallback branches (by_name, get_with_obj, base_font_name)
- Remove unused font_base_names parameter from extract_text_from_operand
- Skip U+FFFD replacement characters in CMap decode (PDF notdef glyph markers)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
