* chore(ci): bump GitHub Actions to current majors
Node 20 action runtimes are deprecated on GitHub runners; bump every
first-party action to its latest major across all workflows:
- actions/checkout v4/v6 -> v7
- actions/cache v4 -> v6
- actions/upload-artifact v4 -> v7, download-artifact v4 -> v8
- actions/setup-node v6 -> v7, setup-python v5 -> v7
- actions/upload-pages-artifact v3 -> v5, deploy-pages v4 -> v5
Third-party pins (dtolnay/rust-toolchain, Swatinem/rust-cache,
setup-zig, setup-bun, taiki-e/install-action, maturin-action) are
already on their latest majors.
* chore(ci): pin all actions to full commit SHAs
Mutable @vN tags can be retagged; in the publish workflows that code
runs with OIDC credentials before npm/PyPI/crates.io publishes. Pin
every action (first- and third-party) to its release commit SHA with
the version in a trailing comment.
dtolnay/rust-toolchain infers the toolchain from its ref name, so the
SHA-pinned invocations pass an explicit toolchain: stable input.
Adds x86_64-unknown-linux-musl, aarch64-unknown-linux-gnu, and
aarch64-unknown-linux-musl to the napi build targets so
@firecrawl/pdf-inspector works on Alpine and ARM64 Linux deployments.
- gnu arm64 cross-compiles with --use-napi-cross (old-glibc sysroot),
musl targets with -x (zig + cargo-zigbuild), per the napi-rs template
- new platform packages carry npm libc metadata (glibc/musl)
- smoke-test job runs napi/test.mjs on all six targets before publish
(Alpine containers for musl, ubuntu-24.04-arm runners for ARM64)
- bump to 1.12.0 to trigger publishing of all platform packages
Closes#216
* fix(extractor): don't flag gid Differences names covered by ToUnicode
Pages were marked as having unresolvable gid-encoded fonts whenever any
font's /Differences array used gidNNNN glyph names, and when every page
carried such a font the whole document's markdown was suppressed.
LibreOffice exports do exactly this: subset fonts get /gidNNNN names in
Differences alongside a complete ToUnicode CMap that decodes them, so
ordinary text documents lost their entire markdown output even though
extraction decoded every glyph.
Track the character codes behind the gid names and only flag the font
when its ToUnicode CMap addresses none of them. Partially mapped codes
stay unflagged: an emoji ZWJ sequence maps whole on its first code, and
the remaining component-glyph codes are subset leftovers, not damage.
Fonts without ToUnicode, or whose CMap ignores the gid codes, are
flagged as before, and the downstream garbage/encoding checks still
catch partial breakage.
* fix(extractor): require a usable ToUnicode mapping to clear the gid flag
A mapping to U+FFFD (or an empty string) is rejected by extraction as
an invalid CMap result, so it must not count as decodable when deciding
whether gid-named Differences codes are resolvable.
Single-word bold section headings ('Replace', 'Trash', 'Instructions')
required a paragraph break before AND after, but headings hug their
section's first paragraph — the break-after almost never exists. A
standalone all-bold single word (>=4 chars, paragraph break before or
page top) now classifies; mixed bold lead-ins ('Note: ...') stay
excluded via all_bold.
opendataloader-bench: 0.8567 -> 0.8575, MHS 0.773 -> 0.776; docs 145
+0.118, 069 +0.112 (net of one cover-page layout shuffle at -0.066
where the new output is semantically closer to GT). pdf-evals: 66
snapshots, composite 0.5864 -> 0.5883, sole >0.02 mover positive.
p1244/thermo fixture snapshots regenerated ('Instructions' un-fuses
from its body paragraph — the intended behavior).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(headings): digit-only lines do not define heading tiers
A large bold page number (14pt folio over 11pt body) claimed tier 0:
every real heading demoted one level document-wide, and the bold-size
fallback (which requires an empty tier list) was blocked for documents
whose headings match body size.
Bench-neutral (MHS scores relative hierarchy); pdf-evals: 18 docs get
their heading levels back (#### -> ###), semantic composite +0.0006,
no percentile down.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(headings): exclude digit-only lines from the bold fallback tier pass too
The exclusion in the main pass wasn't enough: with the page-number
tier gone, the bold fallback re-collected the same bold folio. Also
regenerates the thermo-freon12 snapshot (cosmetic churn on the
scrambled legend fixture).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
IntechOpen-family academic PDFs embed Computer Modern math symbol
subsets whose glyphs are misnamed after Latin lookalikes (equal →
/onequarter, plus → /thorn, parens → /eth //Thorn) — and the generated
ToUnicode faithfully propagates the wrong names, so formulas decode as
'S ¼ kB þ 1' instead of 'S = kB + 1'.
Remap the observed misnames, gated strictly on the TeXCMMathsSymbols
base font (subset prefix stripped) so genuine fractions and thorns in
text fonts are untouched. Known limitation: a sibling subset misnames
the slash as /onequarter too, so an occasional '/' renders as '=' —
still strictly better than the previous mojibake.
Affects 4 bench PDFs (028/031 +0.001-0.004 NID) and zero pdf-evals
snapshots.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(headings): rescue wrapped bold headings on interleaved column pages
Report pages whose columns can't be detected (6pt gutters) interleave
both columns' lines, which breaks every whitespace signal the bold
heading heuristic relies on: para_threshold inflates to ~3x line
height and a wrapped heading's own internal line gap defeats isolation.
'9.5. Adapting to the New Normal: Changing / Business Models' merged
into the following paragraph.
Four changes:
- merge_wrapped_bold_heading_groups: 2-3 consecutive all-bold
body-size lines merge into one line when the group is isolated
(column-locally, judged by x-overlapping lines only) or starts with
a section number.
- Section-numbered all-bold lines ('9.5. ...') classify as headings
without the standalone/isolation score gate.
- Line unfusing extends to uppercase-start continuations, gated on a
bold-style mismatch between the runs (a bold heading beside regular
body text) — same-style label rows stay joined.
- The unfuse line-side wordiness requirement drops to 2 words so a
wrapped heading's short last line ('Business Models') still splits
from the neighboring column.
opendataloader-bench: overall 0.8554 -> 0.8576, MHS 0.769 -> 0.777;
docs 037 +0.161, 111 +0.157, 039 +0.091, 198 +0.028, none down.
pdf-evals: 63 snapshots, composite 0.5952 -> 0.5964, sole >0.02 mover
positive. thermo-freon12 snapshot regenerated (cosmetic churn on an
already-scrambled 3-column legend).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(headings): review follow-ups — multi-component section numbers, wholly-bold line gate
Single '1. ' prefixes are ordered list items and no longer bypass
isolation; the uppercase unfuse requires the whole line bold (a
heading), not merely its last run, so mixed bold-label/value rows
stay joined.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(layout): unfuse independent column runs sharing a baseline
Two-column report pages with charts fused headings into the adjacent
column's body text: the columns' ~6pt gutter is below what histogram
valley detection can safely use, so the page grouped single-column and
same-baseline items from both columns joined into one line ('6.2.
Expectations for Re-Hiring Employees' + mid-sentence text, killing
MHS and NID on the whole survey-report doc family).
Three changes:
- Line grouping splits same-baseline runs separated by a wide void
(>3x font size, >=30pt) when the incoming run starts lowercase
(mid-sentence continuation from another column) and both sides are
multi-word prose. TOC page numbers, dot leaders, and table cells
(numbered/capitalized) stay joined.
- Column detection is blind to chart-region text (tight 2pt bounds —
wider padding ate rows adjacent to charts), via a chart-aware line
grouping variant wired from the markdown pipeline.
- validate_and_build_columns computes its vertical span from
histogram-eligible items only, so full-width captions no longer sink
the overlap ratio for partial-page column regions.
opendataloader-bench: overall 0.8532 -> 0.8554, MHS 0.761 -> 0.769;
doc 038 +0.434, no regressions. pdf-evals: 34 snapshots change,
semantic composite wash (0.5749 -> 0.5748), no per-doc mover >0.015.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(layout): review follow-ups — chart-aware band-split grouping, single chart scan per page
Band-split pages now route through the chart-aware grouping too, and
the band loop reuses the precomputed page_chart_map instead of
re-scanning the rect list per page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(headings): bold-size fallback tiers when nothing clears the ratio gate
Books often set section headings barely above body size (11pt bold
over 10pt text). Nothing cleared the 1.2x heading-tier ratio gate, so
tiers stayed empty and every bold heading defaulted to H2 — H1 was
unreachable for the whole document.
When no size clears the gate, build tiers from bold line sizes >=1.05x
body, and let tier matches through detect_header_level down to that
ratio. Documents with real (>=1.2x) tiers are untouched.
Bench-neutral by construction (the MHS metric scores relative
hierarchy, not absolute levels); pdf-evals semantic composite +0.004
on the 11 affected docs with all percentiles up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(headings): require boldness for sub-gate tier matches
Review follow-up: fallback tiers come from bold lines, so honoring
them for non-bold text at the same size would promote captions.
detect_header_level now takes is_bold and only matches tiers below
the 1.2x gate for bold lines; >=1.2x matches stay bold-agnostic.
Also restores the >=1.2x tier-match loop the refactor dropped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(headings): judge line boldness by character mass
Review follow-up: a heading with an unbold section-number prefix
('4. ' + bold title) failed the first-item boldness test. Judge the
line by bold character mass instead, at all three call sites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Link items carry an annotation rect, so y is a box edge — unlike text
items, where y is a baseline. Testing rect-bottom dropped partially
visible links whose bottom edge dipped past the tolerance. Follow-up
to a #160 review comment.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(extractor): clip page content to the visible page box
Single-page extracts and imposed spreads keep neighboring pages'
content in the stream, positioned outside the CropBox. Extracting it
appends invisible sections to the page, scrambles NID, and poisons
font statistics (heading tiers built from off-page text).
Clip items (by center), and — only when off-page text was actually
found — rects and lines (by overlap) to CropBox-else-MediaBox, walking
page-tree inheritance. Rotated pages are left unclipped: their item
coordinates are already transformed out of box space. Degenerate boxes
(<1 inch) are ignored.
opendataloader-bench: overall 0.8445 -> 0.8537, NID +0.008,
MHS +0.013; six docs up (best +0.426), none down.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(extractor): guard page-box clipping with coherence and straddle checks
Two real-document counterexamples: curved display text leaves short
glyph fragments with artifact coordinates outside the box (judge by
character mass, not item count), and some PDFs compute inflated
coordinates for visible body text (an off-page item continuing an
on-page baseline means our transform model is wrong there — skip).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(extractor): clip off-box link annotations when page text was clipped
Review follow-up: annotations from the neighboring page bypassed the
filter. Form fields are left as-is — they're document-scoped and rare
on imposed spreads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Bar charts drawn as filled rects read as cell rects or aligned text:
the cell-rect fallback gridded their axis labels into phantom tables,
and when rect paths rejected them, hint regions and the gap-histogram
heuristic re-gridded the same text. On survey-report pages this
scrambled reading order and swallowed section headings.
Detection (is_chart_bar_cluster): the dominant equal-breadth rect
family arranged in >=2 spaced positions (bars; cell rects touch),
data-driven extent variation (>=1.3x), no same-offset/same-extent
partners across positions (grid rows pair up, chart segments don't),
and only numeric labels inside. Mirrored predicate covers horizontal
bar charts.
Chart clusters are skipped in detect_tables_from_rects (no table, no
hint), and a new detect_chart_regions pass lets the markdown pipeline
pre-claim chart items so heuristic/line/column detection and the
merged-band retry all skip them — the text flows out as plain lines.
opendataloader-bench: overall 0.8389 -> 0.8446, TEDS 0.699 -> 0.708,
MHS 0.742 -> 0.750, NID 0.889 -> 0.894; 7 docs up (best +0.459), none
down. pdf-evals changed-set composite +0.013, TEDS +0.036.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Abimael Martell