Concise Features list and the opendataloader-bench comparison table on
each registry readme, adapted per ecosystem. Bump all three versions
(crate 0.1.6, python 0.2.5, npm 1.11.1) to republish the pages.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The 0.2.3 sdist was 10.09 MiB (packaged tests/fixtures). maturin derives
the sdist file list from Cargo's include allowlist, so it's now 1.35 MiB
— but the allowlist dropped pdf_inspector.pyi, which would strip type
hints from wheels built from the sdist. Add it back and bump to 0.2.4.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
cargo publish of 0.1.5 failed with 413: the crate packaged everything
(260 files, 10.1MiB compressed) and tests/fixtures alone is 10.2MB.
Add an explicit include list (src, external/bcmaps which tounicode.rs
loads at runtime, readme, license) — 1.3MiB compressed.
Also add a workflow_dispatch fallback to publish-crate.yml so a failed
publish can be retried without a version bump (0.1.5 is already on
main, so a re-push won't register as a version change).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
crates.io showed the repo README, which leads with Python/Node quick
starts and repo-relative links. Point the crate readme at
docs/rust-api.md, refreshed with an intro, crates.io install, and CLI
install instructions. Bump to 0.1.5 to republish.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* ci: add PyPI trusted publishing, abi3 wheels, bump to 0.2.1
Adds publish-pypi.yml mirroring the npm/crates.io pattern: triggers on
Cargo.toml version change, builds wheels for 5 platforms via maturin,
publishes with OIDC trusted publishing (no tokens). workflow_dispatch
serves as a manual fallback for the first run after the PyPI project
transfer.
Enables pyo3 abi3-py38 so one wheel per platform covers CPython >=3.8
(previous manual uploads were cp312-only). Bumps version to 0.2.1 since
PyPI already has 0.2.0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci(pypi): guard dispatch to main, support partial-release repair
Review feedback: trusted publishing doesn't match on branch, so
workflow_dispatch needed an explicit main-ref guard. Manual dispatch now
always rebuilds and publishes with skip-existing so a release that
failed after uploading only some wheels can be completed by re-running.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci(pypi): version PyPI package from pyproject.toml, not Cargo.toml
Decouple the Python package version from the crate version, matching
how npm publishing keys off napi/package.json: bump [project] version
in pyproject.toml manually and CI publishes on merge. Reverts the
Cargo.toml bump so this PR no longer triggers a crates.io release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: remove accidentally committed uv.lock
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ci): tolerate missing version key in parent pyproject.toml
The first merge of this workflow has a parent commit where pyproject.toml
still used dynamic = ["version"], so the old-version read would KeyError
and the auto-publish would never fire. Treat a missing key as a change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(lib): detect substitution-cipher garbled text from broken ToUnicode CMaps
ParseBench text_simple__att10k.pdf (issue #118) ships Type0/Identity-H
fonts whose ToUnicode CMaps are authored garbled: every bfrange maps with
a wrong constant delta, so text extracts as pure-ASCII ciphertext
("Certificate" -> "8VceZWZTReV"). The embedded subset font has no cmap
table and no glyph names, so no decode source can recover the real text
(poppler and mupdf emit the same ciphertext). The only correct behavior
is to flag the page for OCR instead of serving the garbage silently --
but the text is 100% printable ASCII with word-like tokens, so it slipped
past is_garbage_text and detect_encoding_issues.
Add CipherGarbleStats, a letter-statistics discriminator that flags a
Latin-dominant sample (>=200 ASCII letters) when vowels are starved
(<=30% of letters) AND either:
- lowercase->uppercase transitions inside words exceed 10% of letter
bigrams (a shifted lowercase alphabet straddles the ASCII uppercase
block), or
- the letter histogram's cosine similarity against English letter
frequencies drops below 0.60 (catches shifts that stay within case
blocks).
Wired into analyze_text_quality (per-page, item-level) and
detect_encoding_issues (markdown-level), so extract_pages_markdown
reports needs_ocr + suspected_garbled_text and suppresses the garbage.
Thresholds validated against the 380-document pdf-evals snapshot corpus
(Swedish, Finnish, Turkish, German, romaji, schematics, all-caps and
camelCase-heavy docs): zero false positives, and byte-identical eval
output vs main. Garbled page measures vowel ratio 0.245 / case-shift
rate 0.225 / cosine 0.532; closest legitimate document on each axis is
0.264 / 0.021 / 0.801.
Fixes#118
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump pdf-inspector to 0.1.4, npm package to 1.9.11
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lib): exempt uniform-case structured content from cipher detection
Address PR review (cubic P2): the frequency branch (english_cosine < 0.60)
fired on any Latin-dominant, low-vowel letter distribution unlike English,
so non-linguistic ASCII — DNA/protein sequences, ticker symbols, hex dumps —
could be suppressed and routed to OCR despite not being garbled. Measured:
DNA cosine 0.428 / vowel ratio 0.260, protein 0.738, tickers 0.747, hex
0.549 — all would have flagged.
Add a mixed-case guard to looks_garbled: garbled English is a permutation of
natural language and carries sentence capitalization (block-straddling shifts
invert the ratio — att10k is 60% uppercase; in-case Caesar shifts preserve it
at ~3%), so both keep some of each case. The exempted structured content is
uniform case (all upper or all lower). Requiring the minority case to be >=1%
of ASCII letters exempts single-case sequences while preserving both garble
signals, including the in-case-shift scenario the frequency branch exists for.
Strictly tightens the detector: it can only remove flags, so the eval corpus
stays at zero false positives (verified byte-identical to a baseline main
binary across all 185 PDFs) and att10k remains flagged. Adds regression tests
for DNA, protein, tickers, and an in-case Caesar shift.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lib): make cipher detection case-agnostic via sorted-histogram shape
Address PR review follow-up: the mixed-case guard from the previous commit
returned before the vowel/frequency checks, creating a blind spot — a
uniform-case (all-lower or all-upper) substitution cipher is a plausible
broken-CMap output and would bypass OCR entirely.
Replace the case proxy with the actual invariant. A substitution cipher is
a bijection over a real language's alphabet, so it preserves the frequency
SHAPE (the sorted histogram) while scrambling letter POSITIONS (the unsorted
histogram). Signal 2 now flags when english_cosine < 0.60 (positions unlike
English) AND english_shape_cosine >= 0.90 (profile is still English-shaped).
This is independent of case, so it catches all-lower, all-upper, and
case-straddling shifts alike.
The exempted structured content fails one half: DNA/hex dumps have too steep
a profile (shape cosine 0.74 / 0.81 < 0.90), while protein sequences, ticker
symbols and base64 are not sufficiently unlike English in position (unsorted
cosine 0.74 / 0.75 / 0.77 >= 0.60). All stay out of OCR.
Still strictly corpus-safe: every real Latin document scores unsorted cosine
>= 0.70 (min 0.80), far above the 0.60 gate, so none can reach Signal 2.
Re-verified byte-identical to a baseline main binary across all 185 eval
PDFs; att10k remains flagged. Drops the now-unused case counters and adds
all-lowercase / all-uppercase shifted-prose regression tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: source Python package version from Cargo.toml via maturin
Address PR review (cubic P2): pyproject.toml pinned version = "0.1.0",
which overrides Cargo.toml, so a maturin build produced a 0.1.0 Python
artifact regardless of the crate version (it had drifted since the PyO3
bindings were added). Switch to dynamic = ["version"] so maturin sources
the version from Cargo.toml [package] version and the two can no longer
diverge. No workflow auto-publishes the Python package, so this is metadata
hygiene rather than a release-path fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
FontCMaps::from_doc can spend 4.5+ seconds decompressing and parsing
large embedded TrueType fonts for CID fonts with sparse ToUnicode
CMaps. For extract_text_in_regions (hybrid OCR pipeline), this is
unnecessary — fonts that can't be decoded cheaply will produce
empty/garbage text, triggering needs_ocr=true and GPU OCR fallback.
Changes:
- Add FontCMaps::from_doc_pages_fast() that skips TrueType font
fallback parsing (build_fallback_cmap_for_type0) and Identity-H/V
second pass entirely
- Add FontCMaps::from_doc_pages() for filtered page sets
- extract_text_in_regions_mem uses fast mode
- Restructure fallback chain: try cheap fallbacks first, only attempt
expensive TrueType parsing when needed and not in fast mode
Benchmark on nihms-1771367.pdf (19-page chemistry paper):
- FontCMaps fast: 201µs
- FontCMaps slow: 4.47s
- 22,000x speedup on font parsing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Both bindings now expose the same 6 function families: process, detect,
classify, extractText, extractTextWithPositions, and extractTextInRegions.
Bumps PyO3 from 0.22 to 0.25 for Python 3.14 support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Points to firecrawl/lopdf@edf6279 which fixes decompressed_content()
for streams without a /Filter entry. Our xobjects.rs workaround remains
as a defensive fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: tagged PDF structure tree support for semantic markdown generation
Parse /StructTreeRoot from tagged PDFs and use semantic roles (H1-H6, P,
LI, BlockQuote, Code, Caption) to improve markdown output. Structure tree
headings add to font-size heuristics without suppressing them. Coverage
threshold (≥50%) ensures only properly tagged PDFs activate this path.
Phase 1: Parse structure tree with role maps, MCID collection, flattening
Phase 2: Capture MCIDs from BMC/BDC operators, tag TextItems
Phase 3: Structure-aware markdown generation in convert loop
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: accumulate consecutive code lines into single fenced block
Per-line code fencing produced broken markdown for multi-line code
blocks (separate ``` open/close per line). Unify struct-tree Code
role and font-based monospace detection into a single is_code_line
check with in_code_block state for proper accumulation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add tagged PDF fixture with Firecrawl docs content
Synthetic 7-page PDF with rich structure tree exercising H1, H2, H3,
P, Code, LI, Caption, TH, TD roles. Generated via fpdf2 script.
Integration test verifies struct tree parsing and code fence output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: remove python PDF generator script from repo
Keep the generated fixture PDF but don't track the generator script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle malformed bare-name struct types in tagged PDFs
Some PDF generators (e.g. fpdf2) write /S Code instead of /S /Code
in structure elements. lopdf silently drops these objects since bare
tokens are invalid PDF syntax. Add a pre-processor that scans for
known bare struct type names and prepends / before loading.
Unifies path and memory loading through the same fix pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: update lopdf dependency to main branch
The firecrawl/zlib-checksum-encrypted branch was merged and deleted.
Point to main which includes all previously merged fixes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: switch lopdf to upstream repo pinned at 845cd3d
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Expose the pdf-inspector Rust library as a Python package using PyO3 + maturin.
Python users can now `pip install` and use `import pdf_inspector` for PDF
classification, text extraction, and markdown conversion with native Rust speed.
Adds:
- src/python.rs: PyO3 bindings (process_pdf, detect_pdf, extract_text, etc.)
- pyproject.toml: maturin build configuration
- pdf_inspector.pyi: type stubs for IDE support
- tests/test_python.py: 21 pytest tests covering all Python API functions
- examples/basic_usage.py: example script demonstrating all features
- Updated README with Python quick start and API reference
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Arabic PDFs using ToUnicode maps to presentation forms (U+FB50-FDFF,
U+FE70-FEFE) produced unreadable output. Apply NFKC normalization to
convert back to base Arabic (U+0600-06FF), then reverse character order
within items that had presentation forms (visual→logical order).
Improves Arabic PDF word overlap with Mistral from ~4.5% to ~78.6%.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Points to firecrawl/lopdf branch that falls back to raw deflate when
zlib adler32 checksum is corrupted after RC4 decryption. Fixes 5
encrypted BrokerCheck PDFs that were misclassified as scanned.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The firecrawl/fix-cmap-optional-slash branch caused a regression on
the Italian PDF (garbled CMap-encoded text). The EPEVER fix was a
wrong diagnosis — the PDF extracts correctly on upstream main.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switches to a new lopdf branch that includes the CMap parser fix for
PDFs with uncompressed ToUnicode streams that omit the leading slash
on metadata keys (CIDSystemInfo, CMapName, CMapType).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract axis-aligned rectangles from W/W* clip operators in content
streams — many PDFs define table cells as clipping paths instead of
stroked rects. Add merged-cluster fallback in detect_tables_from_rects()
that merges all cluster rects when per-cluster detection fails or only
produces narrow false-positives (≤3 columns). Uses rect Y-edges for
rows and text X-clustering for columns.
Also updates lopdf to firecrawl fork (fix-leading-whitespace branch).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The R6 encryption fix has been merged upstream, so switch from the
fork back to the canonical repository.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use abimaelmartell/lopdf firecrawl/fix-r6-encryption branch which
accepts /O and /U values longer than 48 bytes in V5/R6 encrypted PDFs.
This unblocks 4 encrypted PDFs in the bench suite that were previously
rejected with InvalidHashLength.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge CMap and Differences encoding at the byte level for single-byte
fonts, fixing garbled output when partial ToUnicode CMaps caused Latin-1
fallback to block the Differences path. Add Adobe-Korea1 CID-to-Unicode
predefined mapping for Identity-H CID fonts without ToUnicode streams,
and support TrueType cmap extraction via ttf-parser for embedded fonts.
Also handle suffixed glyph names (e.g. zero.tf, a.ss01) per Adobe spec.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates double-parsing of PDFs by using the lopdf document model exclusively
for ToUnicode CMap extraction. The old raw byte scanner parsed PDFs separately
from lopdf and only handled FlateDecode, while lopdf handles FlateDecode + LZW +
ASCII85. The new approach walks page fonts and Form XObject fonts via the document
API, yielding ~7x speedup on text-heavy PDFs and ~1.2x overall.
- Add FontCMaps::from_doc() with recursive Form XObject font walking
- Remove ~270 lines of raw byte scanning code (from_pdf_bytes, extract_stream_from_raw_pdf, etc.)
- Remove flate2 dependency (lopdf handles decompression internally)
- Remove dead CMap lookup fallback branches (by_name, get_with_obj, base_font_name)
- Remove unused font_base_names parameter from extract_text_from_operand
- Skip U+FFFD replacement characters in CMap decode (PDF notdef glyph markers)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --pages flag to insert <!-- Page N --> markers between pages
- Add --select-pages flag to process only specific pages (e.g. 1,3,5-10)
- Wire MarkdownOptions and page filter through process_pdf_with_config
- Fix fuzzy CMap matching that caused Cyrillic substitution on Latin text
- Fix whitespace Tj items not advancing text matrix (broke gap detection)
- Tighten single-char fragment join threshold from 0.25 to 0.20
- Gitignore debug/diagnostic binaries and remove tracked ones
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Image-only pages (e.g. scanned cover pages) were appended at the end of
output instead of appearing in page order. Now flushes tables/images for
intermediate pages with no text lines during page transitions.
Also collapses dot-leader runs (4+ dots) into " ... " for cleaner TOC
rendering, and adds debug_pages binary for inspecting per-page items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add pairwise cross-row column alignment check to find_table_regions_strict().
Real tables have fixed column X positions that repeat across rows (score ~1.0),
while paragraph text has varying word positions (score ~0.3-0.4). Regions with
average pairwise alignment score < 0.5 are now rejected.
Also remove the num_cols >= 5 content-check bypass for BodyFont mode, requiring
body-font tables to always have ≥30% data-like cell content.
Fixes false table detection across 9+ PDFs including AI_Cultural_Collections
(-730 false lines), 131212888 (-740), Data-Processing-Agreement (-108),
CEP_DN_Interest_Rates (-93). No regressions on legitimate tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Smart PDF type detection (text vs scanned) without full document load
- Text extraction using lopdf directly
- Markdown conversion with header/list/code detection
- CLI tools: detect-pdf, pdf2md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
