Abimael MartellandClaude Fable 5
6e5e5849c8
* fix(lib): detect substitution-cipher garbled text from broken ToUnicode CMaps ParseBench text_simple__att10k.pdf (issue #118) ships Type0/Identity-H fonts whose ToUnicode CMaps are authored garbled: every bfrange maps with a wrong constant delta, so text extracts as pure-ASCII ciphertext ("Certificate" -> "8VceZWZTReV"). The embedded subset font has no cmap table and no glyph names, so no decode source can recover the real text (poppler and mupdf emit the same ciphertext). The only correct behavior is to flag the page for OCR instead of serving the garbage silently -- but the text is 100% printable ASCII with word-like tokens, so it slipped past is_garbage_text and detect_encoding_issues. Add CipherGarbleStats, a letter-statistics discriminator that flags a Latin-dominant sample (>=200 ASCII letters) when vowels are starved (<=30% of letters) AND either: - lowercase->uppercase transitions inside words exceed 10% of letter bigrams (a shifted lowercase alphabet straddles the ASCII uppercase block), or - the letter histogram's cosine similarity against English letter frequencies drops below 0.60 (catches shifts that stay within case blocks). Wired into analyze_text_quality (per-page, item-level) and detect_encoding_issues (markdown-level), so extract_pages_markdown reports needs_ocr + suspected_garbled_text and suppresses the garbage. Thresholds validated against the 380-document pdf-evals snapshot corpus (Swedish, Finnish, Turkish, German, romaji, schematics, all-caps and camelCase-heavy docs): zero false positives, and byte-identical eval output vs main. Garbled page measures vowel ratio 0.245 / case-shift rate 0.225 / cosine 0.532; closest legitimate document on each axis is 0.264 / 0.021 / 0.801. Fixes #118 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump pdf-inspector to 0.1.4, npm package to 1.9.11 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(lib): exempt uniform-case structured content from cipher detection Address PR review (cubic P2): the frequency branch (english_cosine < 0.60) fired on any Latin-dominant, low-vowel letter distribution unlike English, so non-linguistic ASCII — DNA/protein sequences, ticker symbols, hex dumps — could be suppressed and routed to OCR despite not being garbled. Measured: DNA cosine 0.428 / vowel ratio 0.260, protein 0.738, tickers 0.747, hex 0.549 — all would have flagged. Add a mixed-case guard to looks_garbled: garbled English is a permutation of natural language and carries sentence capitalization (block-straddling shifts invert the ratio — att10k is 60% uppercase; in-case Caesar shifts preserve it at ~3%), so both keep some of each case. The exempted structured content is uniform case (all upper or all lower). Requiring the minority case to be >=1% of ASCII letters exempts single-case sequences while preserving both garble signals, including the in-case-shift scenario the frequency branch exists for. Strictly tightens the detector: it can only remove flags, so the eval corpus stays at zero false positives (verified byte-identical to a baseline main binary across all 185 PDFs) and att10k remains flagged. Adds regression tests for DNA, protein, tickers, and an in-case Caesar shift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(lib): make cipher detection case-agnostic via sorted-histogram shape Address PR review follow-up: the mixed-case guard from the previous commit returned before the vowel/frequency checks, creating a blind spot — a uniform-case (all-lower or all-upper) substitution cipher is a plausible broken-CMap output and would bypass OCR entirely. Replace the case proxy with the actual invariant. A substitution cipher is a bijection over a real language's alphabet, so it preserves the frequency SHAPE (the sorted histogram) while scrambling letter POSITIONS (the unsorted histogram). Signal 2 now flags when english_cosine < 0.60 (positions unlike English) AND english_shape_cosine >= 0.90 (profile is still English-shaped). This is independent of case, so it catches all-lower, all-upper, and case-straddling shifts alike. The exempted structured content fails one half: DNA/hex dumps have too steep a profile (shape cosine 0.74 / 0.81 < 0.90), while protein sequences, ticker symbols and base64 are not sufficiently unlike English in position (unsorted cosine 0.74 / 0.75 / 0.77 >= 0.60). All stay out of OCR. Still strictly corpus-safe: every real Latin document scores unsorted cosine >= 0.70 (min 0.80), far above the 0.60 gate, so none can reach Signal 2. Re-verified byte-identical to a baseline main binary across all 185 eval PDFs; att10k remains flagged. Drops the now-unused case counters and adds all-lowercase / all-uppercase shifted-prose regression tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: source Python package version from Cargo.toml via maturin Address PR review (cubic P2): pyproject.toml pinned version = "0.1.0", which overrides Cargo.toml, so a maturin build produced a 0.1.0 Python artifact regardless of the crate version (it had drifted since the PyO3 bindings were added). Switch to dynamic = ["version"] so maturin sources the version from Cargo.toml [package] version and the two can no longer diverge. No workflow auto-publishes the Python package, so this is metadata hygiene rather than a release-path fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
24 lines
776 B
TOML
24 lines
776 B
TOML
[build-system]
|
|
requires = ["maturin>=1.0,<2.0"]
|
|
build-backend = "maturin"
|
|
|
|
[project]
|
|
name = "pdf-inspector"
|
|
# Version is sourced from Cargo.toml [package] version by maturin so the Python
|
|
# artifact always tracks the crate release instead of drifting on its own.
|
|
dynamic = ["version"]
|
|
description = "Fast PDF inspection, classification, and text extraction with smart scanned vs text-based detection"
|
|
license = { text = "MIT" }
|
|
requires-python = ">=3.8"
|
|
classifiers = [
|
|
"Programming Language :: Rust",
|
|
"Programming Language :: Python :: Implementation :: CPython",
|
|
"Programming Language :: Python :: 3",
|
|
"License :: OSI Approved :: MIT License",
|
|
"Operating System :: OS Independent",
|
|
"Topic :: Text Processing",
|
|
]
|
|
|
|
[tool.maturin]
|
|
features = ["python"]
|