399e83caf289c7b03679ee220fbbe818be01d983
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
585d36e6a6 |
fix: recover from a corrupted startxref pointer (#230)
* fix: recover from a corrupted startxref pointer Fixes #228. A PDF whose startxref pointer has been corrupted to point at the wrong byte offset — a single flipped digit, which is what damaged writers emit in the wild — was entirely unprocessable: every entry point (classify_pdf, extract_pages_markdown, process_pdf) raised "Invalid PDF structure", even though the file's object data, real xref table, and trailer were all completely intact just past the wrong pointer. Both pypdf and pdfium recover from this by locating the real table directly instead of trusting the pointer; lopdf doesn't. Added a new repair candidate (alongside the existing missing-%%EOF-marker and stripped-leading-bytes repairs in repair_pdf_container_candidates): scan the buffer for the real, standalone `xref` keyword and append a corrected trailing `startxref`/`%%EOF` block. lopdf's own get_xref_start always reads the *last* `%%EOF` in the final 512 bytes of the buffer and the `startxref` value immediately before it, so the appended block transparently supersedes the corrupted one already in the file — no in-place byte surgery on content the original writer produced. Scoped to classic (non-stream) xref tables, matching the reported repro and the common case; a corrupted pointer into a cross-reference *stream* (`N 0 obj << /Type /XRef ...>>`, some PDF 1.5+ writers) would need the containing object's number, not just a byte offset — out of scope here. Verified against the issue's exact repro (a valid one-page PDF with a single corrupted byte in its startxref offset): before this fix, process_pdf/classify_pdf/extract_pages_markdown all raised "Invalid PDF structure"; after, both the page count and the real extracted text ("Order Detail Report by Account", "WIDGET ASSEMBLY", the dollar amount) come back correctly. New regression test added. Full suite (859 tests, 1 new) passes; cargo clippy --all-targets -- -D warnings unchanged at 28 pre-existing/unrelated errors. * fix: validate xref table shape and scan in a single reverse pass Addresses cubic-dev-ai's review of #230. - P2 (correctness/safety): the recovery candidate trusted the last standalone "xref" token unconditionally, without confirming it's actually a cross-reference table. A coincidental "xref" substring inside unrelated content — a stream, a string, uncompressed metadata — could get "repaired" against a bogus offset, letting lopdf load successfully against garbage instead of returning a clean error: a real failure turned into silent data corruption on the fallback path. Added looks_like_xref_subsection_header, which confirms a plausible classic xref subsection header (`<start-id> <count>`, e.g. "0 6" — the shape every real classic table starts with) actually follows the candidate token before accepting it. find_last_valid_xref_table_start now walks backward from the end of the buffer until it finds a token that both stands alone *and* validates, rather than accepting the first (rightmost) standalone match unconditionally. - P2 (performance): the old scan re-invoked `buf[..search_end].windows(4).rposition(...)` on a shrinking prefix every time a candidate token failed the boundary check, which is quadratic on a pathological buffer with many non-standalone "xref" occurrences. Rewrote as a single reverse byte-index walk — O(n) regardless of how many false candidates it has to reject along the way. Added direct unit tests on the byte-level scan (more precise than constructing adversarial full PDFs, and the coincidental-match scenario can't be represented in an integration-test fixture anyway since reportlab compresses page content by default): a coincidental standalone "xref" with no subsection header is rejected; a real classic table is found; a coincidental match positioned *after* the real table in the buffer doesn't shadow it; "xref" as a substring of "startxref" still doesn't match. The original #228 repro (corrupted startxref pointer, real table otherwise intact) is unaffected — verified manually in addition to the existing integration test. Full suite (863 tests, 5 new) passes; cargo clippy --all-targets -- -D warnings unchanged at 28 pre-existing/unrelated errors. * fix: reject xref subsection count runs with trailing garbage looks_like_xref_subsection_header validated that a count run of digits followed the whitespace separator, but never checked what came after it. A coincidental "xref\n0 6garbage" in stream/literal content would still validate as a real subsection header shape and get repaired against a bogus offset. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Abimael Martell <1450169+abimaelmartell@users.noreply.github.com> |
||
|
|
371de80b14 |
fix: extract_pages_markdown's needs_ocr now agrees with classify_pdf (#231)
* fix: extract_pages_markdown's needs_ocr now agrees with classify_pdf Fixes #227. extract_pages_markdown_mem computed its per-page needs_ocr entirely from text-quality signals: decoding/garble issues, empty markdown, GID fonts, garbage-text ratio. It had no awareness of the page's image content at all — so a page that is fundamentally a full-page scan with a little genuine native text drawn over it (a header, a stamp, a cover-sheet annotation) extracts that text cleanly, trips none of the text-quality checks, and reports needs_ocr=false — while classify_pdf/detect_pdf_type correctly see the dominant background image and flag the same page as needing OCR. Two public APIs answering the same question, silently disagreeing, in the unsafe direction (skipping OCR on a page that needs it). Exposed detector::analyze_page_images at crate visibility (was private) and call it per page in extract_pages_markdown_mem's loop — the same "large background image" signal (>50% page coverage) that already powers has_template_image in classify_pdf/detect_pdf_type, rather than reimplementing image-area detection a second time with its own thresholds that could drift out of sync again. When it's true, the page is flagged needs_ocr (with OCR_REASON_SCANNED added to ocr_reasons_by_page, matching how the same signal is already reported elsewhere) and its markdown is blanked, exactly like the existing text-quality-triggered needs_ocr paths already do — no special-casing added for "cleanly-extracted-but-still-a-scan" text. Verified against the issue's exact repro (a full-page raster with one native text line drawn over it, built via reportlab/pillow): before this fix, extract_pages_markdown_bytes reported page 0 needs_ocr=False with the header line as markdown while classify_pdf_bytes correctly flagged pages_needing_ocr=[0]; after, both agree needs_ocr=True and the page's markdown is empty. Confirmed no regression on a normal text-based fixture (nexo-price-en.pdf: needs_ocr stays False, full markdown returned). New Rust regression test added exercising both APIs against the same fixture. Full suite (860 tests, 1 new) passes; cargo clippy --all-targets -- -D warnings unchanged at 28 pre-existing/unrelated errors. * fix: gate has_template_image behind the same OCR signals classify_pdf uses extract_pages_markdown_mem was treating has_template_image alone as sufficient to force needs_ocr=true and discard the page's markdown, but classify_pdf/detect_pdf_type never treats that raw signal alone as needing OCR. A text page with a full-bleed watermark, letterhead, or large figure would get its clean markdown wrongly blanked and routed to OCR. Added page_template_image_needs_ocr(), mirroring the two distinct signals classify_pdf actually uses to decide a template-image page needs OCR: the looks_like_scan gate (image_count <= 1, few text ops, low alphanumeric diversity) used for Mixed-type routing, and the insufficient-text-volume signal (text_operator_count < 10) that routes a page with a dominant background image and only a couple of native text calls to PdfType::ImageBased independent of looks_like_scan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix: match per-page OCR threshold and add missing vector-text signal Two follow-up findings on the has_template_image gate added in the previous commit: 1. insufficient_text used a hard-coded threshold of 10 text operators, but Mixed-type per-page routing (the actual per-page decision this function tries to agree with) uses config.min_text_ops_per_page (default 3). The higher 10 threshold was borrowed from a *different* classify_pdf code path — the effective_min_ops floor used only for whole-document ImageBased/Scanned classification, a cross-page aggregate this per-page function can't replicate anyway. Using the lower per-page threshold removes a real disagreement window (3-9 text ops with high alphanumeric diversity) without breaking the #227 regression fixture (text_ops=1, still well under 3). 2. extract_pages_markdown_mem never checked has_vector_text at all, even though Mixed-type per-page routing always sends vector-outlined-text pages to OCR (outlined glyphs can't be extracted as text). A page with massive path ops plus a short genuine caption could extract that caption cleanly, slipping past the existing empty/garbage-text checks. Added page_has_vector_text() and wired it into needs_ocr the same way has_template_image is. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * perf: compute template-image and vector-text OCR signals in one pass page_template_image_needs_ocr and page_has_vector_text each called analyze_page_content independently, so every requested page's content streams (page + XObjects) and image coverage were decompressed and scanned twice per page with one result discarded each time. detect_from_document avoids this by caching its per-page PageAnalysis; extract_pages_markdown_mem had no such cache. Merged both into page_ocr_signals(), a single analyze_page_content call returning both signals as a tuple. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Abimael Martell <1450169+abimaelmartell@users.noreply.github.com> |
||
|
Abimael Martell
|
12e9a655e3 |
fix(extractor): supply built-in metrics for non-embedded base-14 fonts (#241)
* fix(extractor): supply built-in metrics for non-embedded base-14 fonts PDFs may legally omit /Widths for non-embedded standard fonts (Times, Helvetica, Courier, Symbol, ZapfDingbats) — the spec requires the reader to supply the metrics. We returned None, so every glyph advanced 0 and each text item got width 0, silently breaking every gap-based heuristic downstream: space synthesis, sub/superscript detection, table column detection, heading merging. - src/extractor/base14.rs: Adobe Core-14 AFM width tables keyed by Unicode char, plus the standard Symbol/ZapfDingbats encoding vectors (their glyphs sit at byte positions unrelated to Latin text, so widths must resolve through the built-in encoding, not cp1252) - Width resolution order: Differences -> built-in encoding -> the same cp1252-style fallback the text decoder uses, so a code's advance always matches the character we emit for it - Type3 visual sizing: PK bitmap fonts (dvips) use FontMatrix [1 0 0 -1 0 0] with nominal sizes like 0.12pt; scale by FontBBox height x |matrix_y|. Applied in the page-stream and Form XObject paths. Indirect numeric array elements are resolved before use. Effect on Shannon's 'A Mathematical Theory of Communication' (1998 dvips/Distiller, the reported case): glued sentences 95 -> 5. Corpus impact: 12 of 184 eval documents, e.g. Data-Processing-Agreement recovers a paragraph that a phantom table had shredded into cells. Layout heuristics tuned on the same document (indent-based paragraph breaks, heading reclassification, table script filtering) are held back for a separate PR — they change ~98 further documents and need to be justified against the corpus, not against one PDF. * review: narrow Type3 rescaling to self-inconsistent fonts; dedup + test all width tables Addresses cubic review on #241, plus a follow-up from a local cubic run. - Type3 visual scaling was applied to every Type3 font whose FontBBox height x |matrix_y| deviated >5% from 1.0. FontBBox is the glyph box, not the em box, so a conventional 1/1000-matrix font with a descender..ascender bbox (~700 units) computed 0.7 and had every reported size shrunk by 30% — corrupting the drop-cap, heading-tier, sub/superscript and table heuristics this is meant to fix. First attempt gated on the matrix being unit-scale, but a local cubic run pointed out that wrongly excludes valid non-standard matrices (a 0.005 matrix with a full-em bbox legitimately needs a 5x scale). The product is the right discriminator, not the matrix: a self-consistent font lands near 1.0 because the matrix is the reciprocal of the glyph-space em, so only a wildly inconsistent one (dvips/PK bitmap fonts sit at ~159) is renormalized. Band widened to [0.25, 4.0]. Corpus effect: 12 -> 7 documents change. The 5 that drop out were being wrongly rescaled — including Data-Processing-Agreement, whose phantom-table fix turned out to come from this bug rather than from the width fallback, so it is correctly given up. - base14: all 14 width tables now covered by the sort-invariant test via an ALL_TABLES registry, not a hand-picked subset. - base14: identical tables share one static (all four Courier variants are monospace 600; the oblique Helvetica variants match their upright forms), removing 5 duplicate copies. * test: refresh Shannon snapshot after merging main CI checks out a merge of the PR head with main, and main advanced 8 commits since this branch was cut — including #201 (contextual digit runs), #240 and #253 (markdown fixes). Those change extraction output, so a snapshot generated on the unmerged branch could not match; the Test job failed on the merge commit while passing on the branch itself. The merged behaviour is better: the footnote marker '2' before 'Hartley, R. V. L.' is now recovered instead of dropped. 950 tests pass on the merged tree, clippy clean. |
||
|
Abimael Martell
|
a38efcf142 | feat: --password support for encrypted PDFs (#138) | ||
|
Abimael MartellandClaude Fable 5
|
6e5e5849c8 |
Detect substitution-cipher garbled text from broken ToUnicode CMaps (#120)
* fix(lib): detect substitution-cipher garbled text from broken ToUnicode CMaps ParseBench text_simple__att10k.pdf (issue #118) ships Type0/Identity-H fonts whose ToUnicode CMaps are authored garbled: every bfrange maps with a wrong constant delta, so text extracts as pure-ASCII ciphertext ("Certificate" -> "8VceZWZTReV"). The embedded subset font has no cmap table and no glyph names, so no decode source can recover the real text (poppler and mupdf emit the same ciphertext). The only correct behavior is to flag the page for OCR instead of serving the garbage silently -- but the text is 100% printable ASCII with word-like tokens, so it slipped past is_garbage_text and detect_encoding_issues. Add CipherGarbleStats, a letter-statistics discriminator that flags a Latin-dominant sample (>=200 ASCII letters) when vowels are starved (<=30% of letters) AND either: - lowercase->uppercase transitions inside words exceed 10% of letter bigrams (a shifted lowercase alphabet straddles the ASCII uppercase block), or - the letter histogram's cosine similarity against English letter frequencies drops below 0.60 (catches shifts that stay within case blocks). Wired into analyze_text_quality (per-page, item-level) and detect_encoding_issues (markdown-level), so extract_pages_markdown reports needs_ocr + suspected_garbled_text and suppresses the garbage. Thresholds validated against the 380-document pdf-evals snapshot corpus (Swedish, Finnish, Turkish, German, romaji, schematics, all-caps and camelCase-heavy docs): zero false positives, and byte-identical eval output vs main. Garbled page measures vowel ratio 0.245 / case-shift rate 0.225 / cosine 0.532; closest legitimate document on each axis is 0.264 / 0.021 / 0.801. Fixes #118 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump pdf-inspector to 0.1.4, npm package to 1.9.11 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(lib): exempt uniform-case structured content from cipher detection Address PR review (cubic P2): the frequency branch (english_cosine < 0.60) fired on any Latin-dominant, low-vowel letter distribution unlike English, so non-linguistic ASCII — DNA/protein sequences, ticker symbols, hex dumps — could be suppressed and routed to OCR despite not being garbled. Measured: DNA cosine 0.428 / vowel ratio 0.260, protein 0.738, tickers 0.747, hex 0.549 — all would have flagged. Add a mixed-case guard to looks_garbled: garbled English is a permutation of natural language and carries sentence capitalization (block-straddling shifts invert the ratio — att10k is 60% uppercase; in-case Caesar shifts preserve it at ~3%), so both keep some of each case. The exempted structured content is uniform case (all upper or all lower). Requiring the minority case to be >=1% of ASCII letters exempts single-case sequences while preserving both garble signals, including the in-case-shift scenario the frequency branch exists for. Strictly tightens the detector: it can only remove flags, so the eval corpus stays at zero false positives (verified byte-identical to a baseline main binary across all 185 PDFs) and att10k remains flagged. Adds regression tests for DNA, protein, tickers, and an in-case Caesar shift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(lib): make cipher detection case-agnostic via sorted-histogram shape Address PR review follow-up: the mixed-case guard from the previous commit returned before the vowel/frequency checks, creating a blind spot — a uniform-case (all-lower or all-upper) substitution cipher is a plausible broken-CMap output and would bypass OCR entirely. Replace the case proxy with the actual invariant. A substitution cipher is a bijection over a real language's alphabet, so it preserves the frequency SHAPE (the sorted histogram) while scrambling letter POSITIONS (the unsorted histogram). Signal 2 now flags when english_cosine < 0.60 (positions unlike English) AND english_shape_cosine >= 0.90 (profile is still English-shaped). This is independent of case, so it catches all-lower, all-upper, and case-straddling shifts alike. The exempted structured content fails one half: DNA/hex dumps have too steep a profile (shape cosine 0.74 / 0.81 < 0.90), while protein sequences, ticker symbols and base64 are not sufficiently unlike English in position (unsorted cosine 0.74 / 0.75 / 0.77 >= 0.60). All stay out of OCR. Still strictly corpus-safe: every real Latin document scores unsorted cosine >= 0.70 (min 0.80), far above the 0.60 gate, so none can reach Signal 2. Re-verified byte-identical to a baseline main binary across all 185 eval PDFs; att10k remains flagged. Drops the now-unused case counters and adds all-lowercase / all-uppercase shifted-prose regression tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: source Python package version from Cargo.toml via maturin Address PR review (cubic P2): pyproject.toml pinned version = "0.1.0", which overrides Cargo.toml, so a maturin build produced a 0.1.0 Python artifact regardless of the crate version (it had drifted since the PyO3 bindings were added). Switch to dynamic = ["version"] so maturin sources the version from Cargo.toml [package] version and the two can no longer diverge. No workflow auto-publishes the Python package, so this is metadata hygiene rather than a release-path fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Abimael MartellandClaude Opus 4.7
|
96c0f2102a |
tables/detect_rects: prefer rect-grid edges over text-cluster on N≥3 column tables (#82)
When a wire-bordered table has headers centered/right-aligned in their cells but data left-aligned, cluster_x_positions can both merge adjacent data columns (when the data-to-data gap is below the clamped threshold) and drop the header-only x-positions in its singleton-filter pass. The cell-rect fallback then used text-cluster column edges and lost a column or fragmented neighbor cells. Prefer rect-derived column edges when the rect grid has 3+ columns and every rect column holds multiple text items. The all-cols-populated check protects against decorative or background rects (prose laid out in a frame, cell-fill rects with extra borders) that would otherwise split a logical column into spurious sub-columns. The existing prose-in-frame, well-distributed-columns, and wireless-prose guards still fire for the cases they were built for. Bump napi version 1.8.7 → 1.8.8. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Abimael Martell
|
6086577d11 | tables/detect_rects: emit grid for multiline indented cells (#79) | ||
|
Abimael MartellandCursor
|
f2186ec1aa |
tables/detect_rects: don't accept relaxed grid on wireless prose (#78)
Require rect-derived column evidence before relaxing prose checks for two-column cell-rect fallbacks, so text-position alignment alone cannot synthesize a vector grid on wireless content. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Abimael MartellandClaude Opus 4.7
|
59b17f372a |
tables: tighten prose-in-frame rejection (#77)
* tables: lift detection on shaded-header + alt-row tables (#wired-grids) Production telemetry on `wired_high_confidence`-classified table regions showed `detect_vector_grid_in_region_mem` returning a usable grid only ~27% of the time, with the rest falling through to GLM-OCR. Three surgical fixes target the dominant production shapes: * Path-fill cell backgrounds: when the page has no `re` rects but draws cell backgrounds via `m`/`l`/`h`/`f*` sequences, prefer the fill-derived rects over the few section-level `W*` clip paths that previously won the priority gate. Activated when fill rects outnumber clip rects ≥3×. * Dedup-induced cluster splits: page-background rects could pose as containers in the sub-rect dedup and evict a slightly smaller table-frame rect, breaking adjacency between column-cell groups so each column became its own cluster. Origin-anchored containers are now disqualified from sub-rect dedup. A separate exact-duplicate pass collapses the cell-padding/text-bg/cell-border triple emissions some PDFs produce, preserving original order to avoid reshuffling table output on multi-table pages. * Prose-words rejection: the `cell-rect` fallback's whole-grid prose threshold also rejected real tables that include a description column. Now relaxed when content is well-distributed (≥75% of cols filled), while keeping the original strictness for prose-in-a-frame layouts. Two regression fixtures from the opendataloader-bench corpus, covering the dominant production failure categories: * `greencomp_competence.pdf` — 2-col shaded-header + plain-body glossary. Mirrors production crops #1 (Contractions glossary) and #6 (BIO 350 course header). * `upstage_key_functions.pdf` — 4-col shaded-header + alt-row backgrounds + merged left column. Mirrors production crops #2 (Parameter/Value alt-row), #7 (Spanish XML schema), and #8 (Córdoba multi-row header). Existing fixtures stay green (doc 51 wrapped-label, doc 128 forecast six-cols, td9264 snapshot). 133 unit + integration tests pass; clippy clean. Bumps napi/package.json 1.8.4 → 1.8.5. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * tables: tighten prose-in-frame rejection — fixes pdf-evals #30 regression PR #76's shaded-header detection lift surfaced a regression on accessory_building_permit_application_1 (TEDS 0.10 → 0.05): a paragraph of legal text laid out in a 2-column justified block was being admitted as a 10×2 fake table where every cell holds a sentence fragment ("I agree to comply...", "I", "It is the property owner's responsibility..."). Per pdf-evals PR #30 review, this is the kind of regression production users will notice — the markdown is structurally and semantically misleading. Root cause: PR #76's prose-rejection only fires for `num_cols >= 4`, so the 2-col prose-in-a-frame case slipped past it entirely. The new fill-priority + dedup changes started producing rects for this layout that 1.8.4 correctly ignored. Fix: tighten the prose-in-frame check. - Lower the column-count guard from `>= 4` to `>= 2`. - Add a content-length signal as the primary discriminator: when the prose-words trigger fires AND mean non-empty cell length exceeds 65 chars, reject regardless of column distribution. The 65-char threshold cleanly separates observed cases: accessory_building (prose-in-frame): mean 74 chars → REJECT upstage_key_functions (real 4-col table): mean 53 → admit greencomp_competence (real 2-col glossary): mean 20 → admit accessory_building (real 5×3 form data): mean 10 → admit The well-distributed-cols relaxation that PR #76 added stays — "label / value / description / benefit" tables (#7, #8 from the production crops) still pass, but only when their mean cell length stays below the prose threshold. New regression test `accessory_building_rejects_prose_in_frame` asserts both that the real 5×3 form data table survives AND the 10×2 prose block is rejected. Snapshot test `test_snapshot_td9264` updated to match new output — old snapshot captured the same prose-in-frame bug on regulatory text (paragraphs emitted as 3-col `||text||` fake-table rows). New snapshot emits clean prose paragraphs, which is correct. Verification: - cargo test --all: 424 lib + 133 integration + 2 doc tests pass - cargo fmt --check clean - cargo clippy -- -D warnings clean (lib-level; pre-existing test-level clippy issues on the wired-grids branch unaffected) - Existing fixtures stay green: forecast_table_chart_six_cols (PR #72), bits_pilani_* (PR #73), greencomp_competence_two_cols and upstage_key_functions_four_cols (PR #76). This branch is based on abimaelmartell/wired-grids so it includes PR #76's commits plus this fix on top. Suggest merging this and closing #76, OR rebasing #76 to incorporate this fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * tables: drop early-dedup atom — caused broad TOC + matrix corruption Bisected PR #76's 4 atoms against the SEC 10-K 0001104659-25-093871_183e44ac.pdf which appeared as a TEDS regression in pdf-evals PR #30. Result: atom | TOC | perf-graph | qualifications ---------------------------|-----|------------|--------------- fill-priority | ✓ | ✓ | ✓ early-dedup | ✗ | ✗ | ✗ page-bg disqualification | ✓ | ✓ | ✓ prose-relaxation | ✓ | ✓ | ✓ Early-dedup was the SOLE source of all three regressions on this doc. Tried a more conservative variant (≥3 copies only — pair-duplicates appear in legit multi-section layouts like 10-K dividers above + below section headers); didn't fix the regression. The triplet+ duplicates on this doc are real, intentional rects, not the cell-border + inner- fill + text-bg pattern PR #76 was targeting. Drop early-dedup. Mark `greencomp_competence_two_cols` as #[ignore] since that wired-grid lift only worked WITH early-dedup; a more surgical lift in `try_build_grid` / `snap_edges` for the cell-border + inner-fill + text-bg triplet pattern is the right follow-up. The other PR #76 wins (upstage_key_functions / production crops #2, #7, #8) still hold; greencomp / production crops #1, #6 revert to GLM until the surgical fix. Validation on the regression doc: 0001104659 TOC PART II markers: 4 (matches main, was 2 with PR#76) 0001104659 perf-graph data row: 2 (matches main, was 1) 0001104659 qualifications rows: 9 (matches main, was 6) Validation on the prose-frame doc: accessory_building fake-table: 0 (matches main, was 1 with PR#76) accessory_building prose intact: 1 (matches main) cargo test --all clean, cargo fmt --check clean, cargo clippy --lib -- -D warnings clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Abimael Martell
|
1f28e523fd | tables: keep wrapped labels in TSR output (#73) | ||
|
Abimael Martell
|
97fc32ac70 |
tables: prefer rect edges for cell-grid fallback (#72)
* tables: prefer rect edges in cell-grid fallback * Bump version from 1.8.1 to 1.8.2 |
||
|
Abimael MartellandClaude Opus 4.6
|
7c8b09be67 |
fix: improve table detection for numeric columns and multi-line headers (#35)
* fix: improve heuristic table detection for numeric columns and multi-line headers Two fixes for tables that have clean extractable text but fail heuristic structure detection: 1. Numeric column merge pass (grid.rs): After initial X-position clustering, adjacent clusters are merged when one is sparse (header text) and the other is dense with >50% numeric items (data column). Multi-line wrapped headers often land slightly offset from their data column — the merge closes gaps within 1.5× the clustering threshold. New is_numeric_text() helper matches decimals, percentages, negative numbers, and comma-separated thousands. 2. Duplicate-header skip (detect_heuristic.rs): Spanning super-headers like "First Degree | First Degree | Higher Degree" contain duplicate cells that trigger looks_like_partial_table_ex rejection. Now skips rows with duplicate cells when a better header candidate exists within the next 3 rows (higher fill ratio or numeric cells). Tested on BITS Pilani university report (430 pages, 314 table pages). Page 4 (multi-line header + numeric data) previously returned needs_ocr=true; now correctly detects the table structure. Eval: 197 PDFs, zero regressions, all 104+ tests pass, zero clippy. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * bump version to 0.7.1 --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Abimael MartellandClaude Opus 4.6
|
a199768c4e |
fix(layout): detect and correct rotated page text (#10)
PDFs that embed landscape content in portrait pages via a rotated text matrix (e.g. [0, b, -b, 0, tx, ty] for 90° CCW) produced garbled output because the layout engine assumed x=horizontal, y=vertical. Track the dominant text direction from combined matrices during extraction. When ≥67% of text operators are rotated, swap x↔y coordinates (with y-negation for correct reading order) for all text items, rects, and lines. Also estimate text widths from char count × font size since scale_x ≈ 0 for rotated text. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Abimael MartellandClaude Opus 4.6
|
aa387503c3 |
fix(fonts): suppress garbage CID text on OCR-flagged pages (#9)
When Identity-H or Type3 fonts lack a ToUnicode CMap and the CID-as-Unicode passthrough doesn't produce valid text, the raw CID byte values appear as mojibake (random Latin Extended characters mixed with C1 control codes). The detector already flags these pages in pages_needing_ocr, but the markdown pipeline still emitted the garbage. Now, for TextBased PDFs, we check each OCR-flagged page's extracted text for CID garbage (C1 control characters U+0080–U+009F at ≥5% density) and strip items from pages that fail the check. This is scoped to TextBased PDFs only — Mixed PDFs flag pages for OCR due to template images, not font encoding issues. Adds test fixture (shinagawa_identity_h.pdf) and integration test. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Abimael MartellandClaude Opus 4.6
|
d9c2143c32 |
feat: tagged PDF structure tree support (#4)
* feat: tagged PDF structure tree support for semantic markdown generation Parse /StructTreeRoot from tagged PDFs and use semantic roles (H1-H6, P, LI, BlockQuote, Code, Caption) to improve markdown output. Structure tree headings add to font-size heuristics without suppressing them. Coverage threshold (≥50%) ensures only properly tagged PDFs activate this path. Phase 1: Parse structure tree with role maps, MCID collection, flattening Phase 2: Capture MCIDs from BMC/BDC operators, tag TextItems Phase 3: Structure-aware markdown generation in convert loop Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: accumulate consecutive code lines into single fenced block Per-line code fencing produced broken markdown for multi-line code blocks (separate ``` open/close per line). Unify struct-tree Code role and font-based monospace detection into a single is_code_line check with in_code_block state for proper accumulation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add tagged PDF fixture with Firecrawl docs content Synthetic 7-page PDF with rich structure tree exercising H1, H2, H3, P, Code, LI, Caption, TH, TD roles. Generated via fpdf2 script. Integration test verifies struct tree parsing and code fence output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove python PDF generator script from repo Keep the generated fixture PDF but don't track the generator script. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: handle malformed bare-name struct types in tagged PDFs Some PDF generators (e.g. fpdf2) write /S Code instead of /S /Code in structure elements. lopdf silently drops these objects since bare tokens are invalid PDF syntax. Add a pre-processor that scans for known bare struct type names and prepends / before loading. Unifies path and memory loading through the same fix pipeline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: update lopdf dependency to main branch The firecrawl/zlib-checksum-encrypted branch was merged and deleted. Point to main which includes all previously merged fixes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: switch lopdf to upstream repo pinned at 845cd3d Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Abimael MartellandClaude Opus 4.6
|
1bacf34bd0 |
Add tests for row-stripe table detection
Unit test verifying short sub-header rows (e.g. month names) are not merged as continuation rows in table formatting. Snapshot test for 2013_app2.pdf pinning the full row-stripe table output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Abimael MartellandClaude Opus 4.6
|
28313e1f2d |
test: Add snapshot regression tests with PDF fixtures
Add 5 stripped public-domain PDF fixtures and golden markdown snapshots for CI regression testing. Fix non-deterministic output caused by HashMap iteration order in font stats, table heuristics, and rect clustering by adding deterministic tie-breaking. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |