main
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
585d36e6a6 |
fix: recover from a corrupted startxref pointer (#230)
* fix: recover from a corrupted startxref pointer Fixes #228. A PDF whose startxref pointer has been corrupted to point at the wrong byte offset — a single flipped digit, which is what damaged writers emit in the wild — was entirely unprocessable: every entry point (classify_pdf, extract_pages_markdown, process_pdf) raised "Invalid PDF structure", even though the file's object data, real xref table, and trailer were all completely intact just past the wrong pointer. Both pypdf and pdfium recover from this by locating the real table directly instead of trusting the pointer; lopdf doesn't. Added a new repair candidate (alongside the existing missing-%%EOF-marker and stripped-leading-bytes repairs in repair_pdf_container_candidates): scan the buffer for the real, standalone `xref` keyword and append a corrected trailing `startxref`/`%%EOF` block. lopdf's own get_xref_start always reads the *last* `%%EOF` in the final 512 bytes of the buffer and the `startxref` value immediately before it, so the appended block transparently supersedes the corrupted one already in the file — no in-place byte surgery on content the original writer produced. Scoped to classic (non-stream) xref tables, matching the reported repro and the common case; a corrupted pointer into a cross-reference *stream* (`N 0 obj << /Type /XRef ...>>`, some PDF 1.5+ writers) would need the containing object's number, not just a byte offset — out of scope here. Verified against the issue's exact repro (a valid one-page PDF with a single corrupted byte in its startxref offset): before this fix, process_pdf/classify_pdf/extract_pages_markdown all raised "Invalid PDF structure"; after, both the page count and the real extracted text ("Order Detail Report by Account", "WIDGET ASSEMBLY", the dollar amount) come back correctly. New regression test added. Full suite (859 tests, 1 new) passes; cargo clippy --all-targets -- -D warnings unchanged at 28 pre-existing/unrelated errors. * fix: validate xref table shape and scan in a single reverse pass Addresses cubic-dev-ai's review of #230. - P2 (correctness/safety): the recovery candidate trusted the last standalone "xref" token unconditionally, without confirming it's actually a cross-reference table. A coincidental "xref" substring inside unrelated content — a stream, a string, uncompressed metadata — could get "repaired" against a bogus offset, letting lopdf load successfully against garbage instead of returning a clean error: a real failure turned into silent data corruption on the fallback path. Added looks_like_xref_subsection_header, which confirms a plausible classic xref subsection header (`<start-id> <count>`, e.g. "0 6" — the shape every real classic table starts with) actually follows the candidate token before accepting it. find_last_valid_xref_table_start now walks backward from the end of the buffer until it finds a token that both stands alone *and* validates, rather than accepting the first (rightmost) standalone match unconditionally. - P2 (performance): the old scan re-invoked `buf[..search_end].windows(4).rposition(...)` on a shrinking prefix every time a candidate token failed the boundary check, which is quadratic on a pathological buffer with many non-standalone "xref" occurrences. Rewrote as a single reverse byte-index walk — O(n) regardless of how many false candidates it has to reject along the way. Added direct unit tests on the byte-level scan (more precise than constructing adversarial full PDFs, and the coincidental-match scenario can't be represented in an integration-test fixture anyway since reportlab compresses page content by default): a coincidental standalone "xref" with no subsection header is rejected; a real classic table is found; a coincidental match positioned *after* the real table in the buffer doesn't shadow it; "xref" as a substring of "startxref" still doesn't match. The original #228 repro (corrupted startxref pointer, real table otherwise intact) is unaffected — verified manually in addition to the existing integration test. Full suite (863 tests, 5 new) passes; cargo clippy --all-targets -- -D warnings unchanged at 28 pre-existing/unrelated errors. * fix: reject xref subsection count runs with trailing garbage looks_like_xref_subsection_header validated that a count run of digits followed the whitespace separator, but never checked what came after it. A coincidental "xref\n0 6garbage" in stream/literal content would still validate as a real subsection header shape and get repaired against a bogus offset. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Abimael Martell <1450169+abimaelmartell@users.noreply.github.com> |
||
|
|
371de80b14 |
fix: extract_pages_markdown's needs_ocr now agrees with classify_pdf (#231)
* fix: extract_pages_markdown's needs_ocr now agrees with classify_pdf Fixes #227. extract_pages_markdown_mem computed its per-page needs_ocr entirely from text-quality signals: decoding/garble issues, empty markdown, GID fonts, garbage-text ratio. It had no awareness of the page's image content at all — so a page that is fundamentally a full-page scan with a little genuine native text drawn over it (a header, a stamp, a cover-sheet annotation) extracts that text cleanly, trips none of the text-quality checks, and reports needs_ocr=false — while classify_pdf/detect_pdf_type correctly see the dominant background image and flag the same page as needing OCR. Two public APIs answering the same question, silently disagreeing, in the unsafe direction (skipping OCR on a page that needs it). Exposed detector::analyze_page_images at crate visibility (was private) and call it per page in extract_pages_markdown_mem's loop — the same "large background image" signal (>50% page coverage) that already powers has_template_image in classify_pdf/detect_pdf_type, rather than reimplementing image-area detection a second time with its own thresholds that could drift out of sync again. When it's true, the page is flagged needs_ocr (with OCR_REASON_SCANNED added to ocr_reasons_by_page, matching how the same signal is already reported elsewhere) and its markdown is blanked, exactly like the existing text-quality-triggered needs_ocr paths already do — no special-casing added for "cleanly-extracted-but-still-a-scan" text. Verified against the issue's exact repro (a full-page raster with one native text line drawn over it, built via reportlab/pillow): before this fix, extract_pages_markdown_bytes reported page 0 needs_ocr=False with the header line as markdown while classify_pdf_bytes correctly flagged pages_needing_ocr=[0]; after, both agree needs_ocr=True and the page's markdown is empty. Confirmed no regression on a normal text-based fixture (nexo-price-en.pdf: needs_ocr stays False, full markdown returned). New Rust regression test added exercising both APIs against the same fixture. Full suite (860 tests, 1 new) passes; cargo clippy --all-targets -- -D warnings unchanged at 28 pre-existing/unrelated errors. * fix: gate has_template_image behind the same OCR signals classify_pdf uses extract_pages_markdown_mem was treating has_template_image alone as sufficient to force needs_ocr=true and discard the page's markdown, but classify_pdf/detect_pdf_type never treats that raw signal alone as needing OCR. A text page with a full-bleed watermark, letterhead, or large figure would get its clean markdown wrongly blanked and routed to OCR. Added page_template_image_needs_ocr(), mirroring the two distinct signals classify_pdf actually uses to decide a template-image page needs OCR: the looks_like_scan gate (image_count <= 1, few text ops, low alphanumeric diversity) used for Mixed-type routing, and the insufficient-text-volume signal (text_operator_count < 10) that routes a page with a dominant background image and only a couple of native text calls to PdfType::ImageBased independent of looks_like_scan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix: match per-page OCR threshold and add missing vector-text signal Two follow-up findings on the has_template_image gate added in the previous commit: 1. insufficient_text used a hard-coded threshold of 10 text operators, but Mixed-type per-page routing (the actual per-page decision this function tries to agree with) uses config.min_text_ops_per_page (default 3). The higher 10 threshold was borrowed from a *different* classify_pdf code path — the effective_min_ops floor used only for whole-document ImageBased/Scanned classification, a cross-page aggregate this per-page function can't replicate anyway. Using the lower per-page threshold removes a real disagreement window (3-9 text ops with high alphanumeric diversity) without breaking the #227 regression fixture (text_ops=1, still well under 3). 2. extract_pages_markdown_mem never checked has_vector_text at all, even though Mixed-type per-page routing always sends vector-outlined-text pages to OCR (outlined glyphs can't be extracted as text). A page with massive path ops plus a short genuine caption could extract that caption cleanly, slipping past the existing empty/garbage-text checks. Added page_has_vector_text() and wired it into needs_ocr the same way has_template_image is. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * perf: compute template-image and vector-text OCR signals in one pass page_template_image_needs_ocr and page_has_vector_text each called analyze_page_content independently, so every requested page's content streams (page + XObjects) and image coverage were decompressed and scanned twice per page with one result discarded each time. detect_from_document avoids this by caching its per-page PageAnalysis; extract_pages_markdown_mem had no such cache. Merged both into page_ocr_signals(), a single analyze_page_content call returning both signals as a tuple. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Abimael Martell <1450169+abimaelmartell@users.noreply.github.com> |