mu-ref/pdf-inspector
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(regions): serve invisible (Tr 3) OCR text layers instead of needs_ocr (#351)

Browse Source 
* fix(regions): serve invisible (Tr 3) OCR text layers instead of needs_ocr

Scanned pages (archive.org-style digitizations) carry their text as an
invisible render-mode-3 layer behind the page raster. extract_text_in_regions
extracted only visible items, so such pages yielded nothing but
'[Image: ...]' placeholders and every region fell back to OCR — while the
markdown path already includes the invisible layer for Mixed PDFs. The two
extractors disagreed about the same page.

Mirror the markdown path's gate, page-scoped: when the visible pass is
effectively textless (<40 non-placeholder alphanumerics), retry with
invisible text included and adopt the retry only when it contributes real,
non-garbage text. Pages with real visible text never retry, so double-layer
PDFs (visible text plus an invisible accessibility copy) are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: strict zero-visible adoption gate, whole-layer garbage check, doc placement, discriminating tests

- Adoption now requires ZERO visible text on the page: the invisible pass
  returns visible items too, so adopting alongside any visible text would
  duplicate it. Strict gate instead of fuzzy dedupe; the dead
  inv_alnum > visible_alnum condition goes with it.
- Garbage check judges the whole recovered layer, not the first 200 items.
- Constant/helper moved above the doc block so rustdoc stays attached to
  extract_text_in_regions_mem.
- Tests: any-visible-text-blocks-adoption case (single short visible line)
  with exactly-once assertions; mode-0 guard asserts occurrence count.
- Re-verified against real scanned-book pages: all recover (9.2-11.7K chars,
  needs_ocr=false) — pure OCR-layer scans carry no visible text.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: gate adoption on visible-item presence, not alphanumeric mass

Punctuation-only visible text (zero alphanumerics) could still adopt the
invisible pass; its OCR twin in the layer would duplicate the glyphs. The
gate is now item-presence: any non-image item with non-whitespace text
blocks adoption (whitespace-only artifacts still tolerated). Pinned by a
punctuation-only fixture test; real scanned-book pages re-verified — all
recover unchanged (they carry no visible items at all).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: punctuation-gate test pins preservation, comment states fixture scope

The fixture's visible punctuation is a separate block, not mirrored in the
invisible layer — so it pins the gate, not the duplication scenario. The
doc comment now says so, and a positive assertion checks the punctuation
survives exactly once.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: retry only when invisible text was actually skipped; negative gate tests

- extract_page_text_items now reports skipped_invisible (4th return): the
  Tr-3 suppression sites set it, so the region extractor retries only when
  a recoverable layer exists. Blank pages and image-only scans without an
  OCR layer — the common scanned case — no longer pay a second
  content-stream parse.
- Negative tests: below-floor watermark layer and symbol-garbage layer are
  both rejected (region keeps only the raster placeholder). needs_ocr
  semantics for placeholder-only regions deliberately unchanged — that
  contract predates this PR and downstream pipelines handle it.
- Real scanned-book pages re-verified: recovery unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: flag the ' show-text suppression path; require nonempty strings

- P1: invisible layers shown via the ' operator never set skipped_invisible,
  so those pages kept the placeholder and fell back to GPU OCR. The '
  suppression path now flags too — pinned by a fixture whose layer is shown
  entirely via ' (nothing through Tj/TJ).
- P3: all three flag sites (Tj, TJ, ') require a nonempty string operand —
  numeric-only TJ kerning arrays and empty shows no longer trigger the
  invisible reparse.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Abimael Martell
2026-08-11 14:11:31 -07:00
committed by GitHub
co-authored by Claude Fable 5
parent 9947485a92
commit f65b25906c
4 changed files with 424 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/extractor/content_stream.rs
+47 -8
View File
@@ -137,8 +137,11 @@ fn rise_adjusted(tm: &[f32; 6], rise: f32) -> [f32; 6] {
]
}
/// Returns `(page_extraction, has_gid_fonts)` where `has_gid_fonts` indicates
/// the page uses fonts with unresolvable gid-encoded glyphs.
/// Returns `(page_extraction, has_gid_fonts, coords_rotated, skipped_invisible)`
/// where `has_gid_fonts` indicates the page uses fonts with unresolvable
/// gid-encoded glyphs and `skipped_invisible` reports that invisible (Tr 3)
/// text was present but suppressed — callers can use it to decide whether an
/// `include_invisible` retry could recover anything at all.
pub(crate) fn extract_page_text_items(
doc: &Document,
page_id: ObjectId,
@@ -146,7 +149,7 @@ pub(crate) fn extract_page_text_items(
font_cmaps: &FontCMaps,
include_invisible: bool,
style_cache: &mut FontStyleCache,
) -> Result<(PageExtraction, bool, bool), PdfError> {
) -> Result<(PageExtraction, bool, bool, bool), PdfError> {
use lopdf::content::Content;
let mut items = Vec::new();
@@ -262,12 +265,15 @@ pub(crate) fn extract_page_text_items(
content.operations.len(),
MAX_OPERATIONS
);
return Ok(((Vec::new(), Vec::new(), Vec::new()), false, false));
return Ok(((Vec::new(), Vec::new(), Vec::new()), false, false, false));
}
// Graphics state tracking
let mut ctm = [1.0f32, 0.0, 0.0, 1.0, 0.0, 0.0]; // Current Transformation Matrix
let mut text_rendering_mode: i32 = 0; // 0=fill, 1=stroke, 2=fill+stroke, 3=invisible
// Invisible (Tr 3) text was present but suppressed — reported to callers
// so an include_invisible retry is attempted only when it can recover.
let mut skipped_invisible = false;
let mut line_width: f32 = 1.0;
#[derive(Clone)]
struct SavedGraphicsState {
@@ -494,6 +500,14 @@ pub(crate) fn extract_page_text_items(
// For Mixed/template PDFs, include_invisible=true extracts
// the OCR text layer that sits behind scanned images.
if text_rendering_mode == 3 && !include_invisible {
if op
.operands
.first()
.and_then(get_operand_bytes)
.is_some_and(|raw| !raw.is_empty())
{
skipped_invisible = true;
}
if let Some(w_ts) = w_ts_opt {
text_matrix[4] += w_ts * text_matrix[0];
text_matrix[5] += w_ts * text_matrix[1];
@@ -565,6 +579,16 @@ pub(crate) fn extract_page_text_items(
if in_text_block && !op.operands.is_empty() {
if let Ok(array) = op.operands[0].as_array() {
let font_info = font_widths.get(&current_font);
// Numeric-only TJ arrays (pure kerning) show no
// text — they must not trigger the invisible retry.
if text_rendering_mode == 3
&& !include_invisible
&& array
.iter()
.any(|el| get_operand_bytes(el).is_some_and(|raw| !raw.is_empty()))
{
skipped_invisible = true;
}
let is_invisible = (text_rendering_mode == 3 && !include_invisible)
|| suppress_glyph_extraction;
// Capture first-glyph position for ActualText
@@ -770,6 +794,16 @@ pub(crate) fn extract_page_text_items(
)
})
});
if text_rendering_mode == 3
&& !include_invisible
&& op
.operands
.first()
.and_then(get_operand_bytes)
.is_some_and(|raw| !raw.is_empty())
{
skipped_invisible = true;
}
if !((text_rendering_mode == 3 && !include_invisible)
|| suppress_glyph_extraction
|| op.operands.is_empty())
@@ -1300,7 +1334,12 @@ pub(crate) fn extract_page_text_items(
let items = super::merge_text_items(items);
let items = super::merge_subscript_items(items);
Ok(((items, rects, lines), has_gid_fonts, coords_rotated))
Ok((
(items, rects, lines),
has_gid_fonts,
coords_rotated,
skipped_invisible,
))
}
/// Counts of text operators with horizontal vs rotated combined matrices.
@@ -1498,7 +1537,7 @@ mod tests {
let (doc, page_id) = simple_doc_with_content(content);
let font_cmaps = FontCMaps::from_doc(&doc);
let ((items, _, _), _, _) = extract_page_text_items(
let ((items, _, _), _, _, _) = extract_page_text_items(
&doc,
page_id,
1,
@@ -1736,7 +1775,7 @@ BT /F1 12 Tf 0 1 -1 0 240 100 Tm (WORLD) Tj ET
&mut FontStyleCache::new(),
)
.unwrap();
let ((items, rects, lines), _has_gid, _coords_rotated) = result;
let ((items, rects, lines), _has_gid, _coords_rotated, _skipped_invisible) = result;
assert!(items.is_empty());
assert!(rects.is_empty());
assert!(lines.is_empty());
@@ -1817,7 +1856,7 @@ BT 30 700 Tm <41> Tj ET";
doc.trailer.set("Root", Object::Reference(catalog_id));
let font_cmaps = FontCMaps::from_doc(&doc);
let ((items, _, _), _, _) = extract_page_text_items(
let ((items, _, _), _, _, _) = extract_page_text_items(
&doc,
page_id,
1,
src/extractor/mod.rs
+14 -11
View File
@@ -283,17 +283,20 @@ fn extract_positioned_text_impl(
include_invisible,
&mut style_cache,
);
let ((mut items, mut rects, mut lines), has_gid_fonts, coords_rotated) = match page_result {
Ok(extraction) => extraction,
Err(error) if required_pages.is_some_and(|required| !required.contains(page_num)) => {
debug!(
"page {}: skipping context-only extraction error: {}",
page_num, error
);
continue;
}
Err(error) => return Err(error),
};
let ((mut items, mut rects, mut lines), has_gid_fonts, coords_rotated, _skipped_invisible) =
match page_result {
Ok(extraction) => extraction,
Err(error)
if required_pages.is_some_and(|required| !required.contains(page_num)) =>
{
debug!(
"page {}: skipping context-only extraction error: {}",
page_num, error
);
continue;
}
Err(error) => return Err(error),
};
// Clip to the visible page box: single-page extracts and imposed
// spreads keep neighboring pages' content in the stream, positioned
// outside the CropBox. Extracting it interleaves invisible text into
src/lib.rs
+87 -23
View File
@@ -757,6 +757,23 @@ pub struct PageRegionResult {
pub regions: Vec<RegionText>,
}
/// Minimum alphanumeric mass an invisible (Tr 3) text layer must carry for
/// the OCR-layer fallback in [`extract_text_in_regions_mem`] to adopt it. A
/// real OCR layer carries far more; a stray watermark or artifact does not.
const OCR_LAYER_MIN_ALNUM: usize = 40;
/// Alphanumeric mass of extracted items, ignoring raster placeholders.
/// `[Image: ...]` items (ItemType::Image) are synthesized for image
/// XObjects — they mark that pixels exist, not that text was read, so they
/// must not count as coverage.
fn non_placeholder_alnum(items: &[TextItem]) -> usize {
items
.iter()
.filter(|it| !matches!(it.item_type, types::ItemType::Image))
.map(|it| it.text.chars().filter(|c| c.is_alphanumeric()).count())
.sum()
}
/// Extract text within bounding-box regions from a PDF in memory.
///
/// This is designed for hybrid OCR pipelines: a layout model detects regions
@@ -810,7 +827,7 @@ pub fn extract_text_in_regions_mem(
page_heights.insert(*page_num, height);
// Extract text items for this page
let ((mut items, _rects, _lines), has_gid, coords_rotated) =
let ((mut items, _rects, _lines), mut has_gid, mut coords_rotated, skipped_invisible) =
extractor::content_stream::extract_page_text_items(
&doc,
page_id,
@@ -819,6 +836,51 @@ pub fn extract_text_in_regions_mem(
false,
&mut style_cache,
)?;
// OCR-layer fallback: scanned pages often carry their text as an
// invisible (Tr 3) layer behind the page raster. The visible-only
// pass sees nothing there but `[Image: ...]` placeholders, so every
// region on the page reports needs_ocr even though the exact text is
// embedded in the PDF — and this extractor then disagrees with the
// markdown path, which already retries Mixed PDFs with the invisible
// layer included. Retry page-scoped, and only when (a) the first
// pass actually SKIPPED invisible text — blank pages and image-only
// scans without an OCR layer must not pay a second content-stream
// parse (review catch) — and (b) the page has NO visible text item
// at all (punctuation counts, whitespace-only artifacts don't): an
// invisible OCR layer transcribes the raster, so any visible glyph
// has an invisible twin there and adoption would duplicate it
// (review catches — strict gate, no fuzzy dedupe). Adopt the retry
// only when it contributes real, non-garbage text.
let has_visible_text = items.iter().any(|it| {
!matches!(it.item_type, types::ItemType::Image) && !it.text.trim().is_empty()
});
if skipped_invisible && !has_visible_text {
if let Ok(((inv_items, _inv_rects, _inv_lines), inv_gid, inv_rotated, _)) =
extractor::content_stream::extract_page_text_items(
&doc,
page_id,
*page_num,
&font_cmaps,
true,
&mut style_cache,
)
{
let inv_alnum = non_placeholder_alnum(&inv_items);
// Judge the WHOLE recovered layer, not a prefix — a broken
// OCR layer can hide its garbage past any fixed sample size
// (review catch).
let sample: String = inv_items
.iter()
.filter(|it| !matches!(it.item_type, types::ItemType::Image))
.map(|it| it.text.as_str())
.collect();
if inv_alnum >= OCR_LAYER_MIN_ALNUM && !is_garbage_text(&sample) {
items = inv_items;
has_gid = inv_gid;
coords_rotated = inv_rotated;
}
}
}
let threshold = text_utils::fix_letterspaced_items(&mut items);
if threshold > 0.10 {
page_thresholds.insert(*page_num, threshold);
@@ -975,7 +1037,7 @@ pub fn extract_tables_in_regions_mem(
let height = get_page_height(&doc, page_id).unwrap_or(792.0);
page_heights.insert(*page_num, height);
let ((mut items, rects, lines), has_gid, coords_rotated) =
let ((mut items, rects, lines), has_gid, coords_rotated, _skipped_invisible) =
extractor::content_stream::extract_page_text_items(
&doc,
page_id,
@@ -1286,7 +1348,7 @@ pub fn detect_vector_grid_in_region_mem(
let needed_pages = HashSet::from([page_1idx]);
let font_cmaps = FontCMaps::from_doc_pages_fast(&doc, Some(&needed_pages));
let page_h = get_page_height(&doc, page_id).unwrap_or(792.0);
let ((mut items, rects, lines), _has_gid, coords_rotated) =
let ((mut items, rects, lines), _has_gid, coords_rotated, _skipped_invisible) =
extractor::content_stream::extract_page_text_items(
&doc,
page_id,
@@ -1480,15 +1542,16 @@ mod vector_grid_tests {
let &page_id = pages.get(&1).unwrap();
let needed: HashSet<u32> = HashSet::from([1]);
let cmaps = FontCMaps::from_doc_pages_fast(&doc, Some(&needed));
let ((items, rects, _lines), _has_gid, _rotated) = extract_page_text_items(
&doc,
page_id,
1,
&cmaps,
false,
&mut crate::extractor::FontStyleCache::new(),
)
.unwrap();
let ((items, rects, _lines), _has_gid, _rotated, _skipped_invisible) =
extract_page_text_items(
&doc,
page_id,
1,
&cmaps,
false,
&mut crate::extractor::FontStyleCache::new(),
)
.unwrap();
let (rect_tables, _) = detect_tables_from_rects(&items, &rects, 1);
assert_eq!(rect_tables.len(), 1, "expected one rect-detected table");
@@ -1522,15 +1585,16 @@ mod vector_grid_tests {
let &page_id = pages.get(&page_num).unwrap();
let needed: HashSet<u32> = HashSet::from([page_num]);
let cmaps = FontCMaps::from_doc_pages_fast(&doc, Some(&needed));
let ((items, rects, _lines), _has_gid, _rotated) = extract_page_text_items(
&doc,
page_id,
page_num,
&cmaps,
false,
&mut crate::extractor::FontStyleCache::new(),
)
.unwrap();
let ((items, rects, _lines), _has_gid, _rotated, _skipped_invisible) =
extract_page_text_items(
&doc,
page_id,
page_num,
&cmaps,
false,
&mut crate::extractor::FontStyleCache::new(),
)
.unwrap();
let (rect_tables, _) = detect_tables_from_rects(&items, &rects, page_num);
rect_tables
@@ -2258,7 +2322,7 @@ pub fn extract_tables_with_structure_cells_mem(
let height = get_page_height(&doc, page_id).unwrap_or(792.0);
page_heights.insert(*page_num, height);
let ((mut items, _rects, _lines), _has_gid, coords_rotated) =
let ((mut items, _rects, _lines), _has_gid, coords_rotated, _skipped_invisible) =
extractor::content_stream::extract_page_text_items(
&doc,
page_id,
@@ -3060,7 +3124,7 @@ fn detect_tsr_quality_issue(
let mut needed: HashSet<u32> = HashSet::new();
needed.insert(page_1idx);
let font_cmaps = FontCMaps::from_doc_pages_fast(&doc, Some(&needed));
let ((mut items, _rects, _lines), _has_gid, coords_rotated) =
let ((mut items, _rects, _lines), _has_gid, coords_rotated, _skipped_invisible) =
extractor::content_stream::extract_page_text_items(
&doc,
page_id,
tests/integration_tests.rs
+276
View File
@@ -1654,6 +1654,282 @@ fn test_extract_regions_mem_basic_text_pdf() {
assert_eq!(regions[0].page, 0);
}
/// Build a synthetic "scanned page" PDF: a full-page image XObject with a
/// text layer drawn in the given render mode (3 = invisible OCR overlay,
/// 0 = normal visible fill). `visible_extra` optionally adds a normally
/// rendered line so double-layer behavior can be tested; `layer_lines`
/// overrides the layer content (default: three pangram lines);
/// `quote_ops` shows every layer line via the `'` operator instead of Tj
/// (both are standard show-text encodings for OCR layers).
fn make_pdf_with_custom_text_layer(
text_render_mode: i32,
visible_extra: Option<&str>,
layer_lines: Option<&[&str]>,
quote_ops: bool,
) -> Vec<u8> {
let mut pdf = b"%PDF-1.4\n".to_vec();
let mut offsets = vec![0usize];
fn add_object(pdf: &mut Vec<u8>, offsets: &mut Vec<usize>, id: usize, body: &str) {
offsets.push(pdf.len());
pdf.extend_from_slice(format!("{id} 0 obj\n").as_bytes());
pdf.extend_from_slice(body.as_bytes());
pdf.extend_from_slice(b"\nendobj\n");
}
fn add_stream_object(
pdf: &mut Vec<u8>,
offsets: &mut Vec<usize>,
id: usize,
dict: &str,
stream_bytes: &[u8],
) {
offsets.push(pdf.len());
pdf.extend_from_slice(format!("{id} 0 obj\n").as_bytes());
pdf.extend_from_slice(
format!("<< {} /Length {} >>\nstream\n", dict, stream_bytes.len()).as_bytes(),
);
pdf.extend_from_slice(stream_bytes);
pdf.extend_from_slice(b"\nendstream\nendobj\n");
}
add_object(
&mut pdf,
&mut offsets,
1,
"<< /Type /Catalog /Pages 2 0 R >>",
);
add_object(
&mut pdf,
&mut offsets,
2,
"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
);
add_object(
&mut pdf,
&mut offsets,
3,
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] \
/Resources << /Font << /F1 5 0 R >> /XObject << /Im0 6 0 R >> >> \
/Contents 4 0 R >>",
);
// Full-page raster, then the text layer in the requested render mode —
// several lines so the OCR-layer gate's alnum floor (40) is well cleared.
let mut content = String::from("q 612 0 0 792 0 0 cm /Im0 Do Q\n");
let default_layer = [
"The quick brown fox jumps over the lazy dog",
"Pack my box with five dozen liquor jugs tonight",
"Sphinx of black quartz judge my vow carefully",
];
let layer: &[&str] = layer_lines.unwrap_or(&default_layer);
if quote_ops {
// Every line shown via `'` (move-to-next-line + show) — nothing on
// this layer goes through Tj, pinning the `'` suppression path.
content.push_str(&format!(
"BT /F1 12 Tf {text_render_mode} Tr 16 TL 72 716 Td "
));
for line in layer {
content.push_str(&format!("({line}) ' "));
}
} else {
content.push_str(&format!("BT /F1 12 Tf {text_render_mode} Tr 72 700 Td "));
for (i, line) in layer.iter().enumerate() {
if i > 0 {
content.push_str("0 -16 Td ");
}
content.push_str(&format!("({line}) Tj "));
}
}
content.push_str("ET\n");
if let Some(extra) = visible_extra {
content.push_str(&format!("BT /F1 12 Tf 0 Tr 72 500 Td ({extra}) Tj ET\n"));
}
add_stream_object(&mut pdf, &mut offsets, 4, "", content.as_bytes());
add_object(
&mut pdf,
&mut offsets,
5,
"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
);
let image_pixel = [128u8];
add_stream_object(
&mut pdf,
&mut offsets,
6,
"/Type /XObject /Subtype /Image /Width 1 /Height 1 \
/ColorSpace /DeviceGray /BitsPerComponent 8",
&image_pixel,
);
let xref_start = pdf.len();
pdf.extend_from_slice(format!("xref\n0 {}\n", offsets.len()).as_bytes());
pdf.extend_from_slice(b"0000000000 65535 f \n");
for offset in offsets.iter().skip(1) {
pdf.extend_from_slice(format!("{offset:010} 00000 n \n").as_bytes());
}
pdf.extend_from_slice(
format!(
"trailer\n<< /Size {} /Root 1 0 R >>\nstartxref\n{}\n%%EOF",
offsets.len(),
xref_start
)
.as_bytes(),
);
pdf
}
fn make_pdf_with_text_layer(text_render_mode: i32, visible_extra: Option<&str>) -> Vec<u8> {
make_pdf_with_custom_text_layer(text_render_mode, visible_extra, None, false)
}
/// A scanned page whose only text is an invisible (Tr 3) OCR layer behind
/// the raster must serve that layer from the region extractor instead of
/// reporting the region as needs_ocr — the exact text is already in the PDF.
#[test]
fn test_extract_regions_mem_recovers_invisible_ocr_layer() {
let buf = make_pdf_with_text_layer(3, None);
let regions = extract_text_in_regions_mem(&buf, &full_page_regions(1)).unwrap();
assert_eq!(regions.len(), 1);
let region = &regions[0].regions[0];
assert!(
region.text.contains("quick brown fox"),
"invisible OCR layer should be served as region text, got: {:?}",
region.text
);
assert!(
!region.needs_ocr,
"recovered OCR layer must not fall back to GPU OCR"
);
}
/// ANY visible text on the page — even a single short line — must block the
/// invisible-layer adoption entirely: the invisible pass returns visible
/// items too, so adopting it alongside visible text would duplicate the
/// visible words. Strict zero-visible gate, no fuzzy dedupe.
#[test]
fn test_extract_regions_mem_visible_text_blocks_invisible_layer() {
let buf = make_pdf_with_text_layer(3, Some("Folio 142"));
let regions = extract_text_in_regions_mem(&buf, &full_page_regions(1)).unwrap();
let region = &regions[0].regions[0];
assert!(
region.text.contains("Folio 142"),
"visible text should be extracted, got: {:?}",
region.text
);
assert!(
!region.text.contains("quick brown fox"),
"invisible layer must not be adopted when any visible text exists, got: {:?}",
region.text
);
assert_eq!(
region.text.matches("Folio 142").count(),
1,
"visible text must appear exactly once, got: {:?}",
region.text
);
}
/// An invisible OCR layer shown entirely via the `'` show-text operator
/// (move-to-next-line + show) must also be recovered — the skipped_invisible
/// signal has to fire on every show-text path, not just Tj/TJ.
#[test]
fn test_extract_regions_mem_recovers_quote_operator_layer() {
let buf = make_pdf_with_custom_text_layer(3, None, None, true);
let regions = extract_text_in_regions_mem(&buf, &full_page_regions(1)).unwrap();
let region = &regions[0].regions[0];
assert!(
region.text.contains("quick brown fox"),
"'-operator OCR layer should be recovered, got: {:?}",
region.text
);
assert!(!region.needs_ocr);
}
/// An invisible layer below the 40-alnum floor (a stray watermark line)
/// must NOT be adopted — the region keeps its needs_ocr fallback.
#[test]
fn test_extract_regions_mem_tiny_invisible_layer_not_adopted() {
let buf = make_pdf_with_custom_text_layer(3, None, Some(&["Scanned by ACME"]), false);
let regions = extract_text_in_regions_mem(&buf, &full_page_regions(1)).unwrap();
let region = &regions[0].regions[0];
assert!(
!region.text.contains("Scanned by ACME"),
"below-floor invisible layer must not be adopted, got: {:?}",
region.text
);
// Only the raster placeholder remains — needs_ocr stays whatever main
// reports for placeholder-only regions (false today; downstream
// pipelines route placeholder-only text to OCR themselves, and this PR
// deliberately does not change that contract).
assert!(
region.text.trim().starts_with("[Image:"),
"region should hold only the raster placeholder, got: {:?}",
region.text
);
}
/// An invisible layer that clears the alnum floor but is mostly symbol
/// garbage (a broken OCR run) must be rejected by the garbage gate.
#[test]
fn test_extract_regions_mem_garbage_invisible_layer_not_adopted() {
// Each line: 5 alphanumerics among 15 symbol chars. Ten lines clear the
// 40-alnum floor (50 alnum) while staying well under the half-alnum
// ratio is_garbage_text requires.
let garbage_lines: Vec<&str> = vec!["a@@b%%c&&d==e~~"; 10];
let buf = make_pdf_with_custom_text_layer(3, None, Some(&garbage_lines), false);
let regions = extract_text_in_regions_mem(&buf, &full_page_regions(1)).unwrap();
let region = &regions[0].regions[0];
assert!(
!region.text.contains("a@@b"),
"garbage invisible layer must not be adopted, got: {:?}",
region.text
);
assert!(
region.text.trim().starts_with("[Image:"),
"region should hold only the raster placeholder, got: {:?}",
region.text
);
}
/// Punctuation-only visible text (zero alphanumerics) must ALSO block
/// adoption — the gate is item-presence, not alphanumeric mass. (Real-world
/// rationale: an invisible OCR layer transcribes the raster, so visible
/// glyphs typically have invisible twins there; this fixture's layers are
/// disjoint, so it pins the gate itself, not the duplication scenario.)
#[test]
fn test_extract_regions_mem_punctuation_visible_blocks_invisible_layer() {
let buf = make_pdf_with_text_layer(3, Some("... --- ..."));
let regions = extract_text_in_regions_mem(&buf, &full_page_regions(1)).unwrap();
let region = &regions[0].regions[0];
assert!(
!region.text.contains("quick brown fox"),
"invisible layer must not be adopted over punctuation-only visible text, got: {:?}",
region.text
);
assert_eq!(
region.text.matches("... --- ...").count(),
1,
"visible punctuation must be preserved exactly once, got: {:?}",
region.text
);
}
/// Regression guard: a normal visible-text page (render mode 0) is served
/// once and only once — if the fallback ever mis-fired here and merged a
/// second pass, the phrase would duplicate.
#[test]
fn test_extract_regions_mem_visible_layer_unchanged() {
let buf = make_pdf_with_text_layer(0, None);
let regions = extract_text_in_regions_mem(&buf, &full_page_regions(1)).unwrap();
let region = &regions[0].regions[0];
assert_eq!(
region.text.matches("quick brown fox").count(),
1,
"visible text must appear exactly once, got: {:?}",
region.text
);
assert!(!region.needs_ocr);
}
#[test]
fn test_extract_regions_mem_identity_h_needs_ocr() {
let buf = std::fs::read("tests/fixtures/shinagawa_identity_h.pdf").unwrap();