fix(extractor): content stream comment parsing + CJK mojibake detection (#18)

* fix(extractor): strip PDF comments that break lopdf content stream parsing

Some PDF generators (notably PD4ML used by school districts) embed
comments (% to end of line) in content streams. lopdf's Content::decode
parser fails to parse operators that follow comments, silently dropping
ET (end text) and Q (restore graphics state) operators. This caused
entire pages to produce 0 text items despite having valid text.

Fix: pre-process content streams to strip comments before parsing.
Comments inside string literals (parentheses) and hex strings are
preserved. The comment is replaced with a space to maintain token
separation.

Impact: fixes 13+ school district PDFs and similar PD4ML-generated
documents that were producing near-empty output (454 → 31,955 chars
for a 22-page school improvement plan).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(detect): flag sparse-extraction pages as needing OCR

When a TEXT-BASED PDF produces <50 chars/page average with <500 total
chars, flag all pages as needing OCR. This catches PDFs where the
extractable text is minimal (form templates, image-heavy layouts)
and the bulk of content requires OCR to access.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(detect): improve CID mojibake detection for Japanese/CJK PDFs

Extend is_cid_garbage to detect CID-as-Latin-1 mojibake: when ≥40%
of characters are high Latin-1 (U+00A0-00FF) and <33% are ASCII
letters, the text is likely CID values misinterpreted as Latin-1
characters (common in Japanese/CJK PDFs with broken ToUnicode CMaps).

Also add sparse-extraction OCR flagging: TEXT-BASED PDFs with
<50 chars/page and <500 total chars get all pages flagged for OCR.

Impact: Softbank Japanese PDFs now produce empty output with
pages_needing_ocr=all instead of mojibake garbage. Korean PDFs
with valid extraction (nexo-price-en) remain unaffected.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(detect): sparse extraction check only when markdown is generated

The sparse-extraction OCR check was triggering in Analyze mode where
markdown is not generated (md_len=0), causing false OCR flags on
every PDF processed via detect-pdf --analyze.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

src/extractor/content_stream.rs

@@ -20,6 +20,60 @@ use super::fonts::{
 use super::xobjects::{extract_form_xobject_text, get_page_xobjects, XObjectType};
 use super::{get_number, multiply_matrices};
 
+/// Strip PDF comments (% to end of line) from content stream bytes.
+///
+/// Some PDF generators (e.g. PD4ML) embed comments in content streams that
+/// confuse lopdf's `Content::decode` parser.  Comments inside string literals
+/// (parentheses) are NOT stripped — only top-level comments.
+fn strip_pdf_comments(data: &[u8]) -> Vec<u8> {
+    // Quick check: if no '%' present, return as-is (common case)
+    if !data.contains(&b'%') {
+        return data.to_vec();
+    }
+
+    let mut result = Vec::with_capacity(data.len());
+    let mut i = 0;
+    let mut in_string = 0i32; // parenthesis nesting depth
+    let mut in_hex_string = false;
+
+    while i < data.len() {
+        let b = data[i];
+        match b {
+            b'(' if !in_hex_string => {
+                in_string += 1;
+                result.push(b);
+            }
+            b')' if !in_hex_string && in_string > 0 => {
+                in_string -= 1;
+                result.push(b);
+            }
+            b'<' if in_string == 0 && !in_hex_string => {
+                in_hex_string = true;
+                result.push(b);
+            }
+            b'>' if in_hex_string => {
+                in_hex_string = false;
+                result.push(b);
+            }
+            b'%' if in_string == 0 && !in_hex_string => {
+                // Skip until end of line
+                while i < data.len() && data[i] != b'\n' && data[i] != b'\r' {
+                    i += 1;
+                }
+                // Replace comment with a space to preserve token separation
+                result.push(b' ');
+                continue; // Don't increment i again
+            }
+            _ => {
+                result.push(b);
+            }
+        }
+        i += 1;
+    }
+
+    result
+}
+
 /// Returns `(page_extraction, has_gid_fonts)` where `has_gid_fonts` indicates
 /// the page uses fonts with unresolvable gid-encoded glyphs.
 pub(crate) fn extract_page_text_items(
@@ -113,6 +167,11 @@ pub(crate) fn extract_page_text_items(
         .get_page_content(page_id)
         .map_err(|e| PdfError::Parse(e.to_string()))?;
 
+    // Strip PDF comments (% to end of line) from the content stream.
+    // Some PDF generators (e.g. PD4ML) embed comments that confuse lopdf's
+    // Content::decode parser, causing it to skip operators like ET and Q.
+    let content_data = strip_pdf_comments(&content_data);
+
     let content = Content::decode(&content_data).map_err(|e| PdfError::Parse(e.to_string()))?;
 
     const MAX_OPERATIONS: usize = 1_000_000;
@@ -1174,4 +1233,36 @@ mod tests {
         assert!(rects.is_empty());
         assert!(lines.is_empty());
     }
+
+    #[test]
+    fn test_strip_pdf_comments() {
+        // Basic comment stripping
+        let input = b"BT\n% comment\nTj\nET\n";
+        let output = strip_pdf_comments(input);
+        assert_eq!(output, b"BT\n \nTj\nET\n");
+
+        // No comments = unchanged
+        let input = b"BT\nTj\nET\n";
+        let output = strip_pdf_comments(input);
+        assert_eq!(output, input.to_vec());
+
+        // Don't strip inside string literals
+        let input = b"(text with % not a comment)\n% real comment\n";
+        let output = strip_pdf_comments(input);
+        assert_eq!(output, b"(text with % not a comment)\n \n");
+
+        // Don't strip inside hex strings
+        let input = b"<0033% not a comment>\n% real comment\n";
+        let output = strip_pdf_comments(input);
+        assert_eq!(output, b"<0033% not a comment>\n \n");
+
+        // PD4ML style: comment between Tj and ET
+        let input = b"<0033> Tj\n\t% Mission Statement\n\tET\n";
+        let output = strip_pdf_comments(input);
+        let output_str = String::from_utf8_lossy(&output);
+        assert!(
+            output_str.contains("ET"),
+            "ET should be preserved after comment stripping"
+        );
+    }
 }