feat: tagged PDF structure tree support (#4)

* feat: tagged PDF structure tree support for semantic markdown generation

Parse /StructTreeRoot from tagged PDFs and use semantic roles (H1-H6, P,
LI, BlockQuote, Code, Caption) to improve markdown output. Structure tree
headings add to font-size heuristics without suppressing them. Coverage
threshold (≥50%) ensures only properly tagged PDFs activate this path.

Phase 1: Parse structure tree with role maps, MCID collection, flattening
Phase 2: Capture MCIDs from BMC/BDC operators, tag TextItems
Phase 3: Structure-aware markdown generation in convert loop

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: accumulate consecutive code lines into single fenced block

Per-line code fencing produced broken markdown for multi-line code
blocks (separate ``` open/close per line). Unify struct-tree Code
role and font-based monospace detection into a single is_code_line
check with in_code_block state for proper accumulation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: add tagged PDF fixture with Firecrawl docs content

Synthetic 7-page PDF with rich structure tree exercising H1, H2, H3,
P, Code, LI, Caption, TH, TD roles. Generated via fpdf2 script.
Integration test verifies struct tree parsing and code fence output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove python PDF generator script from repo

Keep the generated fixture PDF but don't track the generator script.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: handle malformed bare-name struct types in tagged PDFs

Some PDF generators (e.g. fpdf2) write /S Code instead of /S /Code
in structure elements. lopdf silently drops these objects since bare
tokens are invalid PDF syntax. Add a pre-processor that scans for
known bare struct type names and prepends / before loading.

Unifies path and memory loading through the same fix pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update lopdf dependency to main branch

The firecrawl/zlib-checksum-encrypted branch was merged and deleted.
Point to main which includes all previously merged fixes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: switch lopdf to upstream repo pinned at 845cd3d

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

tests/integration_tests.rs

@@ -23,6 +23,7 @@ fn make_text_item(text: &str, x: f32, y: f32, font_size: f32, page: u32) -> Text
         is_bold: false,
         is_italic: false,
         item_type: ItemType::Text,
+        mcid: None,
     }
 }
 
@@ -47,6 +48,7 @@ fn make_text_item_with_font(
         is_bold: is_bold_font(font),
         is_italic: is_italic_font(font),
         item_type: ItemType::Text,
+        mcid: None,
     }
 }
 
@@ -999,3 +1001,39 @@ startxref
         );
     }
 }
+
+#[test]
+fn test_firecrawl_tagged_pdf_struct_tree() {
+    use lopdf::Document;
+    use pdf_inspector::structure_tree::{StructRole, StructTree};
+
+    let doc = Document::load("tests/fixtures/firecrawl_docs_tagged.pdf").unwrap();
+    let tree = StructTree::from_doc(&doc).expect("Should have a structure tree");
+
+    // Verify structure tree contains expected roles
+    let page_ids = doc.get_pages();
+    let roles = tree.mcid_to_roles(&page_ids);
+    assert!(!roles.is_empty(), "Should have MCID roles across pages");
+
+    let flat = tree.flatten();
+    let has_code = flat.iter().any(|e| matches!(e.role, StructRole::Code));
+    let has_h1 = flat.iter().any(|e| matches!(e.role, StructRole::H1));
+    let has_li = flat.iter().any(|e| matches!(e.role, StructRole::LI));
+    let has_caption = flat.iter().any(|e| matches!(e.role, StructRole::Caption));
+    assert!(has_code, "Should have Code elements");
+    assert!(has_h1, "Should have H1 elements");
+    assert!(has_li, "Should have LI elements");
+    assert!(has_caption, "Should have Caption elements");
+
+    // Full conversion: code fences should be generated from Code struct elements
+    let buf = std::fs::read("tests/fixtures/firecrawl_docs_tagged.pdf").unwrap();
+    let result = pdf_inspector::process_pdf_mem(&buf).unwrap();
+    let md = result.markdown.unwrap();
+    let fence_count = md.matches("```").count();
+    assert!(
+        fence_count > 0,
+        "Should produce code fences from tagged Code elements"
+    );
+    // Fences come in open/close pairs
+    assert_eq!(fence_count % 2, 0, "Code fences should be balanced");
+}