mu-ref/pdf-inspector
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: tagged PDF structure tree support (#4)

Browse Source 
* feat: tagged PDF structure tree support for semantic markdown generation

Parse /StructTreeRoot from tagged PDFs and use semantic roles (H1-H6, P,
LI, BlockQuote, Code, Caption) to improve markdown output. Structure tree
headings add to font-size heuristics without suppressing them. Coverage
threshold (≥50%) ensures only properly tagged PDFs activate this path.

Phase 1: Parse structure tree with role maps, MCID collection, flattening
Phase 2: Capture MCIDs from BMC/BDC operators, tag TextItems
Phase 3: Structure-aware markdown generation in convert loop

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: accumulate consecutive code lines into single fenced block

Per-line code fencing produced broken markdown for multi-line code
blocks (separate ``` open/close per line). Unify struct-tree Code
role and font-based monospace detection into a single is_code_line
check with in_code_block state for proper accumulation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: add tagged PDF fixture with Firecrawl docs content

Synthetic 7-page PDF with rich structure tree exercising H1, H2, H3,
P, Code, LI, Caption, TH, TD roles. Generated via fpdf2 script.
Integration test verifies struct tree parsing and code fence output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove python PDF generator script from repo

Keep the generated fixture PDF but don't track the generator script.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: handle malformed bare-name struct types in tagged PDFs

Some PDF generators (e.g. fpdf2) write /S Code instead of /S /Code
in structure elements. lopdf silently drops these objects since bare
tokens are invalid PDF syntax. Add a pre-processor that scans for
known bare struct type names and prepends / before loading.

Unifies path and memory loading through the same fix pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update lopdf dependency to main branch

The firecrawl/zlib-checksum-encrypted branch was merged and deleted.
Point to main which includes all previously merged fixes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: switch lopdf to upstream repo pinned at 845cd3d

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Abimael Martell
2026-03-17 20:50:24 -07:00
committed by GitHub
co-authored by Claude Opus 4.6
parent 76ea52680b
commit d9c2143c32
21 changed files with 1650 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/extractor/content_stream.rs
+58 -34
View File
@@ -124,10 +124,18 @@ pub(crate) fn extract_page_text_items(
let mut line_matrix = [1.0f32, 0.0, 0.0, 1.0, 0.0, 0.0];
let mut in_text_block = false;
// Marked content (ActualText) tracking
let mut marked_content_stack: Vec<Option<String>> = Vec::new();
// Marked content tracking: (ActualText, MCID) per nesting level
struct MarkedContentEntry {
actual_text: Option<String>,
mcid: Option<i64>,
}
let mut marked_content_stack: Vec<MarkedContentEntry> = Vec::new();
let mut suppress_glyph_extraction = false;
let mut actual_text_start_tm: Option<[f32; 6]> = None; // text matrix at BDC entry
/// Get the innermost MCID from the marked content stack.
fn current_mcid(stack: &[MarkedContentEntry]) -> Option<i64> {
stack.iter().rev().find_map(|e| e.mcid)
}
for op in &content.operations {
trace!("{} {:?}", op.operator, op.operands);
@@ -292,6 +300,7 @@ pub(crate) fn extract_page_text_items(
is_bold: is_bold_font(base_font),
is_italic: is_italic_font(base_font),
item_type: ItemType::Text,
mcid: current_mcid(&marked_content_stack),
});
}
}
@@ -440,6 +449,7 @@ pub(crate) fn extract_page_text_items(
is_bold: is_bold_font(base_font),
is_italic: is_italic_font(base_font),
item_type: ItemType::Text,
mcid: current_mcid(&marked_content_stack),
});
}
}
@@ -496,6 +506,7 @@ pub(crate) fn extract_page_text_items(
is_bold: is_bold_font(base_font),
is_italic: is_italic_font(base_font),
item_type: ItemType::Text,
mcid: current_mcid(&marked_content_stack),
});
}
}
@@ -531,11 +542,15 @@ pub(crate) fn extract_page_text_items(
}
"BMC" => {
// Begin Marked Content (no properties)
marked_content_stack.push(None);
marked_content_stack.push(MarkedContentEntry {
actual_text: None,
mcid: None,
});
}
"BDC" => {
// Begin Marked Content with properties — extract ActualText
// Begin Marked Content with properties — extract ActualText and MCID
let mut actual_text: Option<String> = None;
let mut mcid: Option<i64> = None;
if op.operands.len() >= 2 {
let dict = match &op.operands[1] {
Object::Dictionary(d) => Some(d.clone()),
@@ -549,47 +564,56 @@ pub(crate) fn extract_page_text_items(
_ => None,
};
}
if let Ok(Object::Integer(id)) = d.get(b"MCID") {
mcid = Some(*id);
}
}
}
if actual_text.is_some() {
suppress_glyph_extraction = true;
actual_text_start_tm = Some(text_matrix);
}
marked_content_stack.push(actual_text);
marked_content_stack.push(MarkedContentEntry { actual_text, mcid });
}
"EMC" => {
// End Marked Content — emit ActualText item with correct width
if let Some(Some(at)) = marked_content_stack.pop() {
// Compute width from text matrix advancement during BDC..EMC
if let Some(start_tm) = actual_text_start_tm.take() {
let combined = multiply_matrices(&start_tm, &ctm);
let rendered_size = effective_font_size(current_font_size, &combined);
let (x, y) = (combined[4], combined[5]);
// Width in device space from text matrix delta
let delta_ts = text_matrix[4] - start_tm[4];
let scale_x = start_tm[0] * ctm[0] + start_tm[1] * ctm[2];
let width = (delta_ts * scale_x).abs();
if !at.trim().is_empty() {
let base_font = font_base_names
.get(&current_font)
.map(|s| s.as_str())
.unwrap_or(&current_font);
items.push(TextItem {
text: expand_ligatures(&at),
x,
y,
width,
height: rendered_size,
font: current_font.clone(),
font_size: rendered_size,
page: page_num,
is_bold: is_bold_font(base_font),
is_italic: is_italic_font(base_font),
item_type: ItemType::Text,
});
if let Some(entry) = marked_content_stack.pop() {
if let Some(at) = entry.actual_text {
// Compute width from text matrix advancement during BDC..EMC
if let Some(start_tm) = actual_text_start_tm.take() {
let combined = multiply_matrices(&start_tm, &ctm);
let rendered_size = effective_font_size(current_font_size, &combined);
let (x, y) = (combined[4], combined[5]);
// Width in device space from text matrix delta
let delta_ts = text_matrix[4] - start_tm[4];
let scale_x = start_tm[0] * ctm[0] + start_tm[1] * ctm[2];
let width = (delta_ts * scale_x).abs();
if !at.trim().is_empty() {
let base_font = font_base_names
.get(&current_font)
.map(|s| s.as_str())
.unwrap_or(&current_font);
items.push(TextItem {
text: expand_ligatures(&at),
x,
y,
width,
height: rendered_size,
font: current_font.clone(),
font_size: rendered_size,
page: page_num,
is_bold: is_bold_font(base_font),
is_italic: is_italic_font(base_font),
item_type: ItemType::Text,
mcid: entry
.mcid
.or_else(|| current_mcid(&marked_content_stack)),
});
}
}
suppress_glyph_extraction =
marked_content_stack.iter().any(|e| e.actual_text.is_some());
}
suppress_glyph_extraction = marked_content_stack.iter().any(|a| a.is_some());
}
}
"re" => {
src/extractor/layout.rs
+1
View File
@@ -796,6 +796,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}
}
src/extractor/links.rs
+2
View File
@@ -79,6 +79,7 @@ pub fn extract_page_links(doc: &Document, page_id: ObjectId, page_num: u32) -> V
is_bold: false,
is_italic: false,
item_type: ItemType::Link(url),
mcid: None,
});
}
}
@@ -316,5 +317,6 @@ pub(crate) fn walk_form_fields(
is_bold: false,
is_italic: false,
item_type: ItemType::FormField,
mcid: None,
});
}
src/extractor/mod.rs
+23
View File
@@ -319,6 +319,7 @@ pub(crate) fn merge_text_items(items: Vec<TextItem>) -> Vec<TextItem> {
is_bold: first.is_bold,
is_italic: first.is_italic,
item_type: first.item_type.clone(),
mcid: first.mcid,
});
i = j;
@@ -359,6 +360,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
TextItem {
text: "World".into(),
@@ -372,6 +374,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
TextItem {
text: "Next line".into(),
@@ -385,6 +388,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
];
@@ -438,6 +442,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
TextItem {
text: "Prague".into(),
@@ -451,6 +456,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
TextItem {
text: "Rules".into(),
@@ -464,6 +470,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
];
@@ -488,6 +495,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
TextItem {
text: "A".into(),
@@ -501,6 +509,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
TextItem {
text: "V".into(),
@@ -514,6 +523,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
];
@@ -540,6 +550,7 @@ mod tests {
is_bold: true,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}
}
@@ -573,6 +584,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}
}
@@ -607,6 +619,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
TextItem {
text: "履行義務".into(),
@@ -620,6 +633,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
TextItem {
text: "を識別す".into(),
@@ -633,6 +647,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
];
@@ -654,6 +669,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}
}
@@ -765,6 +781,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
TextItem {
text: "\u{05D1}".into(), // bet at x=200 (rightmost)
@@ -778,6 +795,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
];
sort_line_items(&mut items);
@@ -801,6 +819,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
TextItem {
text: "World".into(),
@@ -814,6 +833,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
},
];
sort_line_items(&mut items);
@@ -853,6 +873,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}],
};
@@ -896,6 +917,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}],
};
@@ -939,6 +961,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}],
};
src/extractor/xobjects.rs
+2
View File
@@ -408,6 +408,7 @@ fn extract_form_xobject_text_inner(
is_bold: is_bold_font(base_font),
is_italic: is_italic_font(base_font),
item_type: ItemType::Text,
mcid: None,
});
}
}
@@ -549,6 +550,7 @@ fn extract_form_xobject_text_inner(
is_bold: is_bold_font(base_font),
is_italic: is_italic_font(base_font),
item_type: ItemType::Text,
mcid: None,
});
}
}
src/lib.rs
+28 -9
View File
@@ -28,6 +28,7 @@ pub mod extractor;
pub mod glyph_names;
pub mod markdown;
pub mod process_mode;
pub mod structure_tree;
pub mod tables;
pub mod text_utils;
pub mod tounicode;
@@ -273,20 +274,21 @@ pub fn process_pdf_mem_with_config(
/// are combined here, but lopdf loads the full doc in `load()` so we extract
/// page count from it directly to avoid the metadata-only round-trip.
fn load_document_from_path<P: AsRef<Path>>(path: P) -> Result<(Document, u32), PdfError> {
let doc = match Document::load(&path) {
Ok(d) => d,
Err(ref e) if is_encrypted_lopdf_error(e) => Document::load_with_password(&path, "")?,
Err(e) => return Err(e.into()),
};
let page_count = doc.get_pages().len() as u32;
Ok((doc, page_count))
let buffer = std::fs::read(&path)?;
load_document_from_mem(&buffer)
}
/// Load a PDF from a memory buffer.
fn load_document_from_mem(buffer: &[u8]) -> Result<(Document, u32), PdfError> {
let doc = match Document::load_mem(buffer) {
// Fix malformed struct element names before parsing. Some PDF generators
// write bare names (/S Code) instead of proper PDF names (/S /Code), which
// causes lopdf to silently drop the entire object.
let fixed = structure_tree::fix_bare_struct_names(buffer);
let buf = fixed.as_ref();
let doc = match Document::load_mem(buf) {
Ok(d) => d,
Err(ref e) if is_encrypted_lopdf_error(e) => Document::load_mem_with_password(buffer, "")?,
Err(ref e) if is_encrypted_lopdf_error(e) => Document::load_mem_with_password(buf, "")?,
Err(e) => return Err(e.into()),
};
let page_count = doc.get_pages().len() as u32;
@@ -350,6 +352,22 @@ fn process_document(
Some(extracted?)
};
// Parse structure tree for tagged PDFs (reuses the loaded document)
let struct_roles = structure_tree::StructTree::from_doc(&doc).and_then(|tree| {
let page_ids = doc.get_pages();
let roles = tree.mcid_to_roles(&page_ids);
if roles.is_empty() {
None
} else {
log::debug!(
"structure tree: {} pages with MCID roles, {} total MCIDs",
roles.len(),
tree.mcid_count()
);
Some(roles)
}
});
let (markdown, layout, has_encoding_issues) = match extracted {
Some(((items, rects, lines), page_thresholds)) => {
let layout = compute_layout_complexity(&items, &rects, &lines);
@@ -363,6 +381,7 @@ fn process_document(
&rects,
&lines,
&page_thresholds,
struct_roles.as_ref(),
))
};
src/markdown/convert.rs
+407 -28
View File
@@ -2,6 +2,7 @@
use std::collections::HashSet;
use crate::structure_tree::StructRole;
use crate::types::TextLine;
use super::analysis::{
@@ -13,6 +14,50 @@ use super::postprocess::clean_markdown;
use super::preprocess::{merge_drop_caps, merge_heading_lines};
use super::MarkdownOptions;
/// Resolve the dominant structure role for a text line by looking up its items' MCIDs.
///
/// Returns the first non-container role found (skipping Document/Part/Sect/Div/NonStruct/Span).
/// These wrapper roles don't carry useful semantic info for markdown generation.
fn resolve_line_struct_role(
line: &TextLine,
struct_roles: &std::collections::HashMap<u32, std::collections::HashMap<i64, StructRole>>,
) -> Option<StructRole> {
let page_roles = struct_roles.get(&line.page)?;
for item in &line.items {
if let Some(mcid) = item.mcid {
if let Some(role) = page_roles.get(&mcid) {
match role {
// Skip container/wrapper roles — not useful for line classification
StructRole::Document
| StructRole::Part
| StructRole::Art
| StructRole::Sect
| StructRole::Div
| StructRole::NonStruct
| StructRole::Span
| StructRole::Private => continue,
_ => return Some(role.clone()),
}
}
}
}
None
}
/// Map a StructRole heading variant to a markdown heading level (1–6).
fn struct_role_heading_level(role: &StructRole) -> Option<usize> {
match role {
StructRole::H => Some(1), // Generic heading → H1
StructRole::H1 => Some(1),
StructRole::H2 => Some(2),
StructRole::H3 => Some(3),
StructRole::H4 => Some(4),
StructRole::H5 => Some(5),
StructRole::H6 => Some(6),
_ => None,
}
}
/// Merge continuation tables that span across page breaks.
///
/// When consecutive pages each have exactly one table with the same number of columns
@@ -182,6 +227,9 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
page_tables: std::collections::HashMap<u32, Vec<(f32, String)>>,
page_images: std::collections::HashMap<u32, Vec<(f32, String)>>,
band_split_pages: &HashSet<u32>,
struct_roles: Option<
&std::collections::HashMap<u32, std::collections::HashMap<i64, StructRole>>,
>,
) -> String {
if lines.is_empty() && page_tables.is_empty() && page_images.is_empty() {
return String::new();
@@ -215,6 +263,7 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
let mut in_list = false;
let mut in_paragraph = false;
let mut last_list_x: Option<f32> = None;
let mut in_code_block = false;
let mut prev_had_dot_leaders = false;
let mut inserted_tables: HashSet<(u32, usize)> = HashSet::new();
let mut inserted_images: HashSet<(u32, usize)> = HashSet::new();
@@ -233,6 +282,10 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
if line.page != current_page {
// Flush current page's remaining tables and images
if current_page > 0 {
if in_code_block {
output.push_str("```\n");
in_code_block = false;
}
flush_page_tables_and_images(
current_page,
&page_tables,
@@ -352,7 +405,25 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
// Detect figure/table captions and source citations
// These should be on their own line followed by a paragraph break
if is_caption_line(plain_trimmed) {
let struct_role = struct_roles.and_then(|roles| resolve_line_struct_role(&line, roles));
// Determine if this line is code (struct-tree or font-based) for block accumulation
let is_code_line = struct_role
.as_ref()
.is_some_and(|r| matches!(r, StructRole::Code))
|| (options.detect_code && line.items.iter().any(|i| is_monospace_font(&i.font)));
// Close code block when transitioning to non-code
if in_code_block && !is_code_line {
output.push_str("```\n");
in_code_block = false;
}
if struct_role
.as_ref()
.is_some_and(|r| matches!(r, StructRole::Caption))
|| is_caption_line(plain_trimmed)
{
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
@@ -362,27 +433,48 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
continue;
}
// Detect headers by font size
// Note: Headers typically shouldn't have bold markers since they're already emphasized
// Skip very short text (drop caps/labels) and very long text (body paragraphs)
if options.detect_headers
// Detect headers: structure-tree headings win, then font-size heuristics.
// Structure roles ADD headings (e.g. same-size text tagged H2) but do NOT
// suppress headings that the font heuristic would detect (some tagged PDFs
// mark obvious headings as P or Span).
let struct_heading = struct_role.as_ref().and_then(struct_role_heading_level);
let heuristic_heading = if options.detect_headers
&& plain_trimmed.len() > 3
&& plain_trimmed.split_whitespace().count() <= 15
{
let line_font_size = line.items.first().map(|i| i.font_size).unwrap_or(base_size);
if let Some(header_level) =
detect_header_level(line_font_size, base_size, &heading_tiers)
{
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
}
let prefix = "#".repeat(header_level);
// Use plain text for headers to avoid redundant formatting
output.push_str(&format!("{} {}\n\n", prefix, plain_trimmed));
in_list = false;
continue;
detect_header_level(line_font_size, base_size, &heading_tiers)
} else {
None
};
if let Some(level) = struct_heading.or(heuristic_heading) {
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
}
let prefix = "#".repeat(level);
// Use plain text for headers to avoid redundant formatting
output.push_str(&format!("{} {}\n\n", prefix, plain_trimmed));
in_list = false;
continue;
}
// Structure-tree list item (LI only — LBody is a continuation, not a new item)
if struct_role
.as_ref()
.is_some_and(|r| matches!(r, StructRole::LI))
&& !is_list_item(plain_trimmed)
{
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
}
output.push_str(&format!("- {}", trimmed));
output.push('\n');
in_list = true;
last_list_x = line.items.first().map(|i| i.x);
continue;
}
// Detect list items
@@ -428,18 +520,32 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
}
}
// Detect code blocks by font
if options.detect_code {
let is_mono = line.items.iter().any(|i| is_monospace_font(&i.font));
if is_mono {
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
}
// Use plain text for code blocks
output.push_str(&format!("```\n{}\n```\n", plain_trimmed));
continue;
// Structure-tree block quote
if struct_role
.as_ref()
.is_some_and(|r| matches!(r, StructRole::BlockQuote))
{
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
}
output.push_str(&format!("> {}\n", trimmed));
continue;
}
// Code block accumulation (struct-tree Code role or monospace font)
if is_code_line {
if in_paragraph {
output.push_str("\n\n");
in_paragraph = false;
}
if !in_code_block {
output.push_str("```\n");
in_code_block = true;
}
output.push_str(plain_trimmed);
output.push('\n');
continue;
}
// Regular text - join lines within same paragraph with space
@@ -456,6 +562,11 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
prev_had_dot_leaders = cur_dot_leaders;
}
// Close any trailing code block
if in_code_block {
output.push_str("```\n");
}
// Flush current page and any remaining pages with tables/images
// (handles table-only pages after the last text line, and trailing image-only pages)
flush_page_tables_and_images(
@@ -680,3 +791,271 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
// Clean up and post-process
clean_markdown(output, &options)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::structure_tree::StructRole;
use crate::types::TextItem;
use std::collections::HashMap;
fn make_item(text: &str, page: u32, mcid: Option<i64>) -> TextItem {
TextItem {
text: text.to_string(),
x: 72.0,
y: 700.0,
width: 100.0,
height: 12.0,
font: "Helvetica".to_string(),
font_size: 12.0,
page,
is_bold: false,
is_italic: false,
item_type: crate::types::ItemType::Text,
mcid,
}
}
fn make_line(items: Vec<TextItem>) -> TextLine {
let y = items.first().map(|i| i.y).unwrap_or(0.0);
let page = items.first().map(|i| i.page).unwrap_or(1);
TextLine {
items,
y,
page,
adaptive_threshold: 0.10,
}
}
#[test]
fn test_struct_role_heading() {
let lines = vec![
make_line(vec![make_item("Introduction", 1, Some(0))]),
make_line(vec![{
let mut item = make_item("Body text here.", 1, Some(1));
item.y = 680.0;
item
}]),
];
let mut page_roles = HashMap::new();
page_roles.insert(0i64, StructRole::H1);
page_roles.insert(1i64, StructRole::P);
let mut roles = HashMap::new();
roles.insert(1u32, page_roles);
let md = to_markdown_from_lines_with_tables_and_images(
lines,
MarkdownOptions::default(),
HashMap::new(),
HashMap::new(),
&std::collections::HashSet::new(),
Some(&roles),
);
assert!(
md.contains("# Introduction"),
"Should have H1 heading: {md}"
);
assert!(
md.contains("Body text here."),
"Should have body text: {md}"
);
}
#[test]
fn test_struct_role_list_item() {
let lines = vec![make_line(vec![make_item("First item", 1, Some(0))])];
let mut page_roles = HashMap::new();
page_roles.insert(0i64, StructRole::LI);
let mut roles = HashMap::new();
roles.insert(1u32, page_roles);
let md = to_markdown_from_lines_with_tables_and_images(
lines,
MarkdownOptions::default(),
HashMap::new(),
HashMap::new(),
&std::collections::HashSet::new(),
Some(&roles),
);
assert!(
md.contains("- First item"),
"Should format as list item: {md}"
);
}
#[test]
fn test_struct_role_blockquote() {
let lines = vec![make_line(vec![make_item("Quoted text", 1, Some(0))])];
let mut page_roles = HashMap::new();
page_roles.insert(0i64, StructRole::BlockQuote);
let mut roles = HashMap::new();
roles.insert(1u32, page_roles);
let md = to_markdown_from_lines_with_tables_and_images(
lines,
MarkdownOptions::default(),
HashMap::new(),
HashMap::new(),
&std::collections::HashSet::new(),
Some(&roles),
);
assert!(
md.contains("> Quoted text"),
"Should format as blockquote: {md}"
);
}
#[test]
fn test_struct_role_heading_levels() {
let mcids = vec![
(StructRole::H1, "Title"),
(StructRole::H2, "Section"),
(StructRole::H3, "Subsection"),
];
let mut lines = Vec::new();
let mut page_roles = HashMap::new();
for (i, (role, text)) in mcids.iter().enumerate() {
let mut item = make_item(text, 1, Some(i as i64));
item.y = 700.0 - (i as f32 * 30.0);
lines.push(make_line(vec![item]));
page_roles.insert(i as i64, role.clone());
}
let mut roles = HashMap::new();
roles.insert(1u32, page_roles);
let md = to_markdown_from_lines_with_tables_and_images(
lines,
MarkdownOptions::default(),
HashMap::new(),
HashMap::new(),
&std::collections::HashSet::new(),
Some(&roles),
);
assert!(md.contains("# Title"), "H1 → #: {md}");
assert!(md.contains("## Section"), "H2 → ##: {md}");
assert!(md.contains("### Subsection"), "H3 → ###: {md}");
}
#[test]
fn test_no_struct_roles_falls_back_to_heuristics() {
let mut item = make_item("Big Title", 1, None);
item.font_size = 24.0;
item.height = 24.0;
let lines = vec![
make_line(vec![item]),
make_line(vec![{
let mut body = make_item("Normal body text.", 1, None);
body.y = 660.0;
body
}]),
];
let md = to_markdown_from_lines_with_tables_and_images(
lines,
MarkdownOptions::default(),
HashMap::new(),
HashMap::new(),
&std::collections::HashSet::new(),
None,
);
assert!(
md.contains("# Big Title"),
"Font heuristic should detect heading: {md}"
);
}
#[test]
fn test_resolve_line_struct_role_skips_containers() {
let mut page_roles = HashMap::new();
page_roles.insert(0i64, StructRole::Div);
page_roles.insert(1i64, StructRole::H2);
let mut roles = HashMap::new();
roles.insert(1u32, page_roles);
let line = make_line(vec![
make_item("Part ", 1, Some(0)),
make_item("Title", 1, Some(1)),
]);
let role = resolve_line_struct_role(&line, &roles);
assert_eq!(role, Some(StructRole::H2));
}
#[test]
fn test_struct_role_code() {
let lines = vec![make_line(vec![make_item("fn main() {}", 1, Some(0))])];
let mut page_roles = HashMap::new();
page_roles.insert(0i64, StructRole::Code);
let mut roles = HashMap::new();
roles.insert(1u32, page_roles);
let md = to_markdown_from_lines_with_tables_and_images(
lines,
MarkdownOptions::default(),
HashMap::new(),
HashMap::new(),
&std::collections::HashSet::new(),
Some(&roles),
);
assert!(
md.contains("```\nfn main() {}\n```"),
"Should format as code block: {md}"
);
}
#[test]
fn test_struct_role_code_multiline_accumulation() {
let mut line1 = make_item("fn main() {", 1, Some(0));
line1.y = 700.0;
let mut line2 = make_item(" println!(\"hello\");", 1, Some(1));
line2.y = 688.0;
let mut line3 = make_item("}", 1, Some(2));
line3.y = 676.0;
let lines = vec![
make_line(vec![line1]),
make_line(vec![line2]),
make_line(vec![line3]),
];
let mut page_roles = HashMap::new();
page_roles.insert(0i64, StructRole::Code);
page_roles.insert(1i64, StructRole::Code);
page_roles.insert(2i64, StructRole::Code);
let mut roles = HashMap::new();
roles.insert(1u32, page_roles);
let md = to_markdown_from_lines_with_tables_and_images(
lines,
MarkdownOptions::default(),
HashMap::new(),
HashMap::new(),
&std::collections::HashSet::new(),
Some(&roles),
);
// Should produce a single fenced block, not three separate ones
assert!(
md.contains("```\nfn main() {\nprintln!(\"hello\");\n}\n```"),
"Should accumulate consecutive code lines into one block: {md}"
);
// Should NOT have adjacent fences
assert!(
!md.contains("```\n```"),
"Should not have adjacent close/open fences: {md}"
);
}
}
src/markdown/mod.rs
+38 -1
View File
@@ -452,7 +452,7 @@ pub fn to_markdown_from_items_with_rects(
options: MarkdownOptions,
rects: &[crate::types::PdfRect],
) -> String {
to_markdown_from_items_with_rects_and_lines(items, options, rects, &[], &HashMap::new())
to_markdown_from_items_with_rects_and_lines(items, options, rects, &[], &HashMap::new(), None)
}
/// Convert positioned text items to markdown, using rectangles and line segments for table detection.
@@ -465,6 +465,7 @@ pub(crate) fn to_markdown_from_items_with_rects_and_lines(
rects: &[crate::types::PdfRect],
pdf_lines: &[crate::types::PdfLine],
page_thresholds: &HashMap<u32, f32>,
struct_roles: Option<&HashMap<u32, HashMap<i64, crate::structure_tree::StructRole>>>,
) -> String {
use crate::tables::{
detect_tables, detect_tables_from_lines, detect_tables_from_rects, table_to_markdown,
@@ -757,6 +758,40 @@ pub(crate) fn to_markdown_from_items_with_rects_and_lines(
}
}
// Check structure tree coverage on ALL text items (before table filtering)
// to decide whether to use structure-aware markdown generation.
let struct_roles_coverage_ok = struct_roles.is_some_and(|roles| {
let total = text_items.len();
if total == 0 {
return false;
}
let tagged = text_items
.iter()
.filter(|item| {
item.mcid
.and_then(|mcid| {
roles
.get(&item.page)
.and_then(|page_roles| page_roles.get(&mcid))
})
.is_some()
})
.count();
let coverage = tagged as f32 / total as f32;
log::debug!(
"structure tree coverage: {}/{} items ({:.0}%)",
tagged,
total,
coverage * 100.0
);
coverage >= 0.5
});
let effective_struct_roles = if struct_roles_coverage_ok {
struct_roles
} else {
None
};
// Filter out table items and process the rest
let non_table_items: Vec<TextItem> = text_items
.into_iter()
@@ -841,6 +876,7 @@ pub(crate) fn to_markdown_from_items_with_rects_and_lines(
page_tables,
page_images,
&band_split_page_set,
effective_struct_roles,
)
}
@@ -926,6 +962,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: crate::types::ItemType::Text,
mcid: None,
}
}
src/structure_tree.rs
+1035
View File
File diff suppressed because it is too large Load Diff
src/tables/detect_heuristic.rs
+1
View File
@@ -105,6 +105,7 @@ pub(crate) fn merge_adjacent_items(items: &[TextItem]) -> (Vec<TextItem>, Vec<Ve
is_bold: first_item.is_bold,
is_italic: first_item.is_italic,
item_type: first_item.item_type.clone(),
mcid: first_item.mcid,
});
index_map.push(indices);
src/tables/detect_lines.rs
+1
View File
@@ -248,6 +248,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}
}
src/tables/detect_rects.rs
+1
View File
@@ -1450,6 +1450,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}
}
src/tables/financial.rs
+1
View File
@@ -109,6 +109,7 @@ pub(crate) fn try_split_financial_item(item: &TextItem) -> Option<Vec<TextItem>>
is_bold: item.is_bold,
is_italic: item.is_italic,
item_type: item.item_type.clone(),
mcid: item.mcid,
});
}
Some(sub_items)
src/tables/grid.rs
+3
View File
@@ -381,6 +381,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}
}
@@ -727,6 +728,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
page: 1,
},
));
@@ -762,6 +764,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
page: 1,
},
));
src/tables/mod.rs
+4
View File
@@ -231,6 +231,7 @@ fn split_merged_numbers(item: &TextItem, col_boundaries: &[f32]) -> Vec<TextItem
is_bold: item.is_bold,
is_italic: item.is_italic,
item_type: item.item_type.clone(),
mcid: item.mcid,
});
}
@@ -250,6 +251,7 @@ fn split_merged_numbers(item: &TextItem, col_boundaries: &[f32]) -> Vec<TextItem
is_bold: item.is_bold,
is_italic: item.is_italic,
item_type: item.item_type.clone(),
mcid: item.mcid,
});
}
@@ -296,6 +298,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}
}
@@ -312,6 +315,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}
}
src/text_utils.rs
+3
View File
@@ -879,6 +879,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}
}
@@ -997,6 +998,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
});
// Alternate between letter-gap and word-gap to create bimodal distribution
x += w + if wi % 3 == 2 { word_gap } else { letter_gap };
@@ -1072,6 +1074,7 @@ mod tests {
is_bold: false,
is_italic: false,
item_type: ItemType::Text,
mcid: None,
}
}
src/types.rs
+3
View File
@@ -117,6 +117,9 @@ pub struct TextItem {
pub is_italic: bool,
/// Type of item (text, image, link)
pub item_type: ItemType,
/// Marked Content ID from the content stream's BDC/BMC operator.
/// Used to link this item to the PDF structure tree for tagged PDFs.
pub mcid: Option<i64>,
}
/// A line of text (grouped text items)