feat(text): adaptive join threshold for Canva-style letter-spaced PDFs

Canva-generated PDFs render text character-by-character with CSS-style
letter-spacing (~0.5-0.9× font_size). The hardcoded 0.10 threshold
caused every character to get a space inserted ("K a r i b i b").

Detect Canva pages via fix_letterspaced_items (≥50% items match "a b c"
pattern), compute an IQR-based threshold (median × 1.55) on the gap
distribution BEFORE space removal, then propagate per-page thresholds
through PageThresholds → group_into_lines_with_thresholds → TextLine
→ should_join_items.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Abimael Martell
2026-03-12 17:54:49 -07:00
co-authored by Claude Opus 4.6
parent 80a3b81ff9
commit 152de8b56d
7 changed files with 537 additions and 21 deletions
+5
View File
@@ -138,6 +138,7 @@ fn test_text_line_text_method() {
items,
y: 700.0,
page: 1,
adaptive_threshold: 0.10,
};
assert_eq!(line.text(), "Hello World");
}
@@ -149,6 +150,7 @@ fn test_text_line_single_item() {
items,
y: 700.0,
page: 1,
adaptive_threshold: 0.10,
};
assert_eq!(line.text(), "Single");
}
@@ -159,6 +161,7 @@ fn test_text_line_empty() {
items: vec![],
y: 700.0,
page: 1,
adaptive_threshold: 0.10,
};
assert_eq!(line.text(), "");
}
@@ -473,11 +476,13 @@ fn test_markdown_from_lines_basic() {
items: vec![make_text_item("First", 100.0, 700.0, 12.0, 1)],
y: 700.0,
page: 1,
adaptive_threshold: 0.10,
},
TextLine {
items: vec![make_text_item("Second", 100.0, 680.0, 12.0, 1)],
y: 680.0,
page: 1,
adaptive_threshold: 0.10,
},
];
let md = to_markdown_from_lines(lines, MarkdownOptions::default());