mu-ref/pdf-inspector
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(underline): rescue snug-owned underlines from the table-ruling filters (#143)

Browse Source 
* fix(underline): rescue snug-owned underlines from the table-ruling filters

Documents that underline many full-width lines (dense CJK business docs,
legal redlines, 10-K section links) produce span-similar rules at 3+
y-levels — exactly what the repeated-ruling filter treats as table
rulings, so every semantic underline on such pages was discarded. Three
changes fix detection without re-marking real tables:

1. Snug-owner rescue: a rule survives the repeated-ruling filter when
   the union of touching text runs on its baseline row owns it (rule
   contained within the union's span +0.75em, runs cover >=60% of the
   rule, no column-sized gaps between runs). Table row separators fail
   ownership: they overshoot their cells' text or match gapped items.
   Same-row segmented rules (column-header separators) always stay
   discarded, and a rule enclosed by a drawn cell-sized box (rect-grid
   tables) is never rescued.

2. Vertical window widened 0.35em -> 0.72em below the baseline: CJK
   layouts draw underlines under the full em box, measured at ~0.67em.

3. Prose-table guard in the positions-path suppressor: a detected
   'table' whose cells hold flowing prose (>=30% of cells over 100
   chars) is a detection artifact of boxed callouts + stacked rules,
   not a real table — suppressing there erased every underline on the
   page.

Also fixes cluster_x_positions fabricating phantom table columns from
style-split continuation runs (touching items, gap <2pt, now feed one
column start) — the fix that keeps rect-grid table shapes stable while
underlined links inside cells are correctly marked.

Snapshot updates are underline gains on regulation/form fixtures and one
empty spacer-column change in a subscripted header.

Corpus (508-doc public bench sweep): text output byte-identical on all
docs; underlined items +224/-0; strikeout now fires on redline docs.
Item-level GT coverage: is_underline 86->151/405, is_strikeout 0->10/44.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB

* fix(underline): fraction-bar guard + subscript merge across underline marks

Found by a 202-doc real-world corpus diff (pdf-evals) that exercises the
full markdown pipeline, which the bench-corpus item sweep does not:

1. Math fraction bars and lattice grid lines are underline geometry —
   short horizontal rules under digits. Guard: a narrow rule (<=60pt)
   with bar-sized text hanging just below it (denominator) never marks.
   The below-text width bound matters: tightly-leaded REAL underlines
   have a full-width next line below, which must not trip the guard.

2. merge_subscript_items refused to merge when the parent was underlined
   but the tiny digit was not (the drawn rule easily misses the digit's
   own overlap window) — losing the merge broke subscript tokens inside
   table cells (b+2 no longer became b₂). Strikeout boundaries still
   block the merge in both directions; only parent-underlined/digit-bare
   merges, absorbing with the parent's flags.

Corpus after refinement: underlined items +220/-2 (the 2 are fraction
bars the old code wrongly marked), GT rule-text coverage 149/405
underline + 10/44 strikeout, text output identical on all 508 bench
docs and word-count-identical on the 202 pdf-evals docs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB

* fix(underline): address review — grid-evidence veto, strikeout-safe fraction guard, bounded gaps

- Cell-box veto now requires GRID EVIDENCE (a vertically abutting
  neighbor rect with x-overlap) instead of a height window: multiline
  table cells taller than the old 90pt ceiling veto again, and isolated
  filled callout panels (which legitimately contain underlines) no
  longer veto at all.
- The fraction guard gates only UNDERLINE marking; rule_strikes_item
  still evaluates, so short strikeouts near lower text survive.
- Fraction hug distance tightened to 0.3em so a short last-line at
  normal leading is not mistaken for a denominator.
- Continuation-run suppression bounds the negative gap (-4pt): text
  overhanging from an adjacent cell keeps its own column start.

Corpus after review fixes: underlined items +222/-2, GT coverage
150/405 underline + 10/44 strikeout, bench text output identical on
all 508 docs, pdf-evals word loss bounded at equation-reflow noise.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB

* chore: appease clippy (redundant closure)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3U6BKYCS73DVA83odAfYB

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Abimael Martell
2026-07-13 08:47:45 -07:00
committed by GitHub
co-authored by Claude Fable 5
parent cb3906e7b8
commit 39c31a8404
6 changed files with 367 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/extractor/mod.rs
+45 -7
View File
@@ -252,17 +252,47 @@ fn suppress_table_underlines(
}
let mut table_item_indices: HashSet<usize> = HashSet::new();
// A detected "table" that swallows nearly every text item on the page
// is a detection artifact (prose pages with boxed callouts or stacked
// underline rules read as one giant grid), not a real table — letting
// it through here erased every legitimate underline on the page
// (text_dense__underline: rect detection claimed 52/52 items). Real
// ruled tables share the page with headings, captions, and body text.
let plausible = |table: &crate::tables::Table| {
// Content sanity gate: prose pages with boxed callouts and stacked
// underline rules can detect as a structurally rich "table" that
// swallows every item on the page (text_dense__underline: a 4x8
// grid claiming 52/52 items, one "cell" holding 806 chars of body
// text) — suppressing there erased every legitimate underline on
// the page. Real data-table cells are short values; a cell with
// hundreds of characters means the grid captured flowing prose.
let lens: Vec<usize> = table
.cells
.iter()
.flatten()
.filter(|cell| !cell.trim().is_empty())
.map(|cell| cell.chars().count())
.collect();
if lens.is_empty() {
return false;
}
let long = lens.iter().filter(|&&n| n > 100).count();
(long as f32) < (lens.len() as f32) * 0.3
};
if !rects.is_empty() {
let (rect_tables, _) = crate::tables::detect_tables_from_rects(items, rects, page);
for table in rect_tables {
table_item_indices.extend(table.item_indices);
for table in rect_tables.iter().filter(|t| plausible(t)) {
table_item_indices.extend(table.item_indices.iter().copied());
}
}
if !lines.is_empty() {
for table in crate::tables::detect_tables_from_lines(items, lines, page) {
table_item_indices.extend(table.item_indices);
for table in crate::tables::detect_tables_from_lines(items, lines, page)
.iter()
.filter(|t| plausible(t))
{
table_item_indices.extend(table.item_indices.iter().copied());
}
}
@@ -873,9 +903,17 @@ pub(crate) fn merge_subscript_items(items: Vec<TextItem>) -> Vec<TextItem> {
.chars()
.last()
.is_some_and(|c| c.is_alphabetic());
let same_marks = parent.is_underline == item.is_underline
&& parent.is_strikeout == item.is_strikeout;
if parent.font_size >= sub_threshold && ends_with_letter && same_marks {
// Strikeout boundaries block the merge (a struck word
// must not extend its strike over a live footnote digit,
// and a struck digit must not lose its own mark). An
// underlined parent with an unmarked digit DOES merge:
// the drawn rule easily misses the tiny digit's overlap
// window, and refusing costs the whole subscript token
// ("b"+"2" staying split). Visually the rule spans both.
let marks_ok = parent.is_strikeout == item.is_strikeout
&& (parent.is_underline == item.is_underline
|| (parent.is_underline && !item.is_underline));
if parent.font_size >= sub_threshold && ends_with_letter && marks_ok {
let parent_right = parent.x + parent.width;
let gap = item.x - parent_right;
// Subscripts must be tightly adjacent (within ~1pt)
src/extractor/underline.rs
+288 -7
View File
@@ -115,7 +115,13 @@ fn rules_from_graphics(rects: &[PdfRect], lines: &[UnderlineLine], page: u32) ->
rules
}
fn discard_repeated_ruling_rules(rules: Vec<Rule>) -> Vec<Rule> {
fn discard_repeated_ruling_rules(
rules: Vec<Rule>,
items: &[TextItem],
rects: &[PdfRect],
lines: &[UnderlineLine],
page: u32,
) -> Vec<Rule> {
if rules.len() < MIN_REPEATED_RULE_LEVELS {
return rules;
}
@@ -123,12 +129,142 @@ fn discard_repeated_ruling_rules(rules: Vec<Rule>) -> Vec<Rule> {
rules
.iter()
.filter(|rule| {
!is_repeated_ruling_rule(rule, &rules) && !is_segmented_row_ruling_rule(rule, &rules)
// A rule snugly owned by one text line is an underline even when
// span-similar rules repeat down the page — documents that
// underline many full-width lines (dense CJK business docs) look
// exactly like table rulings to the repetition check, which used
// to discard every one of them. Table rulings fail snugness:
// row separators extend past their cells' text (or have no text
// on the baseline above), and multi-column matches are still
// culled by the tabular filter afterwards.
// Same-row segmented rules (column-header separators) are
// always rulings — each segment snugly owns its column label,
// so snugness must not override that check.
!is_segmented_row_ruling_rule(rule, &rules)
&& ((has_snug_text_owner(rule, items)
&& !has_flanking_verticals(rule, rects, lines, page))
|| !is_repeated_ruling_rule(rule, &rules))
})
.cloned()
.collect()
}
/// True when a single text item both matches the rule vertically (baseline
/// window) and horizontally contains it: the rule may not extend past the
/// item's span by more than ~0.75em on either side. Underlines are drawn to
/// the width of the text they decorate; table/form rulings span cells or
/// full table width and overshoot any single item.
/// A rule flanked by vertical strokes at its ends is a table/box border
/// row edge, not an underline — underlined text lines have no vertical
/// rules rising from their ends. Checked against raw stroked lines: a
/// near-vertical segment whose x sits at either end of the rule and whose
/// y-range covers the rule's row.
fn has_flanking_verticals(
rule: &Rule,
rects: &[PdfRect],
lines: &[UnderlineLine],
page: u32,
) -> bool {
// A drawn rect that CONTAINS the rule vetoes rescue only with GRID
// EVIDENCE: another drawn rect abutting it vertically (cell rows tile).
// Height alone can't separate a table cell from a decorative callout
// panel — genuine underlines live inside isolated filled panels, and
// multiline table cells can be arbitrarily tall.
let norm = |r: &PdfRect| {
let (x_lo, x_hi) = if r.width >= 0.0 {
(r.x, r.x + r.width)
} else {
(r.x + r.width, r.x)
};
let (y_lo, y_hi) = if r.height >= 0.0 {
(r.y, r.y + r.height)
} else {
(r.y + r.height, r.y)
};
(x_lo, x_hi, y_lo, y_hi)
};
let page_rects: Vec<(f32, f32, f32, f32)> = rects
.iter()
.filter(|r| r.page == page && r.height.abs() > 6.0)
.map(norm)
.collect();
let rect_flank = page_rects.iter().any(|&(x_lo, x_hi, y_lo, y_hi)| {
let contains = x_lo <= rule.x1 + 2.0
&& x_hi >= rule.x2 - 2.0
&& y_lo <= rule.y + 2.0
&& y_hi >= rule.y - 2.0;
if !contains {
return false;
}
// Grid evidence: a vertically abutting neighbor box with x-overlap.
page_rects.iter().any(|&(nx_lo, nx_hi, ny_lo, ny_hi)| {
let x_overlap = nx_hi.min(x_hi) - nx_lo.max(x_lo);
if x_overlap <= 10.0 {
return false;
}
(ny_lo - y_hi).abs() <= 3.0 || (y_lo - ny_hi).abs() <= 3.0
})
});
if rect_flank {
return true;
}
lines.iter().any(|l| {
if l.page != page || (l.x1 - l.x2).abs() > 2.0 {
return false;
}
let x = (l.x1 + l.x2) / 2.0;
let near_end = (x - rule.x1).abs() <= 6.0 || (x - rule.x2).abs() <= 6.0;
if !near_end {
return false;
}
let (y_lo, y_hi) = if l.y1 <= l.y2 {
(l.y1, l.y2)
} else {
(l.y2, l.y1)
};
y_lo <= rule.y + 2.0 && y_hi >= rule.y - 2.0
})
}
fn has_snug_text_owner(rule: &Rule, items: &[TextItem]) -> bool {
// Underlines are drawn to the width of the text they decorate, but the
// text may be split into several runs (CJK lines mix scripts and font
// switches) — so ownership is judged against the UNION of the runs on
// the rule's baseline row. Table/form rulings overshoot their row's
// text (row separators span cell padding and empty columns), so they
// fail either containment or coverage.
let matched: Vec<&TextItem> = items
.iter()
.filter(|item| is_underline_candidate(item) && rule_matches_item(rule, item))
.collect();
if matched.is_empty() {
return false;
}
let x1 = matched.iter().map(|i| i.x).fold(f32::INFINITY, f32::min);
let x2 = matched
.iter()
.map(|i| i.x + i.width)
.fold(f32::NEG_INFINITY, f32::max);
let max_fs = matched.iter().map(|i| i.font_size).fold(0.0, f32::max);
let pad = (max_fs * 0.75).max(4.0);
if rule.x1 < x1 - pad || rule.x2 > x2 + pad {
return false;
}
let covered: f32 = matched.iter().map(|i| i.width).sum();
if covered < rule.width() * 0.6 {
return false;
}
// A table row also unions to the rule's span — but its cells sit apart.
// An underlined text line is contiguous runs with word-sized gaps; any
// column-sized hole between matched runs means this is a row ruling.
let mut sorted = matched;
sorted.sort_by(|a, b| a.x.total_cmp(&b.x));
sorted.windows(2).all(|pair| {
let gap = pair[1].x - (pair[0].x + pair[0].width);
gap <= (max_fs * 2.0).max(12.0)
})
}
fn is_repeated_ruling_rule(rule: &Rule, rules: &[Rule]) -> bool {
let mut y_levels: Vec<f32> = rules
.iter()
@@ -215,9 +351,11 @@ fn is_underline_candidate(item: &TextItem) -> bool {
fn rule_matches_item(rule: &Rule, item: &TextItem) -> bool {
// Vertical window: underlines sit at or slightly below the baseline.
// Fonts draw them at roughly 5-15% of the em below; allow up to 35%
// (min 3pt) below and 1pt above for rounding.
let below = (item.font_size * 0.35).max(3.0);
// Latin fonts draw them at roughly 5-15% of the em below; CJK layouts
// put them under the full em box, measured up to ~0.67em below the
// baseline (text_dense__underline). Allow 0.72em (min 3pt) below and
// 1pt above for rounding.
let below = (item.font_size * 0.72).max(3.0);
let y_min = item.y - below;
let y_max = item.y + 1.0;
if rule.y < y_min || rule.y > y_max {
@@ -260,12 +398,48 @@ pub(crate) fn mark_underlined_items(
lines: &[UnderlineLine],
page: u32,
) {
let rules = discard_repeated_ruling_rules(rules_from_graphics(rects, lines, page));
let rules = discard_repeated_ruling_rules(
rules_from_graphics(rects, lines, page),
items,
rects,
lines,
page,
);
if rules.is_empty() {
return;
}
let tabular_rules = tabular_row_separator_rule_indices(&rules, items);
// Math fraction bars are short horizontal lines with the numerator just
// above AND the denominator just below — underline geometry from above,
// but no underline has text hanging directly beneath it at fraction
// distance. Only narrow rules qualify: real underlines under short
// labels have their next text line a full line-pitch away.
let fraction_rules: HashSet<usize> = rules
.iter()
.enumerate()
.filter(|(_, rule)| {
rule.width() <= 60.0
&& items.iter().any(|item| {
if !is_underline_candidate(item) {
return false;
}
// A denominator HUGS the bar (fraction typesetting
// leaves ~0.1-0.2em) and is bar-sized. Both bounds
// matter: a short last-line of a paragraph at normal
// leading sits further below, and a full next text
// line is far wider than the rule.
let dy = rule.y - (item.y + item.height);
let overlap = rule.x2.min(item.x + item.width) - rule.x1.max(item.x);
dy > 0.0
&& dy <= item.font_size * 0.3
&& overlap > rule.width() * 0.5
&& item.width <= rule.width() * 1.5
})
})
.map(|(i, _)| i)
.collect();
for item in items.iter_mut() {
if !is_underline_candidate(item) {
continue;
@@ -275,7 +449,10 @@ pub(crate) fn mark_underlined_items(
if tabular_rules.contains(&rule_idx) {
continue;
}
if rule_matches_item(rule, item) {
// The fraction guard only gates UNDERLINE marking — a rule that
// reads as a fraction bar from below can still legitimately
// strike through a line above it.
if !fraction_rules.contains(&rule_idx) && rule_matches_item(rule, item) {
item.is_underline = true;
}
if rule_strikes_item(rule, item) {
@@ -323,6 +500,16 @@ mod tests {
}
}
fn cell_rect(x: f32, y: f32, width: f32, height: f32) -> PdfRect {
PdfRect {
x,
y,
width,
height,
page: 1,
}
}
fn thin_rect(x: f32, y: f32, width: f32) -> PdfRect {
PdfRect {
x,
@@ -543,6 +730,100 @@ mod tests {
assert!(items.iter().all(|item| !item.is_underline));
}
#[test]
fn repeated_snug_underlines_survive_ruling_filter() {
// Dense docs underline many full-width lines: span-similar rules at
// 3+ y-levels used to be discarded wholesale as table rulings.
// Each rule here snugly matches one text line, so all must mark.
let mut items = vec![
item("first underlined line of text", 50.0, 700.0, 300.0, 11.0),
item("second underlined line here", 50.0, 650.0, 300.0, 11.0),
item("third underlined line as well", 50.0, 600.0, 300.0, 11.0),
];
let lines = vec![
hline(50.0, 350.0, 697.0),
hline(50.0, 350.0, 647.0),
hline(50.0, 350.0, 597.0),
];
mark_underlined_items(&mut items, &[], &lines, 1);
assert!(items.iter().all(|item| item.is_underline));
}
#[test]
fn snug_rescue_spans_split_runs_on_one_line() {
// A single underlined line is often split into several runs (script
// or font switches). The union of touching runs owns the rule.
let mut items = vec![
item("run one", 50.0, 700.0, 100.0, 11.0),
item("run two", 150.5, 700.0, 100.0, 11.0),
item("run three", 251.0, 700.0, 99.0, 11.0),
item("other a", 50.0, 650.0, 300.0, 11.0),
item("other b", 50.0, 600.0, 300.0, 11.0),
];
let lines = vec![
hline(50.0, 350.0, 697.0),
hline(50.0, 350.0, 647.0),
hline(50.0, 350.0, 597.0),
];
mark_underlined_items(&mut items, &[], &lines, 1);
assert!(items[0].is_underline && items[1].is_underline && items[2].is_underline);
}
#[test]
fn snug_rescue_denied_for_row_with_cell_gaps() {
// A full-width rule whose baseline row is several items separated by
// column-sized gaps is a table row separator, not an underline —
// even when span-similar rules repeat down the page.
let mut items = vec![
item("cell a", 50.0, 700.0, 60.0, 11.0),
item("cell b", 190.0, 700.0, 60.0, 11.0),
item("cell c", 330.0, 700.0, 70.0, 11.0),
item("cell d", 50.0, 650.0, 60.0, 11.0),
item("cell e", 190.0, 650.0, 60.0, 11.0),
item("cell f", 330.0, 650.0, 70.0, 11.0),
];
let lines = vec![
hline(50.0, 400.0, 697.0),
hline(50.0, 400.0, 647.0),
hline(50.0, 400.0, 597.0),
];
mark_underlined_items(&mut items, &[], &lines, 1);
assert!(items.iter().all(|item| !item.is_underline));
}
#[test]
fn snug_rescue_denied_inside_cell_box() {
// A rule snugly under one text line but enclosed by a drawn cell
// box that TILES with vertical neighbors (grid evidence) is a row
// ruling of a rect-grid table. Isolated boxes (callout panels) do
// not veto — see repeated_snug_underlines_survive_ruling_filter.
let mut items = vec![
item("one wide cell row", 50.0, 700.0, 300.0, 11.0),
item("second wide cell", 50.0, 650.0, 300.0, 11.0),
item("third wide cell", 50.0, 600.0, 300.0, 11.0),
];
let lines = vec![
hline(50.0, 350.0, 697.0),
hline(50.0, 350.0, 647.0),
hline(50.0, 350.0, 597.0),
];
let boxes = vec![
cell_rect(45.0, 690.0, 320.0, 50.0),
cell_rect(45.0, 640.0, 320.0, 50.0),
cell_rect(45.0, 590.0, 320.0, 50.0),
];
mark_underlined_items(&mut items, &boxes, &lines, 1);
assert!(items.iter().all(|item| !item.is_underline));
}
#[test]
fn same_row_spaced_rule_segments_do_not_mark_column_labels() {
let mut items = vec![