fix(markdown): reject headings that end on a relational verb

A heading candidate ending in 'equals', 'denotes', 'implies' and the
like is the first half of a sentence, not a title. This shows up when a
block dissolves and strands its lead-in ahead of the formula it
introduced — opendataloader 01030000000144 produced

    ## Note that the exact error equals
    M - Q(h) = e - 2.7525... = -0.0342....

Deliberately a very short list. Broader variants were tried and
measured, then rejected:

- Function words (of/and/for/the): a heading that WRAPS across lines
  ends on exactly those. Destroyed real IRS Publication 17 headings —
  'Casualty and' -> 'Casualty and Theft Losses', 'Rule 10. You Must Be
  at' -> '... At Least Age 25'. 52 documents affected, -619 headings.
- Copulas and auxiliaries (is/are/be/have): same failure. 'Rule 15.
  Your AGI Must Be', 'What Medical Expenses Are' and 'When Can a Roth
  IRA Be' are real wrapped headings, while 'the tax burden should be'
  is a genuine fragment. The trailing word cannot separate them; that
  needs the next line's context, which this text-only predicate lacks.

The verbs kept never end a heading in any register, so they are safe
without context. Standalone the guard is a no-op on both benchmarks
(0 documents on opendataloader, 4 on pdf-evals with no net heading
change) — its value is as a companion to the table filter in this PR,
which is what strands these lead-ins.

Combined effect on opendataloader (200 docs, vs a control build of
main), where the table filter alone regressed:

                 table filter    + this guard
    overall        -0.0003          +0.0003
    mhs            -0.0019          +0.0003
    doc ...144     -0.063           +0.053
    doc ...144 mhs -0.203           +0.028
This commit is contained in:
Abimael Martell
2026-08-06 12:20:53 -07:00
parent 85a5150062
commit a5a6e8fbea
+99
View File
@@ -244,9 +244,108 @@ pub(crate) fn is_heading_fragment(text: &str) -> bool {
if t.ends_with(':') && t.split_whitespace().any(is_equation_number) {
return true;
}
// Dangling clause: a heading never ends on a word that demands a
// continuation. Combined with no terminal punctuation, that is the first
// half of a sentence some upstream split left behind — "Note that the
// exact error equals" stranded ahead of its formula when a phantom table
// dissolved. Real headings end on a content word ("Introduction", "Error
// Analysis") or on punctuation.
//
// Deliberately a closed list of verbs: a genuine title ending in
// "equals" or "is" is vanishingly rare, whereas guessing from
// part-of-speech would misfire on terse headings.
if !t.ends_with(['.', '!', '?', ':', ';', ')', ']']) {
if let Some(last) = t.split_whitespace().next_back() {
let word: String = last
.trim_matches(|c: char| !c.is_alphanumeric())
.to_lowercase();
// Verbs and copulas only. Articles, prepositions and
// conjunctions were tried and had to be removed: a heading that
// WRAPS across two lines ends on exactly those words, so
// suppressing them destroyed real headings ("Casualty and" ->
// "Casualty and Theft Losses", "Rule 10. You Must Be at" ->
// "... At Least Age 25" in IRS Publication 17). A wrapped
// heading never breaks after its verb, so this list is safe.
// Technical relational verbs ONLY. Copulas and auxiliaries
// (is/are/be/have) were tried and had to be dropped: a heading
// that wraps across lines ends on exactly those words, and
// suppressing them destroyed real headings in IRS Publication 17
// ("Rule 15. Your AGI Must Be" -> "... Less Than ...",
// "What Medical Expenses Are" -> "... Deductible?",
// "When Can a Roth IRA Be" -> "... Opened?"). The same trailing
// word appears in genuine body fragments ("the tax burden should
// be"), so the tail alone cannot separate the two — that needs
// the following line's context, which this text-only predicate
// does not have.
//
// The verbs below never end a heading in any register, so they
// are safe without context.
const DANGLING_TAIL: &[&str] = &[
"equals",
"denotes",
"implies",
"satisfies",
"yields",
"becomes",
"signifies",
];
if DANGLING_TAIL.contains(&word.as_str()) {
return true;
}
}
}
false
}
#[cfg(test)]
mod fragment_heading_tests {
use super::is_heading_fragment;
#[test]
fn dangling_tail_marks_stranded_clause() {
// opendataloader 01030000000144: left behind when a phantom table
// dissolved, ahead of its formula on the next line.
assert!(is_heading_fragment("Note that the exact error equals"));
assert!(is_heading_fragment("The remainder term satisfies"));
}
#[test]
fn real_headings_survive() {
assert!(!is_heading_fragment("Introduction"));
assert!(!is_heading_fragment("Error Analysis"));
assert!(!is_heading_fragment("Materials and Methods"));
assert!(!is_heading_fragment("Results"));
assert!(!is_heading_fragment("3.2 Richardson Extrapolation"));
assert!(!is_heading_fragment("Discussion and Conclusions"));
// Terminal punctuation means the clause is complete.
assert!(!is_heading_fragment("What is a Derivative?"));
assert!(!is_heading_fragment("Procedure:"));
assert!(!is_heading_fragment("Note that this is important."));
}
#[test]
fn wrapped_headings_are_not_fragments() {
// A heading that wraps across lines ends on a function word. These
// are real headings from IRS Publication 17 and must survive.
assert!(!is_heading_fragment("Casualty and"));
assert!(!is_heading_fragment("Rule 10. You Must Be at"));
assert!(!is_heading_fragment("Higher Standard Deduction for"));
assert!(!is_heading_fragment("Qualifying Child of"));
assert!(!is_heading_fragment("When Can I Withdraw or"));
// Copulas and auxiliaries also end real wrapped headings.
assert!(!is_heading_fragment("Rule 15. Your AGI Must Be"));
assert!(!is_heading_fragment("What Medical Expenses Are"));
assert!(!is_heading_fragment("Rule 13. You Must Have"));
assert!(!is_heading_fragment("When Can a Roth IRA Be"));
}
#[test]
fn dangling_check_is_case_insensitive() {
assert!(is_heading_fragment("THE REMAINDER EQUALS"));
}
}
/// Compute the Y-gap threshold for paragraph break detection.
///
/// Instead of using a fixed multiple of base_size (which fails for double-spaced