Compare commits

..
Author SHA1 Message Date
Abimael MartellandClaude Fable 5 573bc3cc90 review: block table roles from heading promotion too
Add Table/TR/TH/TD/THead/TBody/TFoot to is_non_heading_content. When
table reconstruction falls back and cells reach the line loop as plain
text, a short isolated cell (a TH column header in particular) could be
promoted to a heading. Defensive: no change across either regression
corpus, so pure hardening for the fallback path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 22:22:12 -07:00
Abimael MartellandClaude Fable 5 8951d90958 review: extend non-heading role gate; centralize on StructRole method
Move the non-heading-role check to StructRole::is_non_heading_content
and extend it to the content roles the inline allowlist missed: Quote,
Index, Note, Reference, BibEntry, Formula, Form (in addition to the
existing list/quote/caption/toc/code roles).

Figure is deliberately excluded: cover and banner pages routinely tag
the document title inside a Figure next to a seal/logo, and that title
is a real heading — including Figure demoted the LA County protocol
cover title from headings to bold. Verified against the reference.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 22:14:24 -07:00
Abimael MartellandClaude Fable 5 2349474432 fix(markdown): keep isolated headings on sparse pages; gate tagged roles
The isolated-line density guard wiped every isolated line on a page
where they exceeded 25% of lines. On sparse pages (covers, ToC pages
with a lone "CONTENTS" title, section-divider pages) a single heading
is trivially >25%, so the guard erased exactly the line it exists to
find. Require the page to have >=10 lines before the guard runs — the
25% ratio only signals a multi-column misfire on a dense page.

That let more isolated lines through, exposing that the visual heading
heuristic could promote lines already tagged with a non-heading struct
role (list item, blockquote, code, caption, ToC) or set in a monospace
font. Gate the heuristic on those in both converter paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 22:02:07 -07:00
Abimael MartellandClaude Fable 5 3a389f079e fix(markdown): ToC-page suppression, wrapped bold headings, math fragments (#131)
* fix(markdown): ToC-page suppression, wrapped bold headings, math fragments

Three heading-classification improvements:

- After emitting a "Contents"/"Table of Contents" heading, suppress
  heading promotion for the rest of that page: ToC entries are section
  titles that look exactly like headings ("1. Overview of OCR Pack")
  and whole contents pages came out as stacks of ##.
- merge_heading_lines only merged font-size-tier and struct-tree
  headings, so bold-at-body-size headings that wrap emitted two
  separate ## lines. Merge a fully-bold line into the previous
  fully-bold line when it reads as a wrap continuation (starts
  lowercase, tiny Y gap, no terminal punctuation on the previous line).
- Reject display-math fragments from the bold/rarity heading heuristic:
  equations ending in an equation number ("S = kB ln W, (2)") and
  lead-ins referencing one ("Rearranging Equation (8) gives:"). A bare
  trailing colon is deliberately NOT a signal — real headings often end
  with colons ("Procedure:").

The p1244 snapshot change is the bold-merge working as intended:
stacked form labels "**Subtotals** **from pages**" now read
"**Subtotals from pages**".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: guard tier path, protect prev headings in merge, narrow (N) rule

All three review findings applied, calibrated against the corpora:

- is_heading_fragment now gates the font-size-tier path too, not just
  the rarity heuristic.
- The bold wrap-merge requires the previous line to be tier-less as
  suggested; corpus diff confirmed the old behavior was absorbing a
  wrapped list-item fragment into a real heading.
- The bare "(N)" suffix rule suppressed real headings ("Nicaea (325)",
  appendix numbering). It now requires math evidence: an operator
  (=, <=, <<, ...) in the line or ,/: immediately before the number.
  Page-of-total running headers ("PM 2 (10)") get an explicit rule
  since the old blanket suffix check had been catching them only by
  accident.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 21:40:18 -07:00
3 changed files with 145 additions and 3 deletions
+50 -2
View File
@@ -141,8 +141,11 @@ fn find_isolated_lines(lines: &[TextLine], base_size: f32, para_threshold: f32)
}
}
for (&page, &(total, isolated)) in &page_line_counts {
if total > 0 && isolated as f32 / total as f32 > 0.25 {
// Too many isolated lines on this page — remove them all
// The ratio only means something on pages dense enough for a
// multi-column misfire; on sparse pages (covers, ToC pages with a
// lone title) one isolated line is 25%+ of the page and exactly the
// line isolation exists to find.
if total >= 10 && isolated as f32 / total as f32 > 0.25 {
set.retain(|&i| lines[i].page != page);
}
}
@@ -700,7 +703,15 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
_ => false,
};
// Lines explicitly tagged with a non-heading content role must never
// be promoted by the visual heuristic — a tagged list item, quote, or
// code line can look exactly like a heading (short, isolated).
let non_heading_role = struct_role
.as_ref()
.is_some_and(StructRole::is_non_heading_content);
let heuristic_heading = if options.detect_headers
&& !non_heading_role
&& !is_code_line
&& !looks_like_list_continuation
&& plain_trimmed.len() > 3
&& plain_trimmed.split_whitespace().count() <= 15
@@ -1053,6 +1064,7 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
&& !is_toc_entry_line(plain_trimmed)
&& !is_heading_fragment(plain_trimmed)
&& toc_suppress_page != Some(line.page)
&& !(options.detect_code && line.items.iter().any(|i| is_monospace_font(&i.font)))
{
let line_font_size = line.items.first().map(|i| i.font_size).unwrap_or(base_size);
if let Some(header_level) =
@@ -1227,6 +1239,42 @@ mod tests {
}
}
fn line_at(text: &str, page: u32, y: f32) -> TextLine {
let mut item = make_item(text, page, None);
item.y = y;
make_line(vec![item])
}
#[test]
fn isolated_lines_kept_on_sparse_pages() {
// A ToC page with a lone title and one entry far below: the density
// ratio is 50% but the page is too sparse for the multi-column
// misfire the guard targets — the title must stay isolated.
let lines = vec![
line_at("CONTENTS", 1, 700.0),
line_at("Chapter One 5", 1, 500.0),
];
let isolated = find_isolated_lines(&lines, 12.0, 20.0);
assert!(
isolated.contains(&0),
"sparse-page title must stay isolated"
);
}
#[test]
fn isolated_lines_wiped_on_dense_pages() {
// 12 short lines all with paragraph gaps — the multi-column misfire
// shape. The guard must clear them all.
let lines: Vec<TextLine> = (0..12)
.map(|i| line_at("Short column line", 1, 700.0 - i as f32 * 50.0))
.collect();
let isolated = find_isolated_lines(&lines, 12.0, 20.0);
assert!(
isolated.is_empty(),
"dense page of isolated lines must be wiped"
);
}
#[test]
fn test_struct_role_heading() {
let lines = vec![
+94
View File
@@ -76,6 +76,52 @@ pub enum StructRole {
}
impl StructRole {
/// Content roles whose text must never be promoted to a heading by the
/// visual heuristic. These carry an explicit non-heading meaning in the
/// struct tree (lists, quotes, notes, references, captions, formulas,
/// forms, ToC entries), yet their text is often short and visually
/// isolated — exactly what the heuristic keys on. Heading roles (H, H1H6)
/// and generic container/flow roles (P, Div, Sect, Span, …) are excluded
/// so the heuristic can still fire there.
///
/// `Figure` is deliberately NOT in this set: cover/banner pages routinely
/// tag the document title inside a Figure (alongside a seal or logo), and
/// that title is a real heading. `Formula` and `Form` stay — a line
/// explicitly tagged as an equation or form field is never a heading.
///
/// Table roles (Table/TR/TH/TD/THead/TBody/TFoot) are included so that
/// when table reconstruction falls back and cells reach the line loop as
/// plain text, a short isolated cell — a `TH` column header especially —
/// is not promoted to a heading.
pub(crate) fn is_non_heading_content(&self) -> bool {
matches!(
self,
Self::L
| Self::LI
| Self::Lbl
| Self::LBody
| Self::BlockQuote
| Self::Quote
| Self::Caption
| Self::TOC
| Self::TOCI
| Self::Index
| Self::Note
| Self::Reference
| Self::BibEntry
| Self::Code
| Self::Formula
| Self::Form
| Self::Table
| Self::TR
| Self::TH
| Self::TD
| Self::THead
| Self::TBody
| Self::TFoot
)
}
fn from_name(name: &str) -> Self {
match name {
"Document" => Self::Document,
@@ -856,6 +902,54 @@ fn contains_bytes(haystack: &[u8], needle: &[u8]) -> bool {
mod tests {
use super::*;
#[test]
fn non_heading_content_roles() {
for r in [
StructRole::L,
StructRole::LI,
StructRole::BlockQuote,
StructRole::Quote,
StructRole::Caption,
StructRole::TOC,
StructRole::TOCI,
StructRole::Index,
StructRole::Note,
StructRole::Reference,
StructRole::BibEntry,
StructRole::Code,
StructRole::Formula,
StructRole::Form,
StructRole::Table,
StructRole::TR,
StructRole::TH,
StructRole::TD,
StructRole::THead,
StructRole::TBody,
StructRole::TFoot,
] {
assert!(
r.is_non_heading_content(),
"{r:?} should block heading promotion"
);
}
// Heading and generic container/flow roles must NOT block promotion
for r in [
StructRole::H,
StructRole::H1,
StructRole::H3,
StructRole::P,
StructRole::Div,
StructRole::Sect,
StructRole::Span,
StructRole::Figure,
] {
assert!(
!r.is_non_heading_content(),
"{r:?} should allow heading promotion"
);
}
}
#[test]
fn test_struct_role_from_name() {
assert_eq!(StructRole::from_name("H1"), StructRole::H1);
+1 -1
View File
@@ -76,6 +76,6 @@ forms simpler, we would be happy to hear from you. You can write to the Tax Form
**Unreported Tips.—**If you received tips of $20 or more for any month while working for one employer but did not report them to your employer, you must figure and pay social security and Medicare taxes on the unreported tips when you file your tax return. If you have unreported tips, you **must** use Form 1040 and **Form 4137,** Social Security and Medicare Tax on Unreported Tip Income, to report them. You may **not** use Form 1040A or 1040EZ. Employees subject to the Railroad Retirement Tax Act **cannot** use Form 4137 to pay railroad retirement tax on unreported tips. To get railroad retirement credit, you must report tips to your employer. If you do not report tips to your employer as required, you may be charged a penalty of 50% of the social security and Medicare taxes (or railroad retirement tax) due on the unreported tips unless there was reasonable cause for not reporting them. **Additional Information.—**Get **Pub. 531,** Reporting Tip Income, and Form 4137 for more information on tips. If you are an employee of certain large food or beverage establishments, see Pub. 531 for tip allocation rules. **Recordkeeping.—**If you do not keep a daily record of tips, you must keep other reliable proof of the tip income you received. This proof includes copies of restaurant bills and credit card charges that show amounts customers added as tips. Keep your tip income records for as long as the information on them may be needed in the administration of any Internal Revenue law.
**Instructions** *(continued)*
### Instructions (continued)
Use this space to total your tips for the year