mu-ref/pdf-inspector
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: rarity-based heading detection inspired by opendataloader

Browse Source 
Replace ad-hoc bold/ratio heading checks with a unified scoring system
based on font size rarity. For each line, compute:
  score = font_rarity * 0.5 + bold * 0.3 + standalone * 0.2

Font rarity measures how infrequently a font size appears across the
document — heading fonts are rare while body text is common. This
approach (from opendataloader's ModeWeightStatistics) naturally adapts
to each document's font distribution instead of relying on fixed
thresholds.

Guards: require font_size >= 0.95 * base_size (no small-font headings),
word_count >= 3, and standalone (paragraph break before).

Benchmark improvement: MHS 0.56→0.58, MHS-S 0.66→0.70, overall +0.003.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Abimael Martell
2026-04-04 01:38:03 -07:00
co-authored by Claude Opus 4.6
parent b7ec80097b
commit 6e5abd0b48
5 changed files with 75 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/markdown/analysis.rs
+32 -2
View File
@@ -8,6 +8,24 @@ use log::debug;
/// Font statistics for a document
pub(crate) struct FontStats {
pub(crate) most_common_size: f32,
/// Font size frequency distribution (size_key → line count).
/// Used for rarity-based heading detection.
pub(crate) size_counts: HashMap<i32, usize>,
/// Total number of lines counted.
pub(crate) total_lines: usize,
}
/// Compute how rare a font size is in the document (0.0 = most common, 1.0 = unique).
/// Mirrors opendataloader's font rarity boosting approach: heading fonts appear on
/// far fewer lines than body text, so their percentile rank is high.
pub(crate) fn font_size_rarity(font_size: f32, stats: &FontStats) -> f32 {
if stats.total_lines == 0 {
return 0.0;
}
let key = (font_size * 10.0) as i32;
let count = stats.size_counts.get(&key).copied().unwrap_or(0);
// Rarity = 1 - (frequency ratio). A size used on 1/100 lines has rarity ~0.99.
1.0 - (count as f32 / stats.total_lines as f32)
}
/// Calculate font stats directly from items (before grouping into lines)
@@ -21,6 +39,8 @@ pub(crate) fn calculate_font_stats_from_items(items: &[TextItem]) -> FontStats {
}
}
let total_lines = size_counts.values().sum();
// Break ties by preferring the smaller font size for deterministic output
let most_common_size = size_counts
.iter()
@@ -30,7 +50,11 @@ pub(crate) fn calculate_font_stats_from_items(items: &[TextItem]) -> FontStats {
.map(|(size, _)| *size as f32 / 10.0)
.unwrap_or(12.0);
FontStats { most_common_size }
FontStats {
most_common_size,
size_counts,
total_lines,
}
}
/// Calculate font stats from grouped lines
@@ -48,6 +72,8 @@ pub(crate) fn calculate_font_stats(lines: &[TextLine]) -> FontStats {
}
}
let total_lines = size_counts.values().sum();
// Break ties by preferring the smaller font size for deterministic output
let most_common_size = size_counts
.iter()
@@ -57,7 +83,11 @@ pub(crate) fn calculate_font_stats(lines: &[TextLine]) -> FontStats {
.map(|(size, _)| *size as f32 / 10.0)
.unwrap_or(12.0);
FontStats { most_common_size }
FontStats {
most_common_size,
size_counts,
total_lines,
}
}
/// Determine the heading level for a bold-only line that didn't meet the font-size
src/markdown/convert.rs
+36 -21
View File
@@ -7,7 +7,7 @@ use crate::types::TextLine;
use super::analysis::{
bold_heading_level, calculate_font_stats, compute_heading_tiers, compute_paragraph_threshold,
detect_header_level, has_dot_leaders,
detect_header_level, font_size_rarity, has_dot_leaders,
};
use super::classify::{format_list_item, is_caption_line, is_list_item, is_monospace_font};
use super::postprocess::clean_markdown;
@@ -444,23 +444,31 @@ pub(super) fn to_markdown_from_lines_with_tables_and_images(
{
let line_font_size = line.items.first().map(|i| i.font_size).unwrap_or(base_size);
detect_header_level(line_font_size, base_size, &heading_tiers).or_else(|| {
// Rarity-based heading detection (inspired by opendataloader).
// Score = font_rarity * 0.5 + bold * 0.3 + standalone * 0.2
// Lines scoring above threshold are promoted to headings.
// Only consider lines at or above body font size.
if line_font_size < base_size * 0.95 {
return None;
}
let word_count = plain_trimmed.split_whitespace().count();
// Bold-only lines at body font size that are standalone (paragraph break
// before them) are likely section headings. Require ≥3 words to avoid
// promoting short labels/field names.
if !(1..=15).contains(&word_count) {
return None;
}
let rarity = font_size_rarity(line_font_size, &font_stats);
let all_bold = !line.items.is_empty() && line.items.iter().all(|i| i.is_bold);
if all_bold && !in_paragraph && word_count >= 3 {
return Some(bold_heading_level(&heading_tiers));
let standalone = !in_paragraph;
let score = rarity * 0.5
+ if all_bold { 0.3 } else { 0.0 }
+ if standalone { 0.2 } else { 0.0 };
// Require standalone + at least one other signal
if score >= 0.5 && standalone && word_count >= 3 {
Some(bold_heading_level(&heading_tiers))
} else {
None
}
// Lines slightly larger than body text (ratio 1.08-1.2) that are
// standalone and short are also likely headings. This catches
// academic paper headings with only a ~10% font size bump.
// Require ≥1 word to avoid single labels.
let ratio = line_font_size / base_size;
if (1.10..1.2).contains(&ratio) && !in_paragraph && (1..=8).contains(&word_count) {
return Some(bold_heading_level(&heading_tiers));
}
None
})
} else {
None
@@ -718,13 +726,20 @@ pub fn to_markdown_from_lines(lines: Vec<TextLine>, options: MarkdownOptions) ->
let line_font_size = line.items.first().map(|i| i.font_size).unwrap_or(base_size);
if let Some(header_level) =
detect_header_level(line_font_size, base_size, &heading_tiers).or_else(|| {
let word_count = plain_trimmed.split_whitespace().count();
let all_bold = !line.items.is_empty() && line.items.iter().all(|i| i.is_bold);
if all_bold && !in_paragraph && word_count >= 3 {
return Some(bold_heading_level(&heading_tiers));
if line_font_size < base_size * 0.95 {
return None;
}
let ratio = line_font_size / base_size;
if ratio >= 1.05 && !in_paragraph && word_count <= 10 {
let word_count = plain_trimmed.split_whitespace().count();
if !(1..=15).contains(&word_count) {
return None;
}
let rarity = font_size_rarity(line_font_size, &font_stats);
let all_bold = !line.items.is_empty() && line.items.iter().all(|i| i.is_bold);
let standalone = !in_paragraph;
let score = rarity * 0.5
+ if all_bold { 0.3 } else { 0.0 }
+ if standalone { 0.2 } else { 0.0 };
if score >= 0.5 && standalone && word_count >= 3 {
return Some(bold_heading_level(&heading_tiers));
}
None
tests/snapshots/p1244-1996.md
+3 -9
View File
@@ -10,9 +10,7 @@ Department of the Treasury **Internal Revenue Service**
### This publication contains:
### Form 4070A, Employee’s Daily Record of
Tips **Form 4070, Employee’s Report of Tips to** Employer
**Form 4070A, Employee’s Daily Record of** Tips **Form 4070, Employee’s Report of Tips to** Employer
For the period
@@ -70,17 +68,13 @@ Employer’s name and address (include establishment name, if different) **1** C
Month or shorter period in which tips were received **4** Net tips (lines 1 + 2 - 3) from, 19, to, 19 Signature Date
### Paperwork Reduction Act Notice.—We ask for the
information on these forms to carry out the Internal Revenue laws of the United States. You are required to give us the information. We need it to ensure that you are complying with these laws and to allow us to figure and collect the right amount of tax. You are not required to provide the information requested on a form that is subject to the Paperwork Reduction Act unless the form displays a valid OMB control number. Books or records relating to a form or its instructions must be retained as long as their contents may become material in the administration of any Internal Revenue law. Generally, tax returns and return information are confidential, as required by Code section 6103. The time needed to complete Forms 4070 and 4070A will vary depending on individual circumstances. The estimated average times are: Recordkeeping—Form 4070, 7 min.; Form 4070A, 3 hr. and 23 min.; Learning **about the law—each form, 2 min.; Preparing Form 4070,** 13 min.; Form 4070A, 55 min.; and Copying and **providing Form 4070, 10 min.; Form 4070A, 14 min.** If you have comments concerning the accuracy of these time estimates or suggestions for making these
**Paperwork Reduction Act Notice.—We ask for the** information on these forms to carry out the Internal Revenue laws of the United States. You are required to give us the information. We need it to ensure that you are complying with these laws and to allow us to figure and collect the right amount of tax. You are not required to provide the information requested on a form that is subject to the Paperwork Reduction Act unless the form displays a valid OMB control number. Books or records relating to a form or its instructions must be retained as long as their contents may become material in the administration of any Internal Revenue law. Generally, tax returns and return information are confidential, as required by Code section 6103. The time needed to complete Forms 4070 and 4070A will vary depending on individual circumstances. The estimated average times are: Recordkeeping—Form 4070, 7 min.; Form 4070A, 3 hr. and 23 min.; Learning **about the law—each form, 2 min.; Preparing Form 4070,** 13 min.; Form 4070A, 55 min.; and Copying and **providing Form 4070, 10 min.; Form 4070A, 14 min.** If you have comments concerning the accuracy of these time estimates or suggestions for making these
forms simpler, we would be happy to hear from you. You can write to the Tax Forms Committee, Western Area Distribution Center, Rancho Cordova, CA 95743-0001. **Purpose.—Use this form to report tips you receive to** your employer. This includes cash tips, tips you receive from other employees, and credit card tips. You must report tips every month regardless of your total wages and tips for the year. However, you do not have to report tips to your employer for any month you received less than $20 in tips while working for that employer. Report tips by the 10th day of the month following the month that you receive them. If the 10th day is a Saturday, Sunday, or legal holiday, report tips by the next day that is not a Saturday, Sunday, or legal holiday. See Pub. 531, Reporting Tip Income, for more information. You can get additional copies of Pub. 1244, Employee’s Daily Record of Tips and Report to Employer, which contains both Forms 4070A and 4070, by calling 1-800-TAX-FORM (1-800-829-3676).
**Instructions (continued)**
### Unreported Tips.—If you received tips of $20 or
more for any month while working for one employer but did not report them to your employer, you must figure and pay social security and Medicare taxes on the unreported tips when you file your tax return. If you have unreported tips, you must use Form 1040 and Form 4137, Social Security and Medicare Tax on Unreported Tip Income, to report them. You may not use Form 1040A or 1040EZ. Employees subject to the Railroad Retirement Tax Act cannot use Form 4137 to pay railroad retirement tax on unreported tips. To get railroad retirement credit, you must report tips to your employer. If you do not report tips to your employer as required, you may be charged a penalty of 50% of the social security and Medicare taxes (or railroad retirement tax) due on the unreported tips unless there was reasonable cause for not reporting them. **Additional Information.—Get Pub. 531, Reporting** Tip Income, and Form 4137 for more information on tips. If you are an employee of certain large food or beverage establishments, see Pub. 531 for tip allocation rules. **Recordkeeping.—If you do not keep a daily** record of tips, you must keep other reliable proof of the tip income you received. This proof includes copies of restaurant bills and credit card charges that show amounts customers added as tips. Keep your tip income records for as long as the information on them may be needed in the administration of any Internal Revenue law.
**Unreported Tips.—If you received tips of $20 or** more for any month while working for one employer but did not report them to your employer, you must figure and pay social security and Medicare taxes on the unreported tips when you file your tax return. If you have unreported tips, you must use Form 1040 and Form 4137, Social Security and Medicare Tax on Unreported Tip Income, to report them. You may not use Form 1040A or 1040EZ. Employees subject to the Railroad Retirement Tax Act cannot use Form 4137 to pay railroad retirement tax on unreported tips. To get railroad retirement credit, you must report tips to your employer. If you do not report tips to your employer as required, you may be charged a penalty of 50% of the social security and Medicare taxes (or railroad retirement tax) due on the unreported tips unless there was reasonable cause for not reporting them. **Additional Information.—Get Pub. 531, Reporting** Tip Income, and Form 4137 for more information on tips. If you are an employee of certain large food or beverage establishments, see Pub. 531 for tip allocation rules. **Recordkeeping.—If you do not keep a daily** record of tips, you must keep other reliable proof of the tip income you received. This proof includes copies of restaurant bills and credit card charges that show amounts customers added as tips. Keep your tip income records for as long as the information on them may be needed in the administration of any Internal Revenue law.
**Instructions (continued)**
tests/snapshots/real-estate-pricing.md
+3 -13
View File
@@ -6,11 +6,7 @@
8 4 Z E L L / L U R I E R E A L E S T A T E C E N T E R
## Table I: Cap rate correlations
## Cap Rate Correlation With:*
**BBB Corp** **10-Year Bond Yield S&P Dividend** **Treasury (10-15 yr) Yield** Multifamily 0.187 0.771 0.068 Industrial-0.221 0.748-0.307 CBD Office-0.449 0.694-0.458 Retail-0.181 0.649-02.58
**Table I: Cap rate correlations** **Cap Rate Correlation With:*** **BBB Corp** **10-Year Bond Yield S&P Dividend** **Treasury (10-15 yr) Yield** Multifamily 0.187 0.771 0.068 Industrial-0.221 0.748-0.307 CBD Office-0.449 0.694-0.458 Retail-0.181 0.649-02.58
* Based on 25 years of data for the 10-yrT & S&P DivYld; and 14 years for BBB.
**Figure 1:** NCREIF cap rates vs. 10-yearTreasury
@@ -24,9 +20,7 @@ R E V I E W 8 5
**Figure 2:** Capratespreadsover10-yearTreasury
## Basis Points -200
-400
**Basis Points -200** -400
-600
@@ -38,11 +32,7 @@ R E V I E W 8 5
1982 1986 1990 1994 1998 2002 2006
## Table II: Correlationsofspreadsbypropertytype
## Correlation of Cap Rate Spreads Over Treasury
## Multifamily Industrial CBD Office
**Table II: Correlationsofspreadsbypropertytype** **Correlation of Cap Rate Spreads Over Treasury** **Multifamily Industrial CBD Office**
||Multifamily|Industrial|CBD Office|
|---|---|---|---|
tests/snapshots/thermo-freon12.md
+1 -1
View File
@@ -2,7 +2,7 @@
## l T-12 SI
##### DuPont Fluorochemicals
DuPont Fluorochemicals
#### Thermodynamic Properties