feat: strip repeated headers/footers from markdown output
Add strip_repeated_lines() preprocessing step that removes running headers and footers before markdown conversion. Uses multiple guard rails to avoid false positives: - Edge-line detection: only considers lines among the first/last 3 distinct Y positions on each page (not percentage-based margins) - Y-position consistency: requires low variance across pages (stddev < 5% of page span) to distinguish headers from scattered table content - Position-aware removal: only strips instances at page edges, preserving body content that happens to match a header/footer text - Skips structural lines (headings, list items), short lines (<10 chars), and decorative separators (repeated single characters) - Frequency threshold: >= max(3, page_count * 30%) distinct pages New strip_headers_footers option on MarkdownOptions (default: true). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.6
parent
8f652b81a9
commit
7b6bb69feb
@@ -33,28 +33,19 @@ Date Date **a. Tips received**
|
||||
**b. Credit card tips c. Tips paid out to d. Names of employees to whom you**
|
||||
tips of directly from customers received other employees paid tips rec’d. entry and other employees 1 2 3 4 5 **Subtotals** **For Paperwork Reduction Act Notice, see Instructions on the back of Form 4070. Page 1**
|
||||
|
||||
Date Date **a. Tips received**
|
||||
|
||||
**b. Credit card tips c. Tips paid out to d. Names of employees to whom you**
|
||||
tips of directly from customers received other employees paid tips rec’d. entry and other employees
|
||||
received other employees paid tips rec’d. entry and other employees
|
||||
|
||||
7 8 9 10 11 12 13 14 15 **Subtotals**
|
||||
|
||||
**Page 2**
|
||||
|
||||
Date Date **a. Tips received**
|
||||
|
||||
**b. Credit card tips c. Tips paid out to d. Names of employees to whom you**
|
||||
tips of directly from customers received other employees paid tips rec’d. entry and other employees
|
||||
received other employees paid tips rec’d. entry and other employees
|
||||
|
||||
17 18 19 20 21 22 23 24 25 **Subtotals**
|
||||
|
||||
**Page 3**
|
||||
|
||||
Date Date **a. Tips received**
|
||||
|
||||
**b. Credit card tips c. Tips paid out to d. Names of employees to whom you**
|
||||
tips of directly from customers received other employees paid tips rec’d. entry and other employees
|
||||
received other employees paid tips rec’d. entry and other employees
|
||||
|
||||
27 28 29 30 31 **Subtotals** **from pages** **1, 2, and 3** **Totals**
|
||||
|
||||
|
||||
Reference in New Issue
Block a user