feat: strip repeated headers/footers from markdown output

Add strip_repeated_lines() preprocessing step that removes running
headers and footers before markdown conversion. Uses multiple guard
rails to avoid false positives:

- Edge-line detection: only considers lines among the first/last 3
  distinct Y positions on each page (not percentage-based margins)
- Y-position consistency: requires low variance across pages (stddev
  < 5% of page span) to distinguish headers from scattered table content
- Position-aware removal: only strips instances at page edges, preserving
  body content that happens to match a header/footer text
- Skips structural lines (headings, list items), short lines (<10 chars),
  and decorative separators (repeated single characters)
- Frequency threshold: >= max(3, page_count * 30%) distinct pages

New strip_headers_footers option on MarkdownOptions (default: true).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Abimael Martell
2026-03-03 00:04:17 -08:00
co-authored by Claude Opus 4.6
parent 8f652b81a9
commit 7b6bb69feb
3 changed files with 191 additions and 13 deletions
+3 -12
View File
@@ -33,28 +33,19 @@ Date Date **a. Tips received**
**b. Credit card tips c. Tips paid out to d. Names of employees to whom you**
tips of directly from customers received other employees paid tips recd. entry and other employees 1 2 3 4 5 **Subtotals** **For Paperwork Reduction Act Notice, see Instructions on the back of Form 4070. Page 1**
Date Date **a. Tips received**
**b. Credit card tips c. Tips paid out to d. Names of employees to whom you**
tips of directly from customers received other employees paid tips recd. entry and other employees
received other employees paid tips recd. entry and other employees
7 8 9 10 11 12 13 14 15 **Subtotals**
**Page 2**
Date Date **a. Tips received**
**b. Credit card tips c. Tips paid out to d. Names of employees to whom you**
tips of directly from customers received other employees paid tips recd. entry and other employees
received other employees paid tips recd. entry and other employees
17 18 19 20 21 22 23 24 25 **Subtotals**
**Page 3**
Date Date **a. Tips received**
**b. Credit card tips c. Tips paid out to d. Names of employees to whom you**
tips of directly from customers received other employees paid tips recd. entry and other employees
received other employees paid tips recd. entry and other employees
27 28 29 30 31 **Subtotals** **from pages** **1, 2, and 3** **Totals**