fix(forms): resolve widget pages from annotations

AGENTS.md

@@ -61,8 +61,7 @@ src/
 
 - **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.
 - **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.
-- **Regression suite**: sibling repo `pdf-evals` with ~200 snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing. While iterating, prefer a subset run (`bench.py test -q` for the quick set, or `-s <name>` for a named test set) and save the full `bench.py test` for the final pre-commit check.
-- **Semantic quality**: run `bench.py score` in `pdf-evals` for the semantic verdict (TEDS + MHS + reading order + char/word + list preservation, composited). Character-level diff alone misclassifies structural improvements (e.g., column-detection rewrites) as regressions — `score` is the tie-breaker. See `pdf-evals/CLAUDE.md` "Semantic scoring".
+- **Regression suite**: sibling repo `pdf-evals` with 179+ snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing.
 
 ## Debugging
 

CLAUDE.md

@@ -61,7 +61,7 @@ src/
 
 - **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.
 - **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.
-- **Regression suite**: sibling repo `pdf-evals` with ~200 snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing. While iterating, prefer a subset run (`bench.py test -q` for the quick set, or `-s <name>` for a named test set) and save the full `bench.py test` for the final pre-commit check.
+- **Regression suite**: sibling repo `pdf-evals` with 187+ snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing.
 - **Semantic quality**: run `bench.py score` in `pdf-evals` for the semantic verdict (TEDS + MHS + reading order + char/word + list preservation, composited). Character-level diff alone misclassifies structural improvements (e.g., column-detection rewrites) as regressions — `score` is the tie-breaker. See `pdf-evals/CLAUDE.md` "Semantic scoring".
 
 ## Debugging

README.md

@@ -238,7 +238,7 @@ The converter handles:
 |---|---|
 | Headings (H1-H4) | Font size tiers relative to body text, with 0.5pt clustering |
 | Bold/italic | Font name patterns (Bold, Italic, Oblique) |
-| Bullet lists | `•`, `-`, `*`, `○`, `●`, `◦` prefixes |
+| Bullet lists | `*`, `-`, `*`, `○`, `●`, `◦` prefixes |
 | Numbered lists | `1.`, `1)`, `(1)` patterns |
 | Letter lists | `a.`, `a)`, `(a)` patterns |
 | Code blocks | Monospace fonts (Courier, Consolas, Monaco, Menlo, Fira Code, JetBrains Mono) and keyword detection |

docs/python.md

@@ -111,8 +111,7 @@ class PdfResult:                     # process_pdf / detect_pdf
     markdown: str | None             # extracted Markdown (None for detect_pdf)
     page_count: int
     processing_time_ms: int
-    pages_needing_ocr: list[int]     # 1-indexed
-    ocr_reasons_by_page: list[PageOcrReasons]
+    pages_needing_ocr: list[int]
     title: str | None
     confidence: float                # 0.0 - 1.0
     is_complex_layout: bool
@@ -120,10 +119,6 @@ class PdfResult:                     # process_pdf / detect_pdf
     pages_with_columns: list[int]
     has_encoding_issues: bool        # broken font encodings — consider OCR fallback
 
-class PageOcrReasons:                # per-page OCR diagnostics
-    page: int                        # 1-indexed
-    reasons: list[str]               # machine-readable reason identifiers
-
 class PdfClassification:             # classify_pdf
     pdf_type: str
     page_count: int
@@ -145,20 +140,14 @@ class TextItem:                      # extract_text_with_positions
     is_strikeout: bool
     item_type: str
 
-class RegionText:                    # extract_text_in_regions
-    text: str
-    needs_ocr: bool
-    ocr_reason: str | None           # machine-readable OCR reason
-
 class PageRegionTexts:               # extract_text_in_regions
     page: int                        # 0-indexed
-    regions: list[RegionText]
+    regions: list[RegionText]        # RegionText: text: str, needs_ocr: bool
 
 class PagesExtractionResult:         # extract_pages_markdown
-    pages: list[PageMarkdown]        # PageMarkdown: page (0-indexed), markdown, needs_ocr, ocr_reason
+    pages: list[PageMarkdown]        # PageMarkdown: page (0-indexed), markdown, needs_ocr
     pages_with_tables: list[int]     # 1-indexed
     pages_with_columns: list[int]    # 1-indexed
     pages_needing_ocr: list[int]     # 1-indexed
-    ocr_reasons_by_page: list[PageOcrReasons]
     is_complex: bool                 # any page has tables or multi-column layout
 ```

pdf_inspector.pyi

@@ -10,9 +10,6 @@ class PdfResult:
     page_count: int
     processing_time_ms: int
     pages_needing_ocr: list[int]
-    """1-indexed page numbers that need OCR."""
-    ocr_reasons_by_page: list["PageOcrReasons"]
-    """Machine-readable OCR reasons by 1-indexed page."""
     title: Optional[str]
     confidence: float
     is_complex_layout: bool
@@ -20,13 +17,6 @@ class PdfResult:
     pages_with_columns: list[int]
     has_encoding_issues: bool
 
-class PageOcrReasons:
-    """OCR reasons for a single 1-indexed page."""
-    page: int
-    """1-indexed page number."""
-    reasons: list[str]
-    """Machine-readable OCR reason identifiers."""
-
 class PdfClassification:
     """Lightweight PDF classification result."""
     pdf_type: str
@@ -57,8 +47,6 @@ class RegionText:
     text: str
     needs_ocr: bool
     """True when the text should not be trusted."""
-    ocr_reason: Optional[str]
-    """Machine-readable OCR reason when the cause is known."""
 
 class PageRegionTexts:
     """Extracted text for one page's regions."""
@@ -74,8 +62,6 @@ class PageMarkdown:
     """Formatted markdown for this page (empty string when needs_ocr is True)."""
     needs_ocr: bool
     """True when text on this page is unreliable and OCR should be used instead."""
-    ocr_reason: Optional[str]
-    """Machine-readable OCR reason when the cause is known."""
 
 class PagesExtractionResult:
     """Per-page markdown output with document-wide layout classification."""
@@ -87,8 +73,6 @@ class PagesExtractionResult:
     """1-indexed pages where multi-column layout was detected."""
     pages_needing_ocr: list[int]
     """1-indexed pages that need OCR."""
-    ocr_reasons_by_page: list[PageOcrReasons]
-    """Machine-readable OCR reasons by 1-indexed page."""
     is_complex: bool
     """True if any page has tables or multi-column layout."""
 

src/bin/pdf2md.rs

@@ -2,8 +2,8 @@
 
 use pdf_inspector::extractor::ItemType;
 use pdf_inspector::{
-    extract_text_with_positions_pages_with_password, process_pdf_with_options, LayoutComplexity,
-    PdfOptions, PdfType, ProcessMode, TextItem,
+    extract_text_with_positions_pages, process_pdf_with_options, LayoutComplexity, PdfOptions,
+    PdfType, ProcessMode, TextItem,
 };
 use std::collections::HashSet;
 use std::env;
@@ -103,18 +103,9 @@ fn format_items_json(items: &[TextItem]) -> String {
     )
 }
 
-fn extract_items_json(
-    pdf_path: &str,
-    page_filter: Option<&HashSet<u32>>,
-    password: Option<&str>,
-) -> Result<String, pdf_inspector::PdfError> {
-    extract_text_with_positions_pages_with_password(pdf_path, page_filter, password)
-        .map(|items| format_items_json(&items))
-}
-
 #[cfg(test)]
 mod tests {
-    use super::{extract_items_json, format_items_json};
+    use super::format_items_json;
     use pdf_inspector::extractor::ItemType;
     use pdf_inspector::TextItem;
 
@@ -146,24 +137,6 @@ mod tests {
         assert!(json.contains(r#""item_type":"text""#));
         assert!(json.contains(r#""mcid":7"#));
     }
-
-    #[test]
-    fn items_json_uses_supplied_pdf_password() {
-        let path = "tests/fixtures/encrypted-secret123.pdf";
-
-        let without_password = extract_items_json(path, None, None);
-        assert!(
-            without_password.is_err(),
-            "encrypted fixture unexpectedly extracted without a password"
-        );
-
-        let json = extract_items_json(path, None, Some("secret123"))
-            .expect("correct password should decrypt positioned text");
-        assert!(
-            json.contains("Procurement"),
-            "decrypted item JSON should contain fixture text, got {json}"
-        );
-    }
 }
 
 /// Parse a page specification like "1,3,5-10,20" into a HashSet of page numbers.
@@ -284,8 +257,8 @@ fn main() {
         });
 
     if items_json_output {
-        match extract_items_json(pdf_path, page_filter.as_ref(), password.as_deref()) {
-            Ok(json) => println!("{}", json),
+        match extract_text_with_positions_pages(pdf_path, page_filter.as_ref()) {
+            Ok(items) => println!("{}", format_items_json(&items)),
             Err(e) => {
                 println!(r#"{{"error":"{}"}}"#, json_escape(&e.to_string()));
                 process::exit(1);

src/extractor/base14.rs

@@ -1,649 +0,0 @@
-//! Built-in glyph metrics for the 14 standard PDF fonts.
-//!
-//! PDFs may omit `/Widths` for non-embedded base-14 fonts (Times, Helvetica,
-//! Courier, Symbol, ZapfDingbats); per the PDF spec the reader must supply
-//! the metrics. Without them every text item gets width 0, which breaks
-//! space synthesis, sub/superscript detection, and table column detection
-//! (common in 1990s dvips/Distiller output).
-//!
-//! Tables are generated from the Adobe Core 14 AFM files (via reportlab's
-//! `_fontdata`), keyed by Unicode char, sorted for binary search.
-//! Generator: scratchpad/gen_base14.py (session tooling, not checked in).
-
-/// Width in 1000ths of an em for `c` in the given base-14 font, or `None`
-/// if the font is not one of the base 14 (after name normalization) or the
-/// char has no glyph in its AFM.
-pub(crate) fn base14_char_width(base_font: &str, c: char) -> Option<u16> {
-    let table = base14_table(base_font)?;
-    // AFM tables key visible glyphs only; alias the invisible variants the
-    // cp1252 fallback can produce so they get the metric of their visible
-    // counterpart instead of the generic default.
-    let c = match c {
-        '\u{00A0}' => ' ', // no-break space -> space
-        '\u{00AD}' => '-', // soft hyphen -> hyphen
-        _ => c,
-    };
-    table
-        .binary_search_by_key(&c, |&(ch, _)| ch)
-        .ok()
-        .map(|i| table[i].1)
-}
-
-/// True when the base font name normalizes to one of the standard 14 fonts.
-pub(crate) fn is_base14_font(base_font: &str) -> bool {
-    base14_table(base_font).is_some()
-}
-
-/// Code → Unicode through the font's BUILT-IN encoding, for the base-14
-/// fonts whose repertoire is not Latin (Symbol, ZapfDingbats). Their glyphs
-/// live at byte positions that have nothing to do with cp1252 (Symbol 0x61
-/// renders α, Zapf 0x21 renders ✁), so advance widths must be resolved
-/// through this mapping — the renderer draws these glyphs regardless of how
-/// the text decoder transliterates them. Returns `None` for the Latin text
-/// fonts, which follow standard single-byte encodings.
-pub(crate) fn builtin_encoding_char(base_font: &str, code: u8) -> Option<char> {
-    let table = base14_table(base_font)?;
-    let enc: &[(u8, char)] = if std::ptr::eq(table, SYMBOL) {
-        SYMBOL_ENCODING
-    } else if std::ptr::eq(table, ZAPFDINGBATS) {
-        ZAPFDINGBATS_ENCODING
-    } else {
-        return None;
-    };
-    enc.binary_search_by_key(&code, |&(b, _)| b)
-        .ok()
-        .map(|i| enc[i].1)
-}
-
-/// Map a BaseFont name (possibly subset-prefixed, e.g. "ABCDEF+Times-Bold",
-/// or a common alias like "Arial" / "TimesNewRomanPSMT") to its width table.
-fn base14_table(base_font: &str) -> Option<&'static [(char, u16)]> {
-    // Strip subset prefix "ABCDEF+"
-    let name = match base_font.split_once('+') {
-        Some((prefix, rest))
-            if prefix.len() == 6 && prefix.chars().all(|c| c.is_ascii_uppercase()) =>
-        {
-            rest
-        }
-        _ => base_font,
-    };
-    let lower = name.to_ascii_lowercase();
-    let bold = lower.contains("bold");
-    let italic = lower.contains("italic") || lower.contains("oblique");
-    if lower.contains("courier") {
-        return Some(match (bold, italic) {
-            (false, false) => COURIER,
-            (true, false) => COURIER_BOLD,
-            (false, true) => COURIER_OBLIQUE,
-            (true, true) => COURIER_BOLDOBLIQUE,
-        });
-    }
-    if lower.contains("helvetica") || lower.contains("arial") {
-        return Some(match (bold, italic) {
-            (false, false) => HELVETICA,
-            (true, false) => HELVETICA_BOLD,
-            (false, true) => HELVETICA_OBLIQUE,
-            (true, true) => HELVETICA_BOLDOBLIQUE,
-        });
-    }
-    if lower.contains("times") {
-        return Some(match (bold, italic) {
-            (false, false) => TIMES_ROMAN,
-            (true, false) => TIMES_BOLD,
-            (false, true) => TIMES_ITALIC,
-            (true, true) => TIMES_BOLDITALIC,
-        });
-    }
-    // Symbol and ZapfDingbats have unique glyph repertoires, so only exact
-    // names (plus the common MT/ITC aliases) qualify — a custom font that
-    // merely mentions "Symbol" in its name must not get these metrics.
-    match lower.as_str() {
-        "zapfdingbats" | "dingbats" | "itczapfdingbats" | "zapfdingbatsitc" => {
-            return Some(ZAPFDINGBATS)
-        }
-        "symbol" | "symbolmt" | "symbolitc" => return Some(SYMBOL),
-        _ => {}
-    }
-    None
-}
-
-#[rustfmt::skip]
-static COURIER: &[(char, u16)] = &[
-    (' ', 600), ('!', 600), ('"', 600), ('#', 600), ('$', 600), ('%', 600),
-    ('&', 600), ('\'', 600), ('(', 600), (')', 600), ('*', 600), ('+', 600),
-    (',', 600), ('-', 600), ('.', 600), ('/', 600), ('0', 600), ('1', 600),
-    ('2', 600), ('3', 600), ('4', 600), ('5', 600), ('6', 600), ('7', 600),
-    ('8', 600), ('9', 600), (':', 600), (';', 600), ('<', 600), ('=', 600),
-    ('>', 600), ('?', 600), ('@', 600), ('A', 600), ('B', 600), ('C', 600),
-    ('D', 600), ('E', 600), ('F', 600), ('G', 600), ('H', 600), ('I', 600),
-    ('J', 600), ('K', 600), ('L', 600), ('M', 600), ('N', 600), ('O', 600),
-    ('P', 600), ('Q', 600), ('R', 600), ('S', 600), ('T', 600), ('U', 600),
-    ('V', 600), ('W', 600), ('X', 600), ('Y', 600), ('Z', 600), ('[', 600),
-    ('\\', 600), (']', 600), ('^', 600), ('_', 600), ('`', 600), ('a', 600),
-    ('b', 600), ('c', 600), ('d', 600), ('e', 600), ('f', 600), ('g', 600),
-    ('h', 600), ('i', 600), ('j', 600), ('k', 600), ('l', 600), ('m', 600),
-    ('n', 600), ('o', 600), ('p', 600), ('q', 600), ('r', 600), ('s', 600),
-    ('t', 600), ('u', 600), ('v', 600), ('w', 600), ('x', 600), ('y', 600),
-    ('z', 600), ('{', 600), ('|', 600), ('}', 600), ('~', 600), ('\u{00A1}', 600),
-    ('\u{00A2}', 600), ('\u{00A3}', 600), ('\u{00A4}', 600), ('\u{00A5}', 600), ('\u{00A6}', 600), ('\u{00A7}', 600),
-    ('\u{00A8}', 600), ('\u{00A9}', 600), ('\u{00AA}', 600), ('\u{00AB}', 600), ('\u{00AC}', 600), ('\u{00AE}', 600),
-    ('\u{00AF}', 600), ('\u{00B0}', 600), ('\u{00B1}', 600), ('\u{00B2}', 600), ('\u{00B3}', 600), ('\u{00B4}', 600),
-    ('\u{00B5}', 600), ('\u{00B6}', 600), ('\u{00B7}', 600), ('\u{00B8}', 600), ('\u{00B9}', 600), ('\u{00BA}', 600),
-    ('\u{00BB}', 600), ('\u{00BC}', 600), ('\u{00BD}', 600), ('\u{00BE}', 600), ('\u{00BF}', 600), ('\u{00C0}', 600),
-    ('\u{00C1}', 600), ('\u{00C2}', 600), ('\u{00C3}', 600), ('\u{00C4}', 600), ('\u{00C5}', 600), ('\u{00C6}', 600),
-    ('\u{00C7}', 600), ('\u{00C8}', 600), ('\u{00C9}', 600), ('\u{00CA}', 600), ('\u{00CB}', 600), ('\u{00CC}', 600),
-    ('\u{00CD}', 600), ('\u{00CE}', 600), ('\u{00CF}', 600), ('\u{00D0}', 600), ('\u{00D1}', 600), ('\u{00D2}', 600),
-    ('\u{00D3}', 600), ('\u{00D4}', 600), ('\u{00D5}', 600), ('\u{00D6}', 600), ('\u{00D7}', 600), ('\u{00D8}', 600),
-    ('\u{00D9}', 600), ('\u{00DA}', 600), ('\u{00DB}', 600), ('\u{00DC}', 600), ('\u{00DD}', 600), ('\u{00DE}', 600),
-    ('\u{00DF}', 600), ('\u{00E0}', 600), ('\u{00E1}', 600), ('\u{00E2}', 600), ('\u{00E3}', 600), ('\u{00E4}', 600),
-    ('\u{00E5}', 600), ('\u{00E6}', 600), ('\u{00E7}', 600), ('\u{00E8}', 600), ('\u{00E9}', 600), ('\u{00EA}', 600),
-    ('\u{00EB}', 600), ('\u{00EC}', 600), ('\u{00ED}', 600), ('\u{00EE}', 600), ('\u{00EF}', 600), ('\u{00F0}', 600),
-    ('\u{00F1}', 600), ('\u{00F2}', 600), ('\u{00F3}', 600), ('\u{00F4}', 600), ('\u{00F5}', 600), ('\u{00F6}', 600),
-    ('\u{00F7}', 600), ('\u{00F8}', 600), ('\u{00F9}', 600), ('\u{00FA}', 600), ('\u{00FB}', 600), ('\u{00FC}', 600),
-    ('\u{00FD}', 600), ('\u{00FE}', 600), ('\u{00FF}', 600), ('\u{0131}', 600), ('\u{0141}', 600), ('\u{0142}', 600),
-    ('\u{0152}', 600), ('\u{0153}', 600), ('\u{0160}', 600), ('\u{0161}', 600), ('\u{0178}', 600), ('\u{017D}', 600),
-    ('\u{017E}', 600), ('\u{0192}', 600), ('\u{02C6}', 600), ('\u{02C7}', 600), ('\u{02D8}', 600), ('\u{02D9}', 600),
-    ('\u{02DA}', 600), ('\u{02DB}', 600), ('\u{02DC}', 600), ('\u{02DD}', 600), ('\u{2013}', 600), ('\u{2014}', 600),
-    ('\u{2018}', 600), ('\u{2019}', 600), ('\u{201A}', 600), ('\u{201C}', 600), ('\u{201D}', 600), ('\u{201E}', 600),
-    ('\u{2020}', 600), ('\u{2021}', 600), ('\u{2022}', 600), ('\u{2026}', 600), ('\u{2030}', 600), ('\u{2039}', 600),
-    ('\u{203A}', 600), ('\u{2044}', 600), ('\u{20AC}', 600), ('\u{2122}', 600), ('\u{2212}', 600), ('\u{FB01}', 600),
-    ('\u{FB02}', 600),
-];
-
-static COURIER_BOLD: &[(char, u16)] = COURIER;
-
-static COURIER_OBLIQUE: &[(char, u16)] = COURIER;
-
-static COURIER_BOLDOBLIQUE: &[(char, u16)] = COURIER;
-
-#[rustfmt::skip]
-static HELVETICA: &[(char, u16)] = &[
-    (' ', 278), ('!', 278), ('"', 355), ('#', 556), ('$', 556), ('%', 889),
-    ('&', 667), ('\'', 191), ('(', 333), (')', 333), ('*', 389), ('+', 584),
-    (',', 278), ('-', 333), ('.', 278), ('/', 278), ('0', 556), ('1', 556),
-    ('2', 556), ('3', 556), ('4', 556), ('5', 556), ('6', 556), ('7', 556),
-    ('8', 556), ('9', 556), (':', 278), (';', 278), ('<', 584), ('=', 584),
-    ('>', 584), ('?', 556), ('@', 1015), ('A', 667), ('B', 667), ('C', 722),
-    ('D', 722), ('E', 667), ('F', 611), ('G', 778), ('H', 722), ('I', 278),
-    ('J', 500), ('K', 667), ('L', 556), ('M', 833), ('N', 722), ('O', 778),
-    ('P', 667), ('Q', 778), ('R', 722), ('S', 667), ('T', 611), ('U', 722),
-    ('V', 667), ('W', 944), ('X', 667), ('Y', 667), ('Z', 611), ('[', 278),
-    ('\\', 278), (']', 278), ('^', 469), ('_', 556), ('`', 333), ('a', 556),
-    ('b', 556), ('c', 500), ('d', 556), ('e', 556), ('f', 278), ('g', 556),
-    ('h', 556), ('i', 222), ('j', 222), ('k', 500), ('l', 222), ('m', 833),
-    ('n', 556), ('o', 556), ('p', 556), ('q', 556), ('r', 333), ('s', 500),
-    ('t', 278), ('u', 556), ('v', 500), ('w', 722), ('x', 500), ('y', 500),
-    ('z', 500), ('{', 334), ('|', 260), ('}', 334), ('~', 584), ('\u{00A1}', 333),
-    ('\u{00A2}', 556), ('\u{00A3}', 556), ('\u{00A4}', 556), ('\u{00A5}', 556), ('\u{00A6}', 260), ('\u{00A7}', 556),
-    ('\u{00A8}', 333), ('\u{00A9}', 737), ('\u{00AA}', 370), ('\u{00AB}', 556), ('\u{00AC}', 584), ('\u{00AE}', 737),
-    ('\u{00AF}', 333), ('\u{00B0}', 400), ('\u{00B1}', 584), ('\u{00B2}', 333), ('\u{00B3}', 333), ('\u{00B4}', 333),
-    ('\u{00B5}', 556), ('\u{00B6}', 537), ('\u{00B7}', 278), ('\u{00B8}', 333), ('\u{00B9}', 333), ('\u{00BA}', 365),
-    ('\u{00BB}', 556), ('\u{00BC}', 834), ('\u{00BD}', 834), ('\u{00BE}', 834), ('\u{00BF}', 611), ('\u{00C0}', 667),
-    ('\u{00C1}', 667), ('\u{00C2}', 667), ('\u{00C3}', 667), ('\u{00C4}', 667), ('\u{00C5}', 667), ('\u{00C6}', 1000),
-    ('\u{00C7}', 722), ('\u{00C8}', 667), ('\u{00C9}', 667), ('\u{00CA}', 667), ('\u{00CB}', 667), ('\u{00CC}', 278),
-    ('\u{00CD}', 278), ('\u{00CE}', 278), ('\u{00CF}', 278), ('\u{00D0}', 722), ('\u{00D1}', 722), ('\u{00D2}', 778),
-    ('\u{00D3}', 778), ('\u{00D4}', 778), ('\u{00D5}', 778), ('\u{00D6}', 778), ('\u{00D7}', 584), ('\u{00D8}', 778),
-    ('\u{00D9}', 722), ('\u{00DA}', 722), ('\u{00DB}', 722), ('\u{00DC}', 722), ('\u{00DD}', 667), ('\u{00DE}', 667),
-    ('\u{00DF}', 611), ('\u{00E0}', 556), ('\u{00E1}', 556), ('\u{00E2}', 556), ('\u{00E3}', 556), ('\u{00E4}', 556),
-    ('\u{00E5}', 556), ('\u{00E6}', 889), ('\u{00E7}', 500), ('\u{00E8}', 556), ('\u{00E9}', 556), ('\u{00EA}', 556),
-    ('\u{00EB}', 556), ('\u{00EC}', 278), ('\u{00ED}', 278), ('\u{00EE}', 278), ('\u{00EF}', 278), ('\u{00F0}', 556),
-    ('\u{00F1}', 556), ('\u{00F2}', 556), ('\u{00F3}', 556), ('\u{00F4}', 556), ('\u{00F5}', 556), ('\u{00F6}', 556),
-    ('\u{00F7}', 584), ('\u{00F8}', 611), ('\u{00F9}', 556), ('\u{00FA}', 556), ('\u{00FB}', 556), ('\u{00FC}', 556),
-    ('\u{00FD}', 500), ('\u{00FE}', 556), ('\u{00FF}', 500), ('\u{0131}', 278), ('\u{0141}', 556), ('\u{0142}', 222),
-    ('\u{0152}', 1000), ('\u{0153}', 944), ('\u{0160}', 667), ('\u{0161}', 500), ('\u{0178}', 667), ('\u{017D}', 611),
-    ('\u{017E}', 500), ('\u{0192}', 556), ('\u{02C6}', 333), ('\u{02C7}', 333), ('\u{02D8}', 333), ('\u{02D9}', 333),
-    ('\u{02DA}', 333), ('\u{02DB}', 333), ('\u{02DC}', 333), ('\u{02DD}', 333), ('\u{2013}', 556), ('\u{2014}', 1000),
-    ('\u{2018}', 222), ('\u{2019}', 222), ('\u{201A}', 222), ('\u{201C}', 333), ('\u{201D}', 333), ('\u{201E}', 333),
-    ('\u{2020}', 556), ('\u{2021}', 556), ('\u{2022}', 350), ('\u{2026}', 1000), ('\u{2030}', 1000), ('\u{2039}', 333),
-    ('\u{203A}', 333), ('\u{2044}', 167), ('\u{20AC}', 556), ('\u{2122}', 1000), ('\u{2212}', 584), ('\u{FB01}', 500),
-    ('\u{FB02}', 500),
-];
-
-#[rustfmt::skip]
-static HELVETICA_BOLD: &[(char, u16)] = &[
-    (' ', 278), ('!', 333), ('"', 474), ('#', 556), ('$', 556), ('%', 889),
-    ('&', 722), ('\'', 238), ('(', 333), (')', 333), ('*', 389), ('+', 584),
-    (',', 278), ('-', 333), ('.', 278), ('/', 278), ('0', 556), ('1', 556),
-    ('2', 556), ('3', 556), ('4', 556), ('5', 556), ('6', 556), ('7', 556),
-    ('8', 556), ('9', 556), (':', 333), (';', 333), ('<', 584), ('=', 584),
-    ('>', 584), ('?', 611), ('@', 975), ('A', 722), ('B', 722), ('C', 722),
-    ('D', 722), ('E', 667), ('F', 611), ('G', 778), ('H', 722), ('I', 278),
-    ('J', 556), ('K', 722), ('L', 611), ('M', 833), ('N', 722), ('O', 778),
-    ('P', 667), ('Q', 778), ('R', 722), ('S', 667), ('T', 611), ('U', 722),
-    ('V', 667), ('W', 944), ('X', 667), ('Y', 667), ('Z', 611), ('[', 333),
-    ('\\', 278), (']', 333), ('^', 584), ('_', 556), ('`', 333), ('a', 556),
-    ('b', 611), ('c', 556), ('d', 611), ('e', 556), ('f', 333), ('g', 611),
-    ('h', 611), ('i', 278), ('j', 278), ('k', 556), ('l', 278), ('m', 889),
-    ('n', 611), ('o', 611), ('p', 611), ('q', 611), ('r', 389), ('s', 556),
-    ('t', 333), ('u', 611), ('v', 556), ('w', 778), ('x', 556), ('y', 556),
-    ('z', 500), ('{', 389), ('|', 280), ('}', 389), ('~', 584), ('\u{00A1}', 333),
-    ('\u{00A2}', 556), ('\u{00A3}', 556), ('\u{00A4}', 556), ('\u{00A5}', 556), ('\u{00A6}', 280), ('\u{00A7}', 556),
-    ('\u{00A8}', 333), ('\u{00A9}', 737), ('\u{00AA}', 370), ('\u{00AB}', 556), ('\u{00AC}', 584), ('\u{00AE}', 737),
-    ('\u{00AF}', 333), ('\u{00B0}', 400), ('\u{00B1}', 584), ('\u{00B2}', 333), ('\u{00B3}', 333), ('\u{00B4}', 333),
-    ('\u{00B5}', 611), ('\u{00B6}', 556), ('\u{00B7}', 278), ('\u{00B8}', 333), ('\u{00B9}', 333), ('\u{00BA}', 365),
-    ('\u{00BB}', 556), ('\u{00BC}', 834), ('\u{00BD}', 834), ('\u{00BE}', 834), ('\u{00BF}', 611), ('\u{00C0}', 722),
-    ('\u{00C1}', 722), ('\u{00C2}', 722), ('\u{00C3}', 722), ('\u{00C4}', 722), ('\u{00C5}', 722), ('\u{00C6}', 1000),
-    ('\u{00C7}', 722), ('\u{00C8}', 667), ('\u{00C9}', 667), ('\u{00CA}', 667), ('\u{00CB}', 667), ('\u{00CC}', 278),
-    ('\u{00CD}', 278), ('\u{00CE}', 278), ('\u{00CF}', 278), ('\u{00D0}', 722), ('\u{00D1}', 722), ('\u{00D2}', 778),
-    ('\u{00D3}', 778), ('\u{00D4}', 778), ('\u{00D5}', 778), ('\u{00D6}', 778), ('\u{00D7}', 584), ('\u{00D8}', 778),
-    ('\u{00D9}', 722), ('\u{00DA}', 722), ('\u{00DB}', 722), ('\u{00DC}', 722), ('\u{00DD}', 667), ('\u{00DE}', 667),
-    ('\u{00DF}', 611), ('\u{00E0}', 556), ('\u{00E1}', 556), ('\u{00E2}', 556), ('\u{00E3}', 556), ('\u{00E4}', 556),
-    ('\u{00E5}', 556), ('\u{00E6}', 889), ('\u{00E7}', 556), ('\u{00E8}', 556), ('\u{00E9}', 556), ('\u{00EA}', 556),
-    ('\u{00EB}', 556), ('\u{00EC}', 278), ('\u{00ED}', 278), ('\u{00EE}', 278), ('\u{00EF}', 278), ('\u{00F0}', 611),
-    ('\u{00F1}', 611), ('\u{00F2}', 611), ('\u{00F3}', 611), ('\u{00F4}', 611), ('\u{00F5}', 611), ('\u{00F6}', 611),
-    ('\u{00F7}', 584), ('\u{00F8}', 611), ('\u{00F9}', 611), ('\u{00FA}', 611), ('\u{00FB}', 611), ('\u{00FC}', 611),
-    ('\u{00FD}', 556), ('\u{00FE}', 611), ('\u{00FF}', 556), ('\u{0131}', 278), ('\u{0141}', 611), ('\u{0142}', 278),
-    ('\u{0152}', 1000), ('\u{0153}', 944), ('\u{0160}', 667), ('\u{0161}', 556), ('\u{0178}', 667), ('\u{017D}', 611),
-    ('\u{017E}', 500), ('\u{0192}', 556), ('\u{02C6}', 333), ('\u{02C7}', 333), ('\u{02D8}', 333), ('\u{02D9}', 333),
-    ('\u{02DA}', 333), ('\u{02DB}', 333), ('\u{02DC}', 333), ('\u{02DD}', 333), ('\u{2013}', 556), ('\u{2014}', 1000),
-    ('\u{2018}', 278), ('\u{2019}', 278), ('\u{201A}', 278), ('\u{201C}', 500), ('\u{201D}', 500), ('\u{201E}', 500),
-    ('\u{2020}', 556), ('\u{2021}', 556), ('\u{2022}', 350), ('\u{2026}', 1000), ('\u{2030}', 1000), ('\u{2039}', 333),
-    ('\u{203A}', 333), ('\u{2044}', 167), ('\u{20AC}', 556), ('\u{2122}', 1000), ('\u{2212}', 584), ('\u{FB01}', 611),
-    ('\u{FB02}', 611),
-];
-
-static HELVETICA_OBLIQUE: &[(char, u16)] = HELVETICA;
-
-static HELVETICA_BOLDOBLIQUE: &[(char, u16)] = HELVETICA_BOLD;
-
-#[rustfmt::skip]
-static TIMES_ROMAN: &[(char, u16)] = &[
-    (' ', 250), ('!', 333), ('"', 408), ('#', 500), ('$', 500), ('%', 833),
-    ('&', 778), ('\'', 180), ('(', 333), (')', 333), ('*', 500), ('+', 564),
-    (',', 250), ('-', 333), ('.', 250), ('/', 278), ('0', 500), ('1', 500),
-    ('2', 500), ('3', 500), ('4', 500), ('5', 500), ('6', 500), ('7', 500),
-    ('8', 500), ('9', 500), (':', 278), (';', 278), ('<', 564), ('=', 564),
-    ('>', 564), ('?', 444), ('@', 921), ('A', 722), ('B', 667), ('C', 667),
-    ('D', 722), ('E', 611), ('F', 556), ('G', 722), ('H', 722), ('I', 333),
-    ('J', 389), ('K', 722), ('L', 611), ('M', 889), ('N', 722), ('O', 722),
-    ('P', 556), ('Q', 722), ('R', 667), ('S', 556), ('T', 611), ('U', 722),
-    ('V', 722), ('W', 944), ('X', 722), ('Y', 722), ('Z', 611), ('[', 333),
-    ('\\', 278), (']', 333), ('^', 469), ('_', 500), ('`', 333), ('a', 444),
-    ('b', 500), ('c', 444), ('d', 500), ('e', 444), ('f', 333), ('g', 500),
-    ('h', 500), ('i', 278), ('j', 278), ('k', 500), ('l', 278), ('m', 778),
-    ('n', 500), ('o', 500), ('p', 500), ('q', 500), ('r', 333), ('s', 389),
-    ('t', 278), ('u', 500), ('v', 500), ('w', 722), ('x', 500), ('y', 500),
-    ('z', 444), ('{', 480), ('|', 200), ('}', 480), ('~', 541), ('\u{00A1}', 333),
-    ('\u{00A2}', 500), ('\u{00A3}', 500), ('\u{00A4}', 500), ('\u{00A5}', 500), ('\u{00A6}', 200), ('\u{00A7}', 500),
-    ('\u{00A8}', 333), ('\u{00A9}', 760), ('\u{00AA}', 276), ('\u{00AB}', 500), ('\u{00AC}', 564), ('\u{00AE}', 760),
-    ('\u{00AF}', 333), ('\u{00B0}', 400), ('\u{00B1}', 564), ('\u{00B2}', 300), ('\u{00B3}', 300), ('\u{00B4}', 333),
-    ('\u{00B5}', 500), ('\u{00B6}', 453), ('\u{00B7}', 250), ('\u{00B8}', 333), ('\u{00B9}', 300), ('\u{00BA}', 310),
-    ('\u{00BB}', 500), ('\u{00BC}', 750), ('\u{00BD}', 750), ('\u{00BE}', 750), ('\u{00BF}', 444), ('\u{00C0}', 722),
-    ('\u{00C1}', 722), ('\u{00C2}', 722), ('\u{00C3}', 722), ('\u{00C4}', 722), ('\u{00C5}', 722), ('\u{00C6}', 889),
-    ('\u{00C7}', 667), ('\u{00C8}', 611), ('\u{00C9}', 611), ('\u{00CA}', 611), ('\u{00CB}', 611), ('\u{00CC}', 333),
-    ('\u{00CD}', 333), ('\u{00CE}', 333), ('\u{00CF}', 333), ('\u{00D0}', 722), ('\u{00D1}', 722), ('\u{00D2}', 722),
-    ('\u{00D3}', 722), ('\u{00D4}', 722), ('\u{00D5}', 722), ('\u{00D6}', 722), ('\u{00D7}', 564), ('\u{00D8}', 722),
-    ('\u{00D9}', 722), ('\u{00DA}', 722), ('\u{00DB}', 722), ('\u{00DC}', 722), ('\u{00DD}', 722), ('\u{00DE}', 556),
-    ('\u{00DF}', 500), ('\u{00E0}', 444), ('\u{00E1}', 444), ('\u{00E2}', 444), ('\u{00E3}', 444), ('\u{00E4}', 444),
-    ('\u{00E5}', 444), ('\u{00E6}', 667), ('\u{00E7}', 444), ('\u{00E8}', 444), ('\u{00E9}', 444), ('\u{00EA}', 444),
-    ('\u{00EB}', 444), ('\u{00EC}', 278), ('\u{00ED}', 278), ('\u{00EE}', 278), ('\u{00EF}', 278), ('\u{00F0}', 500),
-    ('\u{00F1}', 500), ('\u{00F2}', 500), ('\u{00F3}', 500), ('\u{00F4}', 500), ('\u{00F5}', 500), ('\u{00F6}', 500),
-    ('\u{00F7}', 564), ('\u{00F8}', 500), ('\u{00F9}', 500), ('\u{00FA}', 500), ('\u{00FB}', 500), ('\u{00FC}', 500),
-    ('\u{00FD}', 500), ('\u{00FE}', 500), ('\u{00FF}', 500), ('\u{0131}', 278), ('\u{0141}', 611), ('\u{0142}', 278),
-    ('\u{0152}', 889), ('\u{0153}', 722), ('\u{0160}', 556), ('\u{0161}', 389), ('\u{0178}', 722), ('\u{017D}', 611),
-    ('\u{017E}', 444), ('\u{0192}', 500), ('\u{02C6}', 333), ('\u{02C7}', 333), ('\u{02D8}', 333), ('\u{02D9}', 333),
-    ('\u{02DA}', 333), ('\u{02DB}', 333), ('\u{02DC}', 333), ('\u{02DD}', 333), ('\u{2013}', 500), ('\u{2014}', 1000),
-    ('\u{2018}', 333), ('\u{2019}', 333), ('\u{201A}', 333), ('\u{201C}', 444), ('\u{201D}', 444), ('\u{201E}', 444),
-    ('\u{2020}', 500), ('\u{2021}', 500), ('\u{2022}', 350), ('\u{2026}', 1000), ('\u{2030}', 1000), ('\u{2039}', 333),
-    ('\u{203A}', 333), ('\u{2044}', 167), ('\u{20AC}', 500), ('\u{2122}', 980), ('\u{2212}', 564), ('\u{FB01}', 556),
-    ('\u{FB02}', 556),
-];
-
-#[rustfmt::skip]
-static TIMES_BOLD: &[(char, u16)] = &[
-    (' ', 250), ('!', 333), ('"', 555), ('#', 500), ('$', 500), ('%', 1000),
-    ('&', 833), ('\'', 278), ('(', 333), (')', 333), ('*', 500), ('+', 570),
-    (',', 250), ('-', 333), ('.', 250), ('/', 278), ('0', 500), ('1', 500),
-    ('2', 500), ('3', 500), ('4', 500), ('5', 500), ('6', 500), ('7', 500),
-    ('8', 500), ('9', 500), (':', 333), (';', 333), ('<', 570), ('=', 570),
-    ('>', 570), ('?', 500), ('@', 930), ('A', 722), ('B', 667), ('C', 722),
-    ('D', 722), ('E', 667), ('F', 611), ('G', 778), ('H', 778), ('I', 389),
-    ('J', 500), ('K', 778), ('L', 667), ('M', 944), ('N', 722), ('O', 778),
-    ('P', 611), ('Q', 778), ('R', 722), ('S', 556), ('T', 667), ('U', 722),
-    ('V', 722), ('W', 1000), ('X', 722), ('Y', 722), ('Z', 667), ('[', 333),
-    ('\\', 278), (']', 333), ('^', 581), ('_', 500), ('`', 333), ('a', 500),
-    ('b', 556), ('c', 444), ('d', 556), ('e', 444), ('f', 333), ('g', 500),
-    ('h', 556), ('i', 278), ('j', 333), ('k', 556), ('l', 278), ('m', 833),
-    ('n', 556), ('o', 500), ('p', 556), ('q', 556), ('r', 444), ('s', 389),
-    ('t', 333), ('u', 556), ('v', 500), ('w', 722), ('x', 500), ('y', 500),
-    ('z', 444), ('{', 394), ('|', 220), ('}', 394), ('~', 520), ('\u{00A1}', 333),
-    ('\u{00A2}', 500), ('\u{00A3}', 500), ('\u{00A4}', 500), ('\u{00A5}', 500), ('\u{00A6}', 220), ('\u{00A7}', 500),
-    ('\u{00A8}', 333), ('\u{00A9}', 747), ('\u{00AA}', 300), ('\u{00AB}', 500), ('\u{00AC}', 570), ('\u{00AE}', 747),
-    ('\u{00AF}', 333), ('\u{00B0}', 400), ('\u{00B1}', 570), ('\u{00B2}', 300), ('\u{00B3}', 300), ('\u{00B4}', 333),
-    ('\u{00B5}', 556), ('\u{00B6}', 540), ('\u{00B7}', 250), ('\u{00B8}', 333), ('\u{00B9}', 300), ('\u{00BA}', 330),
-    ('\u{00BB}', 500), ('\u{00BC}', 750), ('\u{00BD}', 750), ('\u{00BE}', 750), ('\u{00BF}', 500), ('\u{00C0}', 722),
-    ('\u{00C1}', 722), ('\u{00C2}', 722), ('\u{00C3}', 722), ('\u{00C4}', 722), ('\u{00C5}', 722), ('\u{00C6}', 1000),
-    ('\u{00C7}', 722), ('\u{00C8}', 667), ('\u{00C9}', 667), ('\u{00CA}', 667), ('\u{00CB}', 667), ('\u{00CC}', 389),
-    ('\u{00CD}', 389), ('\u{00CE}', 389), ('\u{00CF}', 389), ('\u{00D0}', 722), ('\u{00D1}', 722), ('\u{00D2}', 778),
-    ('\u{00D3}', 778), ('\u{00D4}', 778), ('\u{00D5}', 778), ('\u{00D6}', 778), ('\u{00D7}', 570), ('\u{00D8}', 778),
-    ('\u{00D9}', 722), ('\u{00DA}', 722), ('\u{00DB}', 722), ('\u{00DC}', 722), ('\u{00DD}', 722), ('\u{00DE}', 611),
-    ('\u{00DF}', 556), ('\u{00E0}', 500), ('\u{00E1}', 500), ('\u{00E2}', 500), ('\u{00E3}', 500), ('\u{00E4}', 500),
-    ('\u{00E5}', 500), ('\u{00E6}', 722), ('\u{00E7}', 444), ('\u{00E8}', 444), ('\u{00E9}', 444), ('\u{00EA}', 444),
-    ('\u{00EB}', 444), ('\u{00EC}', 278), ('\u{00ED}', 278), ('\u{00EE}', 278), ('\u{00EF}', 278), ('\u{00F0}', 500),
-    ('\u{00F1}', 556), ('\u{00F2}', 500), ('\u{00F3}', 500), ('\u{00F4}', 500), ('\u{00F5}', 500), ('\u{00F6}', 500),
-    ('\u{00F7}', 570), ('\u{00F8}', 500), ('\u{00F9}', 556), ('\u{00FA}', 556), ('\u{00FB}', 556), ('\u{00FC}', 556),
-    ('\u{00FD}', 500), ('\u{00FE}', 556), ('\u{00FF}', 500), ('\u{0131}', 278), ('\u{0141}', 667), ('\u{0142}', 278),
-    ('\u{0152}', 1000), ('\u{0153}', 722), ('\u{0160}', 556), ('\u{0161}', 389), ('\u{0178}', 722), ('\u{017D}', 667),
-    ('\u{017E}', 444), ('\u{0192}', 500), ('\u{02C6}', 333), ('\u{02C7}', 333), ('\u{02D8}', 333), ('\u{02D9}', 333),
-    ('\u{02DA}', 333), ('\u{02DB}', 333), ('\u{02DC}', 333), ('\u{02DD}', 333), ('\u{2013}', 500), ('\u{2014}', 1000),
-    ('\u{2018}', 333), ('\u{2019}', 333), ('\u{201A}', 333), ('\u{201C}', 500), ('\u{201D}', 500), ('\u{201E}', 500),
-    ('\u{2020}', 500), ('\u{2021}', 500), ('\u{2022}', 350), ('\u{2026}', 1000), ('\u{2030}', 1000), ('\u{2039}', 333),
-    ('\u{203A}', 333), ('\u{2044}', 167), ('\u{20AC}', 500), ('\u{2122}', 1000), ('\u{2212}', 570), ('\u{FB01}', 556),
-    ('\u{FB02}', 556),
-];
-
-#[rustfmt::skip]
-static TIMES_ITALIC: &[(char, u16)] = &[
-    (' ', 250), ('!', 333), ('"', 420), ('#', 500), ('$', 500), ('%', 833),
-    ('&', 778), ('\'', 214), ('(', 333), (')', 333), ('*', 500), ('+', 675),
-    (',', 250), ('-', 333), ('.', 250), ('/', 278), ('0', 500), ('1', 500),
-    ('2', 500), ('3', 500), ('4', 500), ('5', 500), ('6', 500), ('7', 500),
-    ('8', 500), ('9', 500), (':', 333), (';', 333), ('<', 675), ('=', 675),
-    ('>', 675), ('?', 500), ('@', 920), ('A', 611), ('B', 611), ('C', 667),
-    ('D', 722), ('E', 611), ('F', 611), ('G', 722), ('H', 722), ('I', 333),
-    ('J', 444), ('K', 667), ('L', 556), ('M', 833), ('N', 667), ('O', 722),
-    ('P', 611), ('Q', 722), ('R', 611), ('S', 500), ('T', 556), ('U', 722),
-    ('V', 611), ('W', 833), ('X', 611), ('Y', 556), ('Z', 556), ('[', 389),
-    ('\\', 278), (']', 389), ('^', 422), ('_', 500), ('`', 333), ('a', 500),
-    ('b', 500), ('c', 444), ('d', 500), ('e', 444), ('f', 278), ('g', 500),
-    ('h', 500), ('i', 278), ('j', 278), ('k', 444), ('l', 278), ('m', 722),
-    ('n', 500), ('o', 500), ('p', 500), ('q', 500), ('r', 389), ('s', 389),
-    ('t', 278), ('u', 500), ('v', 444), ('w', 667), ('x', 444), ('y', 444),
-    ('z', 389), ('{', 400), ('|', 275), ('}', 400), ('~', 541), ('\u{00A1}', 389),
-    ('\u{00A2}', 500), ('\u{00A3}', 500), ('\u{00A4}', 500), ('\u{00A5}', 500), ('\u{00A6}', 275), ('\u{00A7}', 500),
-    ('\u{00A8}', 333), ('\u{00A9}', 760), ('\u{00AA}', 276), ('\u{00AB}', 500), ('\u{00AC}', 675), ('\u{00AE}', 760),
-    ('\u{00AF}', 333), ('\u{00B0}', 400), ('\u{00B1}', 675), ('\u{00B2}', 300), ('\u{00B3}', 300), ('\u{00B4}', 333),
-    ('\u{00B5}', 500), ('\u{00B6}', 523), ('\u{00B7}', 250), ('\u{00B8}', 333), ('\u{00B9}', 300), ('\u{00BA}', 310),
-    ('\u{00BB}', 500), ('\u{00BC}', 750), ('\u{00BD}', 750), ('\u{00BE}', 750), ('\u{00BF}', 500), ('\u{00C0}', 611),
-    ('\u{00C1}', 611), ('\u{00C2}', 611), ('\u{00C3}', 611), ('\u{00C4}', 611), ('\u{00C5}', 611), ('\u{00C6}', 889),
-    ('\u{00C7}', 667), ('\u{00C8}', 611), ('\u{00C9}', 611), ('\u{00CA}', 611), ('\u{00CB}', 611), ('\u{00CC}', 333),
-    ('\u{00CD}', 333), ('\u{00CE}', 333), ('\u{00CF}', 333), ('\u{00D0}', 722), ('\u{00D1}', 667), ('\u{00D2}', 722),
-    ('\u{00D3}', 722), ('\u{00D4}', 722), ('\u{00D5}', 722), ('\u{00D6}', 722), ('\u{00D7}', 675), ('\u{00D8}', 722),
-    ('\u{00D9}', 722), ('\u{00DA}', 722), ('\u{00DB}', 722), ('\u{00DC}', 722), ('\u{00DD}', 556), ('\u{00DE}', 611),
-    ('\u{00DF}', 500), ('\u{00E0}', 500), ('\u{00E1}', 500), ('\u{00E2}', 500), ('\u{00E3}', 500), ('\u{00E4}', 500),
-    ('\u{00E5}', 500), ('\u{00E6}', 667), ('\u{00E7}', 444), ('\u{00E8}', 444), ('\u{00E9}', 444), ('\u{00EA}', 444),
-    ('\u{00EB}', 444), ('\u{00EC}', 278), ('\u{00ED}', 278), ('\u{00EE}', 278), ('\u{00EF}', 278), ('\u{00F0}', 500),
-    ('\u{00F1}', 500), ('\u{00F2}', 500), ('\u{00F3}', 500), ('\u{00F4}', 500), ('\u{00F5}', 500), ('\u{00F6}', 500),
-    ('\u{00F7}', 675), ('\u{00F8}', 500), ('\u{00F9}', 500), ('\u{00FA}', 500), ('\u{00FB}', 500), ('\u{00FC}', 500),
-    ('\u{00FD}', 444), ('\u{00FE}', 500), ('\u{00FF}', 444), ('\u{0131}', 278), ('\u{0141}', 556), ('\u{0142}', 278),
-    ('\u{0152}', 944), ('\u{0153}', 667), ('\u{0160}', 500), ('\u{0161}', 389), ('\u{0178}', 556), ('\u{017D}', 556),
-    ('\u{017E}', 389), ('\u{0192}', 500), ('\u{02C6}', 333), ('\u{02C7}', 333), ('\u{02D8}', 333), ('\u{02D9}', 333),
-    ('\u{02DA}', 333), ('\u{02DB}', 333), ('\u{02DC}', 333), ('\u{02DD}', 333), ('\u{2013}', 500), ('\u{2014}', 889),
-    ('\u{2018}', 333), ('\u{2019}', 333), ('\u{201A}', 333), ('\u{201C}', 556), ('\u{201D}', 556), ('\u{201E}', 556),
-    ('\u{2020}', 500), ('\u{2021}', 500), ('\u{2022}', 350), ('\u{2026}', 889), ('\u{2030}', 1000), ('\u{2039}', 333),
-    ('\u{203A}', 333), ('\u{2044}', 167), ('\u{20AC}', 500), ('\u{2122}', 980), ('\u{2212}', 675), ('\u{FB01}', 500),
-    ('\u{FB02}', 500),
-];
-
-#[rustfmt::skip]
-static TIMES_BOLDITALIC: &[(char, u16)] = &[
-    (' ', 250), ('!', 389), ('"', 555), ('#', 500), ('$', 500), ('%', 833),
-    ('&', 778), ('\'', 278), ('(', 333), (')', 333), ('*', 500), ('+', 570),
-    (',', 250), ('-', 333), ('.', 250), ('/', 278), ('0', 500), ('1', 500),
-    ('2', 500), ('3', 500), ('4', 500), ('5', 500), ('6', 500), ('7', 500),
-    ('8', 500), ('9', 500), (':', 333), (';', 333), ('<', 570), ('=', 570),
-    ('>', 570), ('?', 500), ('@', 832), ('A', 667), ('B', 667), ('C', 667),
-    ('D', 722), ('E', 667), ('F', 667), ('G', 722), ('H', 778), ('I', 389),
-    ('J', 500), ('K', 667), ('L', 611), ('M', 889), ('N', 722), ('O', 722),
-    ('P', 611), ('Q', 722), ('R', 667), ('S', 556), ('T', 611), ('U', 722),
-    ('V', 667), ('W', 889), ('X', 667), ('Y', 611), ('Z', 611), ('[', 333),
-    ('\\', 278), (']', 333), ('^', 570), ('_', 500), ('`', 333), ('a', 500),
-    ('b', 500), ('c', 444), ('d', 500), ('e', 444), ('f', 333), ('g', 500),
-    ('h', 556), ('i', 278), ('j', 278), ('k', 500), ('l', 278), ('m', 778),
-    ('n', 556), ('o', 500), ('p', 500), ('q', 500), ('r', 389), ('s', 389),
-    ('t', 278), ('u', 556), ('v', 444), ('w', 667), ('x', 500), ('y', 444),
-    ('z', 389), ('{', 348), ('|', 220), ('}', 348), ('~', 570), ('\u{00A1}', 389),
-    ('\u{00A2}', 500), ('\u{00A3}', 500), ('\u{00A4}', 500), ('\u{00A5}', 500), ('\u{00A6}', 220), ('\u{00A7}', 500),
-    ('\u{00A8}', 333), ('\u{00A9}', 747), ('\u{00AA}', 266), ('\u{00AB}', 500), ('\u{00AC}', 606), ('\u{00AE}', 747),
-    ('\u{00AF}', 333), ('\u{00B0}', 400), ('\u{00B1}', 570), ('\u{00B2}', 300), ('\u{00B3}', 300), ('\u{00B4}', 333),
-    ('\u{00B5}', 576), ('\u{00B6}', 500), ('\u{00B7}', 250), ('\u{00B8}', 333), ('\u{00B9}', 300), ('\u{00BA}', 300),
-    ('\u{00BB}', 500), ('\u{00BC}', 750), ('\u{00BD}', 750), ('\u{00BE}', 750), ('\u{00BF}', 500), ('\u{00C0}', 667),
-    ('\u{00C1}', 667), ('\u{00C2}', 667), ('\u{00C3}', 667), ('\u{00C4}', 667), ('\u{00C5}', 667), ('\u{00C6}', 944),
-    ('\u{00C7}', 667), ('\u{00C8}', 667), ('\u{00C9}', 667), ('\u{00CA}', 667), ('\u{00CB}', 667), ('\u{00CC}', 389),
-    ('\u{00CD}', 389), ('\u{00CE}', 389), ('\u{00CF}', 389), ('\u{00D0}', 722), ('\u{00D1}', 722), ('\u{00D2}', 722),
-    ('\u{00D3}', 722), ('\u{00D4}', 722), ('\u{00D5}', 722), ('\u{00D6}', 722), ('\u{00D7}', 570), ('\u{00D8}', 722),
-    ('\u{00D9}', 722), ('\u{00DA}', 722), ('\u{00DB}', 722), ('\u{00DC}', 722), ('\u{00DD}', 611), ('\u{00DE}', 611),
-    ('\u{00DF}', 500), ('\u{00E0}', 500), ('\u{00E1}', 500), ('\u{00E2}', 500), ('\u{00E3}', 500), ('\u{00E4}', 500),
-    ('\u{00E5}', 500), ('\u{00E6}', 722), ('\u{00E7}', 444), ('\u{00E8}', 444), ('\u{00E9}', 444), ('\u{00EA}', 444),
-    ('\u{00EB}', 444), ('\u{00EC}', 278), ('\u{00ED}', 278), ('\u{00EE}', 278), ('\u{00EF}', 278), ('\u{00F0}', 500),
-    ('\u{00F1}', 556), ('\u{00F2}', 500), ('\u{00F3}', 500), ('\u{00F4}', 500), ('\u{00F5}', 500), ('\u{00F6}', 500),
-    ('\u{00F7}', 570), ('\u{00F8}', 500), ('\u{00F9}', 556), ('\u{00FA}', 556), ('\u{00FB}', 556), ('\u{00FC}', 556),
-    ('\u{00FD}', 444), ('\u{00FE}', 500), ('\u{00FF}', 444), ('\u{0131}', 278), ('\u{0141}', 611), ('\u{0142}', 278),
-    ('\u{0152}', 944), ('\u{0153}', 722), ('\u{0160}', 556), ('\u{0161}', 389), ('\u{0178}', 611), ('\u{017D}', 611),
-    ('\u{017E}', 389), ('\u{0192}', 500), ('\u{02C6}', 333), ('\u{02C7}', 333), ('\u{02D8}', 333), ('\u{02D9}', 333),
-    ('\u{02DA}', 333), ('\u{02DB}', 333), ('\u{02DC}', 333), ('\u{02DD}', 333), ('\u{2013}', 500), ('\u{2014}', 1000),
-    ('\u{2018}', 333), ('\u{2019}', 333), ('\u{201A}', 333), ('\u{201C}', 500), ('\u{201D}', 500), ('\u{201E}', 500),
-    ('\u{2020}', 500), ('\u{2021}', 500), ('\u{2022}', 350), ('\u{2026}', 1000), ('\u{2030}', 1000), ('\u{2039}', 333),
-    ('\u{203A}', 333), ('\u{2044}', 167), ('\u{20AC}', 500), ('\u{2122}', 1000), ('\u{2212}', 606), ('\u{FB01}', 556),
-    ('\u{FB02}', 556),
-];
-
-#[rustfmt::skip]
-static SYMBOL: &[(char, u16)] = &[
-    (' ', 250), ('!', 333), ('#', 500), ('%', 833), ('&', 778), ('(', 333),
-    (')', 333), ('+', 549), (',', 250), ('.', 250), ('/', 278), ('0', 500),
-    ('1', 500), ('2', 500), ('3', 500), ('4', 500), ('5', 500), ('6', 500),
-    ('7', 500), ('8', 500), ('9', 500), (':', 278), (';', 278), ('<', 549),
-    ('=', 549), ('>', 549), ('?', 444), ('[', 333), (']', 333), ('_', 500),
-    ('{', 480), ('|', 200), ('}', 480), ('\u{00AC}', 713), ('\u{00B0}', 400), ('\u{00B1}', 549),
-    ('\u{00B5}', 576), ('\u{00D7}', 549), ('\u{00F7}', 549), ('\u{0192}', 500), ('\u{0391}', 722), ('\u{0392}', 667),
-    ('\u{0393}', 603), ('\u{0395}', 611), ('\u{0396}', 611), ('\u{0397}', 722), ('\u{0398}', 741), ('\u{0399}', 333),
-    ('\u{039A}', 722), ('\u{039B}', 686), ('\u{039C}', 889), ('\u{039D}', 722), ('\u{039E}', 645), ('\u{039F}', 722),
-    ('\u{03A0}', 768), ('\u{03A1}', 556), ('\u{03A3}', 592), ('\u{03A4}', 611), ('\u{03A5}', 690), ('\u{03A6}', 763),
-    ('\u{03A7}', 722), ('\u{03A8}', 795), ('\u{03B1}', 631), ('\u{03B2}', 549), ('\u{03B3}', 411), ('\u{03B4}', 494),
-    ('\u{03B5}', 439), ('\u{03B6}', 494), ('\u{03B7}', 603), ('\u{03B8}', 521), ('\u{03B9}', 329), ('\u{03BA}', 549),
-    ('\u{03BB}', 549), ('\u{03BD}', 521), ('\u{03BE}', 493), ('\u{03BF}', 549), ('\u{03C0}', 549), ('\u{03C1}', 549),
-    ('\u{03C2}', 439), ('\u{03C3}', 603), ('\u{03C4}', 439), ('\u{03C5}', 576), ('\u{03C6}', 521), ('\u{03C7}', 549),
-    ('\u{03C8}', 686), ('\u{03C9}', 686), ('\u{03D1}', 631), ('\u{03D2}', 620), ('\u{03D5}', 603), ('\u{03D6}', 713),
-    ('\u{2022}', 460), ('\u{2026}', 1000), ('\u{2032}', 247), ('\u{2033}', 411), ('\u{2044}', 167), ('\u{20AC}', 750),
-    ('\u{2111}', 686), ('\u{2118}', 987), ('\u{211C}', 795), ('\u{2126}', 768), ('\u{2135}', 823), ('\u{2190}', 987),
-    ('\u{2191}', 603), ('\u{2192}', 987), ('\u{2193}', 603), ('\u{2194}', 1042), ('\u{21B5}', 658), ('\u{21D0}', 987),
-    ('\u{21D1}', 603), ('\u{21D2}', 987), ('\u{21D3}', 603), ('\u{21D4}', 1042), ('\u{2200}', 713), ('\u{2202}', 494),
-    ('\u{2203}', 549), ('\u{2205}', 823), ('\u{2206}', 612), ('\u{2207}', 713), ('\u{2208}', 713), ('\u{2209}', 713),
-    ('\u{220B}', 439), ('\u{220F}', 823), ('\u{2211}', 713), ('\u{2212}', 549), ('\u{2217}', 500), ('\u{221A}', 549),
-    ('\u{221D}', 713), ('\u{221E}', 713), ('\u{2220}', 768), ('\u{2227}', 603), ('\u{2228}', 603), ('\u{2229}', 768),
-    ('\u{222A}', 768), ('\u{222B}', 274), ('\u{2234}', 863), ('\u{223C}', 549), ('\u{2245}', 549), ('\u{2248}', 549),
-    ('\u{2260}', 549), ('\u{2261}', 549), ('\u{2264}', 549), ('\u{2265}', 549), ('\u{2282}', 713), ('\u{2283}', 713),
-    ('\u{2284}', 713), ('\u{2286}', 713), ('\u{2287}', 713), ('\u{2295}', 768), ('\u{2297}', 768), ('\u{22A5}', 658),
-    ('\u{22C5}', 250), ('\u{2320}', 686), ('\u{2321}', 686), ('\u{2329}', 329), ('\u{232A}', 329), ('\u{25CA}', 494),
-    ('\u{2660}', 753), ('\u{2663}', 753), ('\u{2665}', 753), ('\u{2666}', 753), ('\u{F6D9}', 790), ('\u{F6DA}', 790),
-    ('\u{F6DB}', 890), ('\u{F8E5}', 500), ('\u{F8E6}', 603), ('\u{F8E7}', 1000), ('\u{F8E8}', 790), ('\u{F8E9}', 790),
-    ('\u{F8EA}', 786), ('\u{F8EB}', 384), ('\u{F8EC}', 384), ('\u{F8ED}', 384), ('\u{F8EE}', 384), ('\u{F8EF}', 384),
-    ('\u{F8F0}', 384), ('\u{F8F1}', 494), ('\u{F8F2}', 494), ('\u{F8F3}', 494), ('\u{F8F4}', 494), ('\u{F8F5}', 686),
-    ('\u{F8F6}', 384), ('\u{F8F7}', 384), ('\u{F8F8}', 384), ('\u{F8F9}', 384), ('\u{F8FA}', 384), ('\u{F8FB}', 384),
-    ('\u{F8FC}', 494), ('\u{F8FD}', 494), ('\u{F8FE}', 494), ('\u{F8FF}', 790),
-];
-
-#[rustfmt::skip]
-static ZAPFDINGBATS: &[(char, u16)] = &[
-    (' ', 278), ('\u{2192}', 838), ('\u{2194}', 1016), ('\u{2195}', 458), ('\u{2460}', 788), ('\u{2461}', 788),
-    ('\u{2462}', 788), ('\u{2463}', 788), ('\u{2464}', 788), ('\u{2465}', 788), ('\u{2466}', 788), ('\u{2467}', 788),
-    ('\u{2468}', 788), ('\u{2469}', 788), ('\u{25A0}', 761), ('\u{25B2}', 892), ('\u{25BC}', 892), ('\u{25C6}', 788),
-    ('\u{25CF}', 791), ('\u{25D7}', 438), ('\u{2605}', 816), ('\u{260E}', 719), ('\u{261B}', 960), ('\u{261E}', 939),
-    ('\u{2660}', 626), ('\u{2663}', 776), ('\u{2665}', 694), ('\u{2666}', 595), ('\u{2701}', 974), ('\u{2702}', 961),
-    ('\u{2703}', 974), ('\u{2704}', 980), ('\u{2706}', 789), ('\u{2707}', 790), ('\u{2708}', 791), ('\u{2709}', 690),
-    ('\u{270C}', 549), ('\u{270D}', 855), ('\u{270E}', 911), ('\u{270F}', 933), ('\u{2710}', 911), ('\u{2711}', 945),
-    ('\u{2712}', 974), ('\u{2713}', 755), ('\u{2714}', 846), ('\u{2715}', 762), ('\u{2716}', 761), ('\u{2717}', 571),
-    ('\u{2718}', 677), ('\u{2719}', 763), ('\u{271A}', 760), ('\u{271B}', 759), ('\u{271C}', 754), ('\u{271D}', 494),
-    ('\u{271E}', 552), ('\u{271F}', 537), ('\u{2720}', 577), ('\u{2721}', 692), ('\u{2722}', 786), ('\u{2723}', 788),
-    ('\u{2724}', 788), ('\u{2725}', 790), ('\u{2726}', 793), ('\u{2727}', 794), ('\u{2729}', 823), ('\u{272A}', 789),
-    ('\u{272B}', 841), ('\u{272C}', 823), ('\u{272D}', 833), ('\u{272E}', 816), ('\u{272F}', 831), ('\u{2730}', 923),
-    ('\u{2731}', 744), ('\u{2732}', 723), ('\u{2733}', 749), ('\u{2734}', 790), ('\u{2735}', 792), ('\u{2736}', 695),
-    ('\u{2737}', 776), ('\u{2738}', 768), ('\u{2739}', 792), ('\u{273A}', 759), ('\u{273B}', 707), ('\u{273C}', 708),
-    ('\u{273D}', 682), ('\u{273E}', 701), ('\u{273F}', 826), ('\u{2740}', 815), ('\u{2741}', 789), ('\u{2742}', 789),
-    ('\u{2743}', 707), ('\u{2744}', 687), ('\u{2745}', 696), ('\u{2746}', 689), ('\u{2747}', 786), ('\u{2748}', 787),
-    ('\u{2749}', 713), ('\u{274A}', 791), ('\u{274B}', 785), ('\u{274D}', 873), ('\u{274F}', 762), ('\u{2750}', 762),
-    ('\u{2751}', 759), ('\u{2752}', 759), ('\u{2756}', 784), ('\u{2758}', 138), ('\u{2759}', 277), ('\u{275A}', 415),
-    ('\u{275B}', 392), ('\u{275C}', 392), ('\u{275D}', 668), ('\u{275E}', 668), ('\u{2761}', 732), ('\u{2762}', 544),
-    ('\u{2763}', 544), ('\u{2764}', 910), ('\u{2765}', 667), ('\u{2766}', 760), ('\u{2767}', 760), ('\u{2768}', 390),
-    ('\u{2769}', 390), ('\u{276A}', 317), ('\u{276B}', 317), ('\u{276C}', 276), ('\u{276D}', 276), ('\u{276E}', 509),
-    ('\u{276F}', 509), ('\u{2770}', 410), ('\u{2771}', 410), ('\u{2772}', 234), ('\u{2773}', 234), ('\u{2774}', 334),
-    ('\u{2775}', 334), ('\u{2776}', 788), ('\u{2777}', 788), ('\u{2778}', 788), ('\u{2779}', 788), ('\u{277A}', 788),
-    ('\u{277B}', 788), ('\u{277C}', 788), ('\u{277D}', 788), ('\u{277E}', 788), ('\u{277F}', 788), ('\u{2780}', 788),
-    ('\u{2781}', 788), ('\u{2782}', 788), ('\u{2783}', 788), ('\u{2784}', 788), ('\u{2785}', 788), ('\u{2786}', 788),
-    ('\u{2787}', 788), ('\u{2788}', 788), ('\u{2789}', 788), ('\u{278A}', 788), ('\u{278B}', 788), ('\u{278C}', 788),
-    ('\u{278D}', 788), ('\u{278E}', 788), ('\u{278F}', 788), ('\u{2790}', 788), ('\u{2791}', 788), ('\u{2792}', 788),
-    ('\u{2793}', 788), ('\u{2794}', 894), ('\u{2798}', 748), ('\u{2799}', 924), ('\u{279A}', 748), ('\u{279B}', 918),
-    ('\u{279C}', 927), ('\u{279D}', 928), ('\u{279E}', 928), ('\u{279F}', 834), ('\u{27A0}', 873), ('\u{27A1}', 828),
-    ('\u{27A2}', 924), ('\u{27A3}', 924), ('\u{27A4}', 917), ('\u{27A5}', 930), ('\u{27A6}', 931), ('\u{27A7}', 463),
-    ('\u{27A8}', 883), ('\u{27A9}', 836), ('\u{27AA}', 836), ('\u{27AB}', 867), ('\u{27AC}', 867), ('\u{27AD}', 696),
-    ('\u{27AE}', 696), ('\u{27AF}', 874), ('\u{27B1}', 874), ('\u{27B2}', 760), ('\u{27B3}', 946), ('\u{27B4}', 771),
-    ('\u{27B5}', 865), ('\u{27B6}', 771), ('\u{27B7}', 888), ('\u{27B8}', 967), ('\u{27B9}', 888), ('\u{27BA}', 831),
-    ('\u{27BB}', 873), ('\u{27BC}', 927), ('\u{27BD}', 970), ('\u{27BE}', 918),
-];
-
-#[rustfmt::skip]
-static SYMBOL_ENCODING: &[(u8, char)] = &[
-    (0x20, ' '), (0x21, '!'), (0x22, '\u{2200}'), (0x23, '#'), (0x24, '\u{2203}'), (0x25, '%'),
-    (0x26, '&'), (0x27, '\u{220B}'), (0x28, '('), (0x29, ')'), (0x2A, '\u{2217}'), (0x2B, '+'),
-    (0x2C, ','), (0x2D, '\u{2212}'), (0x2E, '.'), (0x2F, '/'), (0x30, '0'), (0x31, '1'),
-    (0x32, '2'), (0x33, '3'), (0x34, '4'), (0x35, '5'), (0x36, '6'), (0x37, '7'),
-    (0x38, '8'), (0x39, '9'), (0x3A, ':'), (0x3B, ';'), (0x3C, '<'), (0x3D, '='),
-    (0x3E, '>'), (0x3F, '?'), (0x40, '\u{2245}'), (0x41, '\u{0391}'), (0x42, '\u{0392}'), (0x43, '\u{03A7}'),
-    (0x44, '\u{2206}'), (0x45, '\u{0395}'), (0x46, '\u{03A6}'), (0x47, '\u{0393}'), (0x48, '\u{0397}'), (0x49, '\u{0399}'),
-    (0x4A, '\u{03D1}'), (0x4B, '\u{039A}'), (0x4C, '\u{039B}'), (0x4D, '\u{039C}'), (0x4E, '\u{039D}'), (0x4F, '\u{039F}'),
-    (0x50, '\u{03A0}'), (0x51, '\u{0398}'), (0x52, '\u{03A1}'), (0x53, '\u{03A3}'), (0x54, '\u{03A4}'), (0x55, '\u{03A5}'),
-    (0x56, '\u{03C2}'), (0x57, '\u{2126}'), (0x58, '\u{039E}'), (0x59, '\u{03A8}'), (0x5A, '\u{0396}'), (0x5B, '['),
-    (0x5C, '\u{2234}'), (0x5D, ']'), (0x5E, '\u{22A5}'), (0x5F, '_'), (0x60, '\u{F8E5}'), (0x61, '\u{03B1}'),
-    (0x62, '\u{03B2}'), (0x63, '\u{03C7}'), (0x64, '\u{03B4}'), (0x65, '\u{03B5}'), (0x66, '\u{03C6}'), (0x67, '\u{03B3}'),
-    (0x68, '\u{03B7}'), (0x69, '\u{03B9}'), (0x6A, '\u{03D5}'), (0x6B, '\u{03BA}'), (0x6C, '\u{03BB}'), (0x6D, '\u{00B5}'),
-    (0x6E, '\u{03BD}'), (0x6F, '\u{03BF}'), (0x70, '\u{03C0}'), (0x71, '\u{03B8}'), (0x72, '\u{03C1}'), (0x73, '\u{03C3}'),
-    (0x74, '\u{03C4}'), (0x75, '\u{03C5}'), (0x76, '\u{03D6}'), (0x77, '\u{03C9}'), (0x78, '\u{03BE}'), (0x79, '\u{03C8}'),
-    (0x7A, '\u{03B6}'), (0x7B, '{'), (0x7C, '|'), (0x7D, '}'), (0x7E, '\u{223C}'), (0xA0, '\u{20AC}'),
-    (0xA1, '\u{03D2}'), (0xA2, '\u{2032}'), (0xA3, '\u{2264}'), (0xA4, '\u{2044}'), (0xA5, '\u{221E}'), (0xA6, '\u{0192}'),
-    (0xA7, '\u{2663}'), (0xA8, '\u{2666}'), (0xA9, '\u{2665}'), (0xAA, '\u{2660}'), (0xAB, '\u{2194}'), (0xAC, '\u{2190}'),
-    (0xAD, '\u{2191}'), (0xAE, '\u{2192}'), (0xAF, '\u{2193}'), (0xB0, '\u{00B0}'), (0xB1, '\u{00B1}'), (0xB2, '\u{2033}'),
-    (0xB3, '\u{2265}'), (0xB4, '\u{00D7}'), (0xB5, '\u{221D}'), (0xB6, '\u{2202}'), (0xB7, '\u{2022}'), (0xB8, '\u{00F7}'),
-    (0xB9, '\u{2260}'), (0xBA, '\u{2261}'), (0xBB, '\u{2248}'), (0xBC, '\u{2026}'), (0xBD, '\u{F8E6}'), (0xBE, '\u{F8E7}'),
-    (0xBF, '\u{21B5}'), (0xC0, '\u{2135}'), (0xC1, '\u{2111}'), (0xC2, '\u{211C}'), (0xC3, '\u{2118}'), (0xC4, '\u{2297}'),
-    (0xC5, '\u{2295}'), (0xC6, '\u{2205}'), (0xC7, '\u{2229}'), (0xC8, '\u{222A}'), (0xC9, '\u{2283}'), (0xCA, '\u{2287}'),
-    (0xCB, '\u{2284}'), (0xCC, '\u{2282}'), (0xCD, '\u{2286}'), (0xCE, '\u{2208}'), (0xCF, '\u{2209}'), (0xD0, '\u{2220}'),
-    (0xD1, '\u{2207}'), (0xD2, '\u{F6DA}'), (0xD3, '\u{F6D9}'), (0xD4, '\u{F6DB}'), (0xD5, '\u{220F}'), (0xD6, '\u{221A}'),
-    (0xD7, '\u{22C5}'), (0xD8, '\u{00AC}'), (0xD9, '\u{2227}'), (0xDA, '\u{2228}'), (0xDB, '\u{21D4}'), (0xDC, '\u{21D0}'),
-    (0xDD, '\u{21D1}'), (0xDE, '\u{21D2}'), (0xDF, '\u{21D3}'), (0xE0, '\u{25CA}'), (0xE1, '\u{2329}'), (0xE2, '\u{F8E8}'),
-    (0xE3, '\u{F8E9}'), (0xE4, '\u{F8EA}'), (0xE5, '\u{2211}'), (0xE6, '\u{F8EB}'), (0xE7, '\u{F8EC}'), (0xE8, '\u{F8ED}'),
-    (0xE9, '\u{F8EE}'), (0xEA, '\u{F8EF}'), (0xEB, '\u{F8F0}'), (0xEC, '\u{F8F1}'), (0xED, '\u{F8F2}'), (0xEE, '\u{F8F3}'),
-    (0xEF, '\u{F8F4}'), (0xF1, '\u{232A}'), (0xF2, '\u{222B}'), (0xF3, '\u{2320}'), (0xF4, '\u{F8F5}'), (0xF5, '\u{2321}'),
-    (0xF6, '\u{F8F6}'), (0xF7, '\u{F8F7}'), (0xF8, '\u{F8F8}'), (0xF9, '\u{F8F9}'), (0xFA, '\u{F8FA}'), (0xFB, '\u{F8FB}'),
-    (0xFC, '\u{F8FC}'), (0xFD, '\u{F8FD}'), (0xFE, '\u{F8FE}'),
-];
-
-#[rustfmt::skip]
-static ZAPFDINGBATS_ENCODING: &[(u8, char)] = &[
-    (0x20, ' '), (0x21, '\u{2701}'), (0x22, '\u{2702}'), (0x23, '\u{2703}'), (0x24, '\u{2704}'), (0x25, '\u{260E}'),
-    (0x26, '\u{2706}'), (0x27, '\u{2707}'), (0x28, '\u{2708}'), (0x29, '\u{2709}'), (0x2A, '\u{261B}'), (0x2B, '\u{261E}'),
-    (0x2C, '\u{270C}'), (0x2D, '\u{270D}'), (0x2E, '\u{270E}'), (0x2F, '\u{270F}'), (0x30, '\u{2710}'), (0x31, '\u{2711}'),
-    (0x32, '\u{2712}'), (0x33, '\u{2713}'), (0x34, '\u{2714}'), (0x35, '\u{2715}'), (0x36, '\u{2716}'), (0x37, '\u{2717}'),
-    (0x38, '\u{2718}'), (0x39, '\u{2719}'), (0x3A, '\u{271A}'), (0x3B, '\u{271B}'), (0x3C, '\u{271C}'), (0x3D, '\u{271D}'),
-    (0x3E, '\u{271E}'), (0x3F, '\u{271F}'), (0x40, '\u{2720}'), (0x41, '\u{2721}'), (0x42, '\u{2722}'), (0x43, '\u{2723}'),
-    (0x44, '\u{2724}'), (0x45, '\u{2725}'), (0x46, '\u{2726}'), (0x47, '\u{2727}'), (0x48, '\u{2605}'), (0x49, '\u{2729}'),
-    (0x4A, '\u{272A}'), (0x4B, '\u{272B}'), (0x4C, '\u{272C}'), (0x4D, '\u{272D}'), (0x4E, '\u{272E}'), (0x4F, '\u{272F}'),
-    (0x50, '\u{2730}'), (0x51, '\u{2731}'), (0x52, '\u{2732}'), (0x53, '\u{2733}'), (0x54, '\u{2734}'), (0x55, '\u{2735}'),
-    (0x56, '\u{2736}'), (0x57, '\u{2737}'), (0x58, '\u{2738}'), (0x59, '\u{2739}'), (0x5A, '\u{273A}'), (0x5B, '\u{273B}'),
-    (0x5C, '\u{273C}'), (0x5D, '\u{273D}'), (0x5E, '\u{273E}'), (0x5F, '\u{273F}'), (0x60, '\u{2740}'), (0x61, '\u{2741}'),
-    (0x62, '\u{2742}'), (0x63, '\u{2743}'), (0x64, '\u{2744}'), (0x65, '\u{2745}'), (0x66, '\u{2746}'), (0x67, '\u{2747}'),
-    (0x68, '\u{2748}'), (0x69, '\u{2749}'), (0x6A, '\u{274A}'), (0x6B, '\u{274B}'), (0x6C, '\u{25CF}'), (0x6D, '\u{274D}'),
-    (0x6E, '\u{25A0}'), (0x6F, '\u{274F}'), (0x70, '\u{2750}'), (0x71, '\u{2751}'), (0x72, '\u{2752}'), (0x73, '\u{25B2}'),
-    (0x74, '\u{25BC}'), (0x75, '\u{25C6}'), (0x76, '\u{2756}'), (0x77, '\u{25D7}'), (0x78, '\u{2758}'), (0x79, '\u{2759}'),
-    (0x7A, '\u{275A}'), (0x7B, '\u{275B}'), (0x7C, '\u{275C}'), (0x7D, '\u{275D}'), (0x7E, '\u{275E}'), (0x80, '\u{2768}'),
-    (0x81, '\u{2769}'), (0x82, '\u{276A}'), (0x83, '\u{276B}'), (0x84, '\u{276C}'), (0x85, '\u{276D}'), (0x86, '\u{276E}'),
-    (0x87, '\u{276F}'), (0x88, '\u{2770}'), (0x89, '\u{2771}'), (0x8A, '\u{2772}'), (0x8B, '\u{2773}'), (0x8C, '\u{2774}'),
-    (0x8D, '\u{2775}'), (0xA1, '\u{2761}'), (0xA2, '\u{2762}'), (0xA3, '\u{2763}'), (0xA4, '\u{2764}'), (0xA5, '\u{2765}'),
-    (0xA6, '\u{2766}'), (0xA7, '\u{2767}'), (0xA8, '\u{2663}'), (0xA9, '\u{2666}'), (0xAA, '\u{2665}'), (0xAB, '\u{2660}'),
-    (0xAC, '\u{2460}'), (0xAD, '\u{2461}'), (0xAE, '\u{2462}'), (0xAF, '\u{2463}'), (0xB0, '\u{2464}'), (0xB1, '\u{2465}'),
-    (0xB2, '\u{2466}'), (0xB3, '\u{2467}'), (0xB4, '\u{2468}'), (0xB5, '\u{2469}'), (0xB6, '\u{2776}'), (0xB7, '\u{2777}'),
-    (0xB8, '\u{2778}'), (0xB9, '\u{2779}'), (0xBA, '\u{277A}'), (0xBB, '\u{277B}'), (0xBC, '\u{277C}'), (0xBD, '\u{277D}'),
-    (0xBE, '\u{277E}'), (0xBF, '\u{277F}'), (0xC0, '\u{2780}'), (0xC1, '\u{2781}'), (0xC2, '\u{2782}'), (0xC3, '\u{2783}'),
-    (0xC4, '\u{2784}'), (0xC5, '\u{2785}'), (0xC6, '\u{2786}'), (0xC7, '\u{2787}'), (0xC8, '\u{2788}'), (0xC9, '\u{2789}'),
-    (0xCA, '\u{278A}'), (0xCB, '\u{278B}'), (0xCC, '\u{278C}'), (0xCD, '\u{278D}'), (0xCE, '\u{278E}'), (0xCF, '\u{278F}'),
-    (0xD0, '\u{2790}'), (0xD1, '\u{2791}'), (0xD2, '\u{2792}'), (0xD3, '\u{2793}'), (0xD4, '\u{2794}'), (0xD5, '\u{2192}'),
-    (0xD6, '\u{2194}'), (0xD7, '\u{2195}'), (0xD8, '\u{2798}'), (0xD9, '\u{2799}'), (0xDA, '\u{279A}'), (0xDB, '\u{279B}'),
-    (0xDC, '\u{279C}'), (0xDD, '\u{279D}'), (0xDE, '\u{279E}'), (0xDF, '\u{279F}'), (0xE0, '\u{27A0}'), (0xE1, '\u{27A1}'),
-    (0xE2, '\u{27A2}'), (0xE3, '\u{27A3}'), (0xE4, '\u{27A4}'), (0xE5, '\u{27A5}'), (0xE6, '\u{27A6}'), (0xE7, '\u{27A7}'),
-    (0xE8, '\u{27A8}'), (0xE9, '\u{27A9}'), (0xEA, '\u{27AA}'), (0xEB, '\u{27AB}'), (0xEC, '\u{27AC}'), (0xED, '\u{27AD}'),
-    (0xEE, '\u{27AE}'), (0xEF, '\u{27AF}'), (0xF1, '\u{27B1}'), (0xF2, '\u{27B2}'), (0xF3, '\u{27B3}'), (0xF4, '\u{27B4}'),
-    (0xF5, '\u{27B5}'), (0xF6, '\u{27B6}'), (0xF7, '\u{27B7}'), (0xF8, '\u{27B8}'), (0xF9, '\u{27B9}'), (0xFA, '\u{27BA}'),
-    (0xFB, '\u{27BB}'), (0xFC, '\u{27BC}'), (0xFD, '\u{27BD}'), (0xFE, '\u{27BE}'),
-];
-
-/// Every width table, for exhaustive testing.
-#[cfg(test)]
-static ALL_TABLES: &[(&str, &[(char, u16)])] = &[
-    ("COURIER", COURIER),
-    ("COURIER_BOLD", COURIER_BOLD),
-    ("COURIER_OBLIQUE", COURIER_OBLIQUE),
-    ("COURIER_BOLDOBLIQUE", COURIER_BOLDOBLIQUE),
-    ("HELVETICA", HELVETICA),
-    ("HELVETICA_BOLD", HELVETICA_BOLD),
-    ("HELVETICA_OBLIQUE", HELVETICA_OBLIQUE),
-    ("HELVETICA_BOLDOBLIQUE", HELVETICA_BOLDOBLIQUE),
-    ("TIMES_ROMAN", TIMES_ROMAN),
-    ("TIMES_BOLD", TIMES_BOLD),
-    ("TIMES_ITALIC", TIMES_ITALIC),
-    ("TIMES_BOLDITALIC", TIMES_BOLDITALIC),
-    ("SYMBOL", SYMBOL),
-    ("ZAPFDINGBATS", ZAPFDINGBATS),
-];
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn times_roman_ascii_widths() {
-        assert_eq!(base14_char_width("Times-Roman", ' '), Some(250));
-        assert_eq!(base14_char_width("Times-Roman", 'M'), Some(889));
-        assert_eq!(base14_char_width("Times-Roman", 'i'), Some(278));
-    }
-
-    #[test]
-    fn subset_prefix_and_aliases_normalize() {
-        assert_eq!(
-            base14_char_width("ABCDEF+Times-Bold", ' '),
-            base14_char_width("Times-Bold", ' ')
-        );
-        assert!(base14_char_width("ArialMT", 'a').is_some());
-        assert!(base14_char_width("TimesNewRomanPSMT", 'a').is_some());
-    }
-
-    #[test]
-    fn non_base14_returns_none() {
-        assert_eq!(base14_char_width("DejaVuSans", 'a'), None);
-        assert!(!is_base14_font("Garamond"));
-    }
-
-    #[test]
-    fn builtin_encoding_resolves_symbol_and_zapf_codes() {
-        // Symbol 0x61 renders alpha; Zapf 0x21 renders U+2701.
-        assert_eq!(builtin_encoding_char("Symbol", 0x61), Some('\u{03B1}'));
-        assert_eq!(builtin_encoding_char("Symbol", 0xA5), Some('\u{221E}'));
-        assert_eq!(
-            builtin_encoding_char("ZapfDingbats", 0x21),
-            Some('\u{2701}')
-        );
-        // Latin text fonts follow standard encodings — no builtin override.
-        assert_eq!(builtin_encoding_char("Times-Roman", 0x61), None);
-        // The resolved chars have real AFM widths.
-        let alpha_w = base14_char_width("Symbol", '\u{03B1}');
-        assert!(alpha_w.is_some() && alpha_w != Some(500));
-    }
-
-    #[test]
-    fn encoding_tables_are_sorted_for_binary_search() {
-        for table in [SYMBOL_ENCODING, ZAPFDINGBATS_ENCODING] {
-            assert!(table.windows(2).all(|w| w[0].0 < w[1].0));
-        }
-    }
-
-    #[test]
-    fn tables_are_sorted_for_binary_search() {
-        // Every table is queried by binary search, so all of them must be
-        // sorted — not just a sample.
-        for (name, table) in ALL_TABLES {
-            assert!(
-                table.windows(2).all(|w| w[0].0 < w[1].0),
-                "{name} is not sorted"
-            );
-        }
-    }
-}

src/extractor/content_stream.rs

@@ -14,9 +14,9 @@ use lopdf::{Document, Encoding, Object, ObjectId};
 use std::collections::HashMap;
 
 use super::fonts::{
-    build_font_encodings, build_font_widths, build_type3_scales, compute_string_width_ts,
-    descriptor_style_flags, extract_text_from_operand, get_font_file2_obj_num, get_operand_bytes,
-    CMapDecisionCache, FontStyleCache,
+    build_font_encodings, build_font_widths, compute_string_width_ts, descriptor_style_flags,
+    extract_text_from_operand, get_font_file2_obj_num, get_operand_bytes, CMapDecisionCache,
+    FontStyleCache,
 };
 use super::underline::UnderlineLine;
 use super::xobjects::{extract_form_xobject_text, get_page_xobjects, XObjectType};
@@ -41,17 +41,6 @@ fn strip_pdf_comments(data: &[u8]) -> Vec<u8> {
     while i < data.len() {
         let b = data[i];
         match b {
-            // Inside a string literal, a backslash escapes the next byte —
-            // `\(`, `\)`, and `\\` must not touch the nesting depth, or a
-            // later `%` glyph inside a string gets stripped as a comment,
-            // corrupting the stream.
-            b'\\' if in_string > 0 => {
-                result.push(b);
-                if let Some(&next) = data.get(i + 1) {
-                    result.push(next);
-                    i += 1;
-                }
-            }
             b'(' if !in_hex_string => {
                 in_string += 1;
                 result.push(b);
@@ -177,7 +166,6 @@ pub(crate) fn extract_page_text_items(
 
     // Build font width info for accurate text positioning
     let font_widths = build_font_widths(doc, &fonts);
-    let type3_scales = build_type3_scales(doc, &fonts);
 
     // Build maps of font resource names to their base font names and ToUnicode object refs
     let mut font_base_names: std::collections::HashMap<String, String> =
@@ -514,8 +502,7 @@ pub(crate) fn extract_page_text_items(
                     ) {
                         let combined =
                             multiply_matrices(&rise_adjusted(&text_matrix, text_rise), &ctm);
-                        let rendered_size = effective_font_size(current_font_size, &combined)
-                            * type3_scales.get(&current_font).copied().unwrap_or(1.0);
+                        let rendered_size = effective_font_size(current_font_size, &combined);
                         let (x, y) = (combined[4], combined[5]);
                         if combined[0].abs() >= combined[1].abs() {
                             rotation_votes.horizontal += 1;
@@ -686,8 +673,7 @@ pub(crate) fn extract_page_text_items(
                             } else {
                                 rotation_votes.rotated += 1;
                             }
-                            let rendered_size = effective_font_size(current_font_size, &combined)
-                                * type3_scales.get(&current_font).copied().unwrap_or(1.0);
+                            let rendered_size = effective_font_size(current_font_size, &combined);
                             let base_font = font_base_names
                                 .get(&current_font)
                                 .map(|s| s.as_str())
@@ -794,8 +780,7 @@ pub(crate) fn extract_page_text_items(
                             } else {
                                 rotation_votes.rotated += 1;
                             }
-                            let rendered_size = effective_font_size(current_font_size, &combined)
-                                * type3_scales.get(&current_font).copied().unwrap_or(1.0);
+                            let rendered_size = effective_font_size(current_font_size, &combined);
                             let (x, y) = (combined[4], combined[5]);
                             let width = w_ts_opt
                                 .map(|w_ts| {
@@ -947,8 +932,7 @@ pub(crate) fn extract_page_text_items(
                             } else {
                                 rotation_votes.rotated += 1;
                             }
-                            let rendered_size = effective_font_size(current_font_size, &combined)
-                                * type3_scales.get(&current_font).copied().unwrap_or(1.0);
+                            let rendered_size = effective_font_size(current_font_size, &combined);
                             let (x, y) = (combined[4], combined[5]);
                             // Width in device space from text matrix delta
                             let delta_ts = text_matrix[4] - start_tm[4];
@@ -1865,26 +1849,4 @@ BT 30 700 Tm <41> Tj ET";
             "ET should be preserved after comment stripping"
         );
     }
-
-    #[test]
-    fn test_strip_pdf_comments_escaped_parens() {
-        // An escaped `\)` must not close the string: the `%` after it is
-        // still string content, not a comment (subset fonts routinely map
-        // glyphs to `%` and to escaped parens in the same TJ array).
-        let input = b"[ (a\\)b) 1 (%) 1 (c) ] TJ\n";
-        let output = strip_pdf_comments(input);
-        assert_eq!(output, input.to_vec());
-
-        // Same for an escaped `\(` — must not open a phantom string that
-        // shields a real comment.
-        let input = b"(x\\(y) Tj % real comment\nET\n";
-        let output = strip_pdf_comments(input);
-        assert_eq!(output, b"(x\\(y) Tj  \nET\n");
-
-        // Escaped backslash before a real close-paren: `\\` ends the escape,
-        // the `)` does close the string, and the comment is stripped.
-        let input = b"(x\\\\) Tj % comment\nET\n";
-        let output = strip_pdf_comments(input);
-        assert_eq!(output, b"(x\\\\) Tj  \nET\n");
-    }
 }

src/extractor/fonts.rs

@@ -138,93 +138,6 @@ pub(crate) fn build_font_widths(
     widths
 }
 
-/// Visual-size scale factors for Type3 fonts, keyed by resource name.
-///
-/// A Type3 font's glyph space maps to text space through FontMatrix, so the
-/// visual height of its glyphs is `nominal_size × |matrix_y| × FontBBox
-/// height`. For a well-behaved font (matrix 0.001, bbox ≈ 1000 units) that
-/// factor is ≈ 1.0 and the nominal size is already right. TeX PK bitmap
-/// fonts (dvips → Distiller) instead use FontMatrix [1 0 0 -1 0 0] with
-/// nominal sizes like 0.12, which makes every downstream font-size heuristic
-/// (drop caps, sub/superscripts, small-font tables, line heights) see
-/// nonsense. Fonts without a usable FontBBox are omitted (treated as 1.0).
-pub(crate) fn build_type3_scales(
-    doc: &Document,
-    fonts: &std::collections::BTreeMap<Vec<u8>, &lopdf::Dictionary>,
-) -> HashMap<String, f32> {
-    let mut scales = HashMap::new();
-    for (font_name, font_dict) in fonts {
-        let is_type3 = font_dict
-            .get(b"Subtype")
-            .ok()
-            .and_then(|o| o.as_name().ok())
-            .is_some_and(|n| n == b"Type3");
-        if !is_type3 {
-            continue;
-        }
-        // Array elements may themselves be indirect references per PDF
-        // syntax — resolve before reading the numeric value.
-        let num = |o: &Object| {
-            let resolved = match o {
-                Object::Reference(r) => match doc.get_object(*r) {
-                    Ok(inner) => inner,
-                    Err(_) => return 0.0,
-                },
-                other => other,
-            };
-            match resolved {
-                Object::Integer(i) => *i as f32,
-                Object::Real(r) => *r,
-                _ => 0.0,
-            }
-        };
-        let Some(matrix) = font_dict
-            .get(b"FontMatrix")
-            .ok()
-            .and_then(|o| resolve_array(doc, o))
-        else {
-            continue;
-        };
-        let Some(bbox) = font_dict
-            .get(b"FontBBox")
-            .ok()
-            .and_then(|o| resolve_array(doc, o))
-        else {
-            continue;
-        };
-        if matrix.len() < 4 || bbox.len() < 4 {
-            continue;
-        }
-        let scale_y = (num(&matrix[2]).powi(2) + num(&matrix[3]).powi(2)).sqrt();
-        let bbox_h = (num(&bbox[3]) - num(&bbox[1])).abs();
-        let scale = bbox_h * scale_y;
-
-        // `scale` is the glyph box measured in text-space units. For a
-        // self-consistent font it lands near 1.0 — the FontMatrix is the
-        // reciprocal of the glyph-space em by construction — so the Tf
-        // operand is already the rendered size and must be left alone.
-        // A modest deviation is normal and must NOT trigger rescaling:
-        // FontBBox is the glyph bounding box, not the em box, so it is
-        // routinely somewhat smaller (descender..ascender ≈ 0.7) or larger
-        // (tall accents > 1.0).
-        //
-        // Only a wildly inconsistent font gets renormalized. dvips/PK
-        // bitmap fonts declare [1 0 0 -1 0 0] with glyphs spanning
-        // hundreds of units, giving scale ≈ 159 against a nominal size of
-        // 0.12pt — there the declared size carries no information. The
-        // band is deliberately wide so that only that class qualifies,
-        // while any matrix scale (including non-standard ones like 0.005
-        // with a full-em bbox, scale = 5.0) is judged on the product
-        // rather than on the matrix alone.
-        const CONSISTENT_LO: f32 = 0.25;
-        const CONSISTENT_HI: f32 = 4.0;
-        if scale.is_finite() && scale > 0.0 && !(CONSISTENT_LO..=CONSISTENT_HI).contains(&scale) {
-            scales.insert(String::from_utf8_lossy(font_name).to_string(), scale);
-        }
-    }
-    scales
-}
-
 /// Parse font widths from a font dictionary, dispatching by Subtype
 pub(crate) fn parse_font_widths(
     doc: &Document,
@@ -236,71 +149,11 @@ pub(crate) fn parse_font_widths(
 
     match subtype_name {
         b"Type0" => parse_type0_widths(doc, font_dict),
-        b"Type1" | b"TrueType" | b"MMType1" => parse_simple_font_widths(doc, font_dict)
-            .or_else(|| base14_fallback_widths(doc, font_dict)),
-        b"Type3" => parse_simple_font_widths(doc, font_dict),
+        b"Type1" | b"TrueType" | b"MMType1" | b"Type3" => parse_simple_font_widths(doc, font_dict),
         _ => None,
     }
 }
 
-/// Fallback metrics for non-embedded base-14 fonts whose dictionary omits
-/// `/FirstChar`/`/Widths` (legal per the PDF spec — the reader must supply
-/// standard-font metrics). Without this, every glyph advances 0 and all
-/// downstream gap-based logic (space synthesis, script detection, table
-/// columns) collapses — common in 1990s dvips/Distiller PDFs.
-///
-/// Widths are resolved per code through the font's Differences encoding when
-/// present, falling back to the same single-byte decode the text extractor
-/// uses (cp1252-style smart punctuation for 0x80..=0x9F, Latin-1 elsewhere) —
-/// so the width of a code always matches the char we extract for it.
-fn base14_fallback_widths(doc: &Document, font_dict: &lopdf::Dictionary) -> Option<FontWidthInfo> {
-    let base_font = font_dict
-        .get(b"BaseFont")
-        .ok()
-        .and_then(|o| o.as_name().ok())
-        .map(|n| String::from_utf8_lossy(n).to_string())?;
-    if !crate::extractor::base14::is_base14_font(&base_font) {
-        return None;
-    }
-
-    let enc_map = parse_font_encoding(doc, font_dict)
-        .map(|r| r.map)
-        .unwrap_or_default();
-
-    let mut widths = HashMap::new();
-    for code in 0u16..=255 {
-        // Resolution order: Differences override, then the font's BUILT-IN
-        // encoding (Symbol/ZapfDingbats glyphs live at positions unrelated
-        // to cp1252 — the renderer draws α for Symbol 0x61 no matter how
-        // the text decoder transliterates it, so the advance must be α's),
-        // then the cp1252-style fallback used by the text decoder.
-        let ch = enc_map
-            .get(&(code as u8))
-            .copied()
-            .or_else(|| crate::extractor::base14::builtin_encoding_char(&base_font, code as u8))
-            .unwrap_or_else(|| decode_single_byte_fallback_char(code as u8, true));
-        if let Some(w) = crate::extractor::base14::base14_char_width(&base_font, ch) {
-            widths.insert(code, w);
-        }
-    }
-    let space_width = widths.get(&32).copied().unwrap_or(250);
-
-    debug!(
-        "  base14 fallback widths for {} ({} codes mapped)",
-        base_font,
-        widths.len()
-    );
-
-    Some(FontWidthInfo {
-        widths,
-        default_width: 500,
-        space_width,
-        is_cid: false,
-        units_scale: 0.001,
-        wmode: 0,
-    })
-}
-
 /// Parse widths for simple fonts (Type1, TrueType, MMType1, Type3)
 /// Reads FirstChar, LastChar, and Widths array.
 /// For Type3 fonts, reads FontMatrix to determine the correct units_scale.
@@ -1617,92 +1470,6 @@ fn score_text(text: &str) -> i32 {
 #[cfg(test)]
 mod tests {
 
-    #[test]
-    fn type3_scale_resolves_indirect_matrix_and_bbox_numbers() {
-        use lopdf::{dictionary, Document, Object};
-        // FontMatrix/FontBBox elements may be indirect references per PDF
-        // syntax; the scale must use their resolved values, not zero.
-        let mut doc = Document::with_version("1.4");
-        let matrix_d = doc.add_object(Object::Real(-1.0));
-        let bbox_top = doc.add_object(Object::Integer(3));
-        let font_dict = dictionary! {
-            "Type" => "Font",
-            "Subtype" => "Type3",
-            "FontMatrix" => vec![
-                Object::Integer(1),
-                Object::Integer(0),
-                Object::Integer(0),
-                Object::Reference(matrix_d),
-                Object::Integer(0),
-                Object::Integer(0),
-            ],
-            "FontBBox" => vec![
-                Object::Integer(1),
-                Object::Integer(-156),
-                Object::Integer(37),
-                Object::Reference(bbox_top),
-            ],
-        };
-        let mut fonts = std::collections::BTreeMap::new();
-        fonts.insert(b"T2".to_vec(), &font_dict);
-        let scales = super::build_type3_scales(&doc, &fonts);
-        let scale = scales.get("T2").copied().unwrap_or(1.0);
-        // bbox height 159 x |matrix_y| 1.0
-        assert!(
-            (scale - 159.0).abs() < 0.5,
-            "scale should use resolved indirect values, got {scale}"
-        );
-    }
-
-    /// Build a one-font Type3 document and return its computed scale, if any.
-    #[cfg(test)]
-    fn type3_scale_for(matrix_y: f32, bbox_lo: i64, bbox_hi: i64) -> Option<f32> {
-        use lopdf::{dictionary, Document, Object};
-        let doc = Document::with_version("1.4");
-        let font_dict = dictionary! {
-            "Type" => "Font",
-            "Subtype" => "Type3",
-            "FontMatrix" => vec![
-                Object::Real(matrix_y), Object::Integer(0), Object::Integer(0),
-                Object::Real(matrix_y), Object::Integer(0), Object::Integer(0),
-            ],
-            "FontBBox" => vec![
-                Object::Integer(0), Object::Integer(bbox_lo),
-                Object::Integer(600), Object::Integer(bbox_hi),
-            ],
-        };
-        let mut fonts = std::collections::BTreeMap::new();
-        fonts.insert(b"T9".to_vec(), &font_dict);
-        super::build_type3_scales(&doc, &fonts).get("T9").copied()
-    }
-
-    #[test]
-    fn type3_scale_skips_self_consistent_fonts() {
-        // Conventional 1/1000 matrix with a descender..ascender bbox of 700
-        // units: scale 0.7. The Tf operand is already the rendered size, so
-        // renormalizing would report every size at 0.7x.
-        assert_eq!(type3_scale_for(0.001, -200, 500), None);
-        // Tall-accent bbox slightly over the em (1100 units, scale 1.1).
-        assert_eq!(type3_scale_for(0.001, -100, 1000), None);
-    }
-
-    #[test]
-    fn type3_scale_applies_to_inconsistent_fonts_at_any_matrix_scale() {
-        // Non-standard but valid matrix (0.005) with a full-em bbox:
-        // scale 5.0, so the declared size is off by 5x and must be fixed.
-        let s = type3_scale_for(0.005, 0, 1000).expect("0.005 matrix should rescale");
-        assert!((s - 5.0).abs() < 0.01, "got {s}");
-        // dvips/PK bitmap pattern: unit matrix, glyphs spanning ~159 units.
-        let s = type3_scale_for(1.0, -156, 3).expect("PK pattern should rescale");
-        assert!((s - 159.0).abs() < 0.5, "got {s}");
-    }
-
-    #[test]
-    fn type3_scale_ignores_degenerate_bbox() {
-        // [0 0 0 0] is legal and carries no size information.
-        assert_eq!(type3_scale_for(0.001, 0, 0), None);
-    }
-
     #[test]
     fn texcm_math_symbols_remap() {
         assert_eq!(

src/extractor/mod.rs

@@ -2,7 +2,6 @@
 //!
 //! This module extracts text with position information for structure detection.
 
-mod base14;
 pub(crate) mod content_stream;
 mod fonts;
 mod layout;
@@ -85,33 +84,17 @@ pub fn extract_text_with_positions_pages<P: AsRef<Path>>(
     path: P,
     page_filter: Option<&HashSet<u32>>,
 ) -> Result<Vec<TextItem>, PdfError> {
-    let (items, _rects, _lines) =
-        extract_text_with_positions_and_rects_with_password(path, page_filter, None)?;
+    let (items, _rects, _lines) = extract_text_with_positions_and_rects(path, page_filter)?;
     Ok(items)
 }
 
-/// Extract text with positions from a file, limited to specific pages and
-/// decrypting with `password` when the PDF is encrypted.
-///
-/// `page_filter` is an optional set of 1-indexed page numbers to process.
-/// When `None`, all pages are processed.
-pub fn extract_text_with_positions_pages_with_password<P: AsRef<Path>>(
+/// Extract text with positions and rectangles from a file.
+pub(crate) fn extract_text_with_positions_and_rects<P: AsRef<Path>>(
     path: P,
     page_filter: Option<&HashSet<u32>>,
-    password: Option<&str>,
-) -> Result<Vec<TextItem>, PdfError> {
-    let (items, _rects, _lines) =
-        extract_text_with_positions_and_rects_with_password(path, page_filter, password)?;
-    Ok(items)
-}
-
-pub(crate) fn extract_text_with_positions_and_rects_with_password<P: AsRef<Path>>(
-    path: P,
-    page_filter: Option<&HashSet<u32>>,
-    password: Option<&str>,
 ) -> Result<PageExtraction, PdfError> {
     crate::validate_pdf_file(&path)?;
-    let (doc, _) = crate::load_document_from_path_with_password(&path, password)?;
+    let (doc, _) = crate::load_document_from_path(&path)?;
     let font_cmaps = FontCMaps::from_doc(&doc);
     let (extraction, _thresholds, _gid_pages) =
         extract_positioned_text_from_doc(&doc, &font_cmaps, page_filter)?;

src/extractor/underline.rs

@@ -44,16 +44,6 @@ const MIN_TABULAR_RULE_ITEMS: usize = 3;
 const MIN_TABULAR_RULE_GAPS: usize = 2;
 const TABULAR_RULE_GAP_EM: f32 = 2.0;
 
-/// Strikeout decorations are text-sized. Diagram connectors, signature
-/// lines, and chart rules often cross glyphs too, but extend well beyond the
-/// text they happen to intersect.
-const STRIKE_OWNER_PAD_EM: f32 = 0.75;
-const STRIKE_OWNER_MIN_PAD: f32 = 4.0;
-const STRIKE_ROW_Y_TOLERANCE_EM: f32 = 0.15;
-const STRIKE_ROW_Y_TOLERANCE_MIN: f32 = 5.0;
-const GRAPHIC_CONNECTION_EPS: f32 = 2.0;
-const GRAPHIC_CONNECTOR_MAX_THICKNESS: f32 = 4.0;
-
 #[derive(Clone)]
 pub(crate) struct UnderlineLine {
     pub(crate) x1: f32,
@@ -397,206 +387,6 @@ fn rule_strikes_item(rule: &Rule, item: &TextItem) -> bool {
     overlap >= min_overlap
 }
 
-fn is_bare_list_marker(text: &str) -> bool {
-    matches!(
-        text.trim(),
-        "•" | "◦" | "▪" | "▫" | "‣" | "⁃" | "●" | "○" | "■" | "□" | "-" | "*"
-    )
-}
-
-fn same_strike_row(left: &TextItem, right: &TextItem) -> bool {
-    let font_size = left.font_size.max(right.font_size);
-    let tolerance = (font_size * STRIKE_ROW_Y_TOLERANCE_EM).max(STRIKE_ROW_Y_TOLERANCE_MIN);
-    (left.y - right.y).abs() <= tolerance
-}
-
-fn is_inline_script(rule: &Rule, candidate: &TextItem, parent: &TextItem) -> bool {
-    if !is_underline_candidate(candidate)
-        || is_bare_list_marker(&candidate.text)
-        || candidate.font_size <= 0.0
-        || candidate.font_size >= parent.font_size * 0.75
-        || candidate.text.len() > 4
-        || !candidate.text.chars().all(|c| c.is_ascii_digit())
-        || (candidate.y - parent.y).abs() > 5.0
-    {
-        return false;
-    }
-
-    let parent_ends_with_letter = parent.text.chars().last().is_some_and(char::is_alphabetic);
-    if !parent_ends_with_letter {
-        return false;
-    }
-
-    let parent_right = parent.x + parent.width;
-    let gap = candidate.x - parent_right;
-    if gap >= parent.font_size * 0.2 || gap <= -parent.font_size * 0.3 {
-        return false;
-    }
-
-    let overlap = rule.x2.min(candidate.x + candidate.width) - rule.x1.max(candidate.x);
-    overlap >= candidate.width * MIN_X_OVERLAP
-}
-
-/// Return the items owned by a snug mid-glyph rule.
-///
-/// Real strikeout decorations track the width of the deleted text, including
-/// runs split by font/style changes and adjacent numeric super/subscripts.
-/// Non-text graphics can cross the same vertical window, but arrow shafts,
-/// signature lines, fraction bars, and chart rules extend materially beyond
-/// the intersected glyphs. Requiring the rule to stay within a small em-sized
-/// pad of a contiguous matched row separates those cases without relying on
-/// document-specific fonts or coordinates.
-///
-/// Ownership is computed once per rule. This keeps the strikeout pass at the
-/// same rule-by-item scale as underline detection instead of rescanning the
-/// whole page for every matching item.
-fn snug_strike_owner_indices(rule: &Rule, items: &[TextItem]) -> Vec<usize> {
-    let mut struck_indices: Vec<usize> = items
-        .iter()
-        .enumerate()
-        .filter_map(|(index, candidate)| {
-            (is_underline_candidate(candidate)
-                && !is_bare_list_marker(&candidate.text)
-                && rule_strikes_item(rule, candidate))
-            .then_some(index)
-        })
-        .collect();
-    if struck_indices.is_empty() {
-        return Vec::new();
-    }
-    struck_indices.sort_by(|&left, &right| items[left].y.total_cmp(&items[right].y));
-
-    let mut rows: Vec<Vec<usize>> = Vec::new();
-    for index in struck_indices {
-        if let Some(row) = rows
-            .last_mut()
-            .filter(|row| same_strike_row(&items[row[0]], &items[index]))
-        {
-            row.push(index);
-        } else {
-            rows.push(vec![index]);
-        }
-    }
-
-    let mut owned_indices = Vec::new();
-    for mut row in rows {
-        row.sort_by(|&left, &right| items[left].x.total_cmp(&items[right].x));
-
-        // Underline detection runs before the extractor's script-merging
-        // pass. Include the same tightly adjacent numeric script shape here
-        // when the rule spans it, so the owner width and semantic mark both
-        // survive that later merge.
-        let scripts: Vec<usize> = items
-            .iter()
-            .enumerate()
-            .filter_map(|(index, candidate)| {
-                let parent_pos =
-                    row.partition_point(|&row_index| items[row_index].x <= candidate.x);
-                let parent_index = parent_pos.checked_sub(1).map(|pos| row[pos])?;
-                is_inline_script(rule, candidate, &items[parent_index]).then_some(index)
-            })
-            .collect();
-        row.extend(scripts);
-        row.sort_by(|&left, &right| items[left].x.total_cmp(&items[right].x));
-        row.dedup();
-
-        let x1 = row
-            .iter()
-            .map(|&index| items[index].x)
-            .fold(f32::INFINITY, f32::min);
-        let x2 = row
-            .iter()
-            .map(|&index| items[index].x + items[index].width)
-            .fold(f32::NEG_INFINITY, f32::max);
-        let max_font_size = row
-            .iter()
-            .map(|&index| items[index].font_size)
-            .fold(0.0, f32::max);
-        let pad = (max_font_size * STRIKE_OWNER_PAD_EM).max(STRIKE_OWNER_MIN_PAD);
-
-        if rule.x1 < x1 - pad || rule.x2 > x2 + pad {
-            continue;
-        }
-
-        let contiguous = row.windows(2).all(|pair| {
-            let gap = items[pair[1]].x - (items[pair[0]].x + items[pair[0]].width);
-            gap <= (max_font_size * 2.0).max(12.0)
-        });
-        if contiguous {
-            owned_indices.extend(row);
-        }
-    }
-
-    owned_indices.sort_unstable();
-    owned_indices.dedup();
-    owned_indices
-}
-
-/// Diagram and table rules participate in larger path geometry. A vertical
-/// or diagonal segment meeting the candidate rule is strong evidence that
-/// the horizontal segment is a connector, border, arrow, or symbol rather
-/// than an isolated text decoration.
-fn has_connected_nonhorizontal_segment(rule: &Rule, lines: &[UnderlineLine], page: u32) -> bool {
-    lines.iter().any(|line| {
-        if line.page != page {
-            return false;
-        }
-
-        let dx = line.x2 - line.x1;
-        let dy = line.y2 - line.y1;
-        if dy.abs() <= MAX_RULE_THICKNESS {
-            return false;
-        }
-
-        let y_min = line.y1.min(line.y2) - GRAPHIC_CONNECTION_EPS;
-        let y_max = line.y1.max(line.y2) + GRAPHIC_CONNECTION_EPS;
-        if rule.y < y_min || rule.y > y_max {
-            return false;
-        }
-
-        let t = (rule.y - line.y1) / dy;
-        if !(-0.05..=1.05).contains(&t) {
-            return false;
-        }
-        let intersection_x = line.x1 + t * dx;
-        intersection_x >= rule.x1 - GRAPHIC_CONNECTION_EPS
-            && intersection_x <= rule.x2 + GRAPHIC_CONNECTION_EPS
-    })
-}
-
-/// Filled diagrams often build connectors from intersecting thin rectangles
-/// instead of stroked path segments. Treat only narrow, vertically elongated
-/// rectangles as connector geometry; broad fills can legitimately sit behind
-/// struck text and must not veto its decoration.
-fn has_connected_nonhorizontal_rect(rule: &Rule, rects: &[PdfRect], page: u32) -> bool {
-    rects.iter().any(|rect| {
-        if rect.page != page {
-            return false;
-        }
-
-        let (x1, x2) = if rect.width >= 0.0 {
-            (rect.x, rect.x + rect.width)
-        } else {
-            (rect.x + rect.width, rect.x)
-        };
-        let (y1, y2) = if rect.height >= 0.0 {
-            (rect.y, rect.y + rect.height)
-        } else {
-            (rect.y + rect.height, rect.y)
-        };
-        let width = x2 - x1;
-        let height = y2 - y1;
-
-        width > 0.0
-            && width <= GRAPHIC_CONNECTOR_MAX_THICKNESS
-            && height > width * 2.0
-            && rule.y >= y1 - GRAPHIC_CONNECTION_EPS
-            && rule.y <= y2 + GRAPHIC_CONNECTION_EPS
-            && x2 >= rule.x1 - GRAPHIC_CONNECTION_EPS
-            && x1 <= rule.x2 + GRAPHIC_CONNECTION_EPS
-    })
-}
-
 /// Mark `is_underline` on text items that have a horizontal rule just
 /// below their baseline, and `is_strikeout` on items whose glyphs a rule
 /// crosses at mid x-height. `items`, `rects`, and `lines` are a single
@@ -650,43 +440,27 @@ pub(crate) fn mark_underlined_items(
         .map(|(i, _)| i)
         .collect();
 
-    let mut strikeout_items = vec![false; items.len()];
-    for (rule_idx, rule) in rules.iter().enumerate() {
-        if tabular_rules.contains(&rule_idx)
-            || has_connected_nonhorizontal_segment(rule, lines, page)
-            || has_connected_nonhorizontal_rect(rule, rects, page)
-        {
-            continue;
-        }
-        for item_idx in snug_strike_owner_indices(rule, items) {
-            strikeout_items[item_idx] = true;
-        }
-    }
-
-    let underlined_items: HashSet<usize> = items
-        .iter()
-        .enumerate()
-        .filter(|(_, item)| {
-            is_underline_candidate(item)
-                && rules.iter().enumerate().any(|(rule_idx, rule)| {
-                    !tabular_rules.contains(&rule_idx)
-                        && !fraction_rules.contains(&rule_idx)
-                        && rule_matches_item(rule, item)
-                })
-        })
-        .map(|(item_idx, _)| item_idx)
-        .collect();
-
-    for (item_idx, item) in items.iter_mut().enumerate() {
+    for item in items.iter_mut() {
         if !is_underline_candidate(item) {
             continue;
         }
 
-        if strikeout_items[item_idx] {
-            item.is_strikeout = true;
-        }
-        if underlined_items.contains(&item_idx) {
-            item.is_underline = true;
+        for (rule_idx, rule) in rules.iter().enumerate() {
+            if tabular_rules.contains(&rule_idx) {
+                continue;
+            }
+            // The fraction guard only gates UNDERLINE marking — a rule that
+            // reads as a fraction bar from below can still legitimately
+            // strike through a line above it.
+            if !fraction_rules.contains(&rule_idx) && rule_matches_item(rule, item) {
+                item.is_underline = true;
+            }
+            if rule_strikes_item(rule, item) {
+                item.is_strikeout = true;
+            }
+            if item.is_underline && item.is_strikeout {
+                break;
+            }
         }
     }
 }
@@ -806,148 +580,6 @@ mod tests {
         assert!(!items[0].is_underline);
     }
 
-    #[test]
-    fn long_connector_crossing_text_is_not_a_strikeout() {
-        let mut items = vec![item("diagram label", 160.0, 500.0, 60.0, 10.0)];
-        let lines = vec![hline(100.0, 280.0, 503.0)];
-
-        mark_underlined_items(&mut items, &[], &lines, 1);
-
-        assert!(!items[0].is_strikeout);
-    }
-
-    #[test]
-    fn chart_rule_ending_inside_short_label_is_not_a_strikeout() {
-        let mut items = vec![item("T 18", 300.0, 500.0, 20.0, 10.0)];
-        let lines = vec![hline(100.0, 315.0, 503.0)];
-
-        mark_underlined_items(&mut items, &[], &lines, 1);
-
-        assert!(!items[0].is_strikeout);
-    }
-
-    #[test]
-    fn connected_diagram_segment_is_not_a_strikeout() {
-        let mut items = vec![item("V8", 100.0, 500.0, 12.0, 10.0)];
-        let lines = vec![
-            hline(99.0, 113.0, 503.0),
-            UnderlineLine {
-                x1: 106.0,
-                y1: 496.0,
-                x2: 109.0,
-                y2: 510.0,
-                stroke_width: 1.0,
-                page: 1,
-            },
-        ];
-
-        mark_underlined_items(&mut items, &[], &lines, 1);
-
-        assert!(!items[0].is_strikeout);
-    }
-
-    #[test]
-    fn connected_filled_rect_is_not_a_strikeout() {
-        let mut items = vec![item("V8", 100.0, 500.0, 12.0, 10.0)];
-        let rects = vec![
-            thin_rect(99.0, 502.6, 14.0),
-            PdfRect {
-                x: 106.0,
-                y: 496.0,
-                width: 2.0,
-                height: 14.0,
-                page: 1,
-            },
-        ];
-
-        mark_underlined_items(&mut items, &rects, &[], 1);
-
-        assert!(!items[0].is_strikeout);
-    }
-
-    #[test]
-    fn broad_fill_behind_text_does_not_block_strikeout() {
-        let mut items = vec![item("deleted", 100.0, 500.0, 40.0, 10.0)];
-        let rects = vec![
-            thin_rect(99.0, 502.6, 42.0),
-            PdfRect {
-                x: 90.0,
-                y: 490.0,
-                width: 100.0,
-                height: 20.0,
-                page: 1,
-            },
-        ];
-
-        mark_underlined_items(&mut items, &rects, &[], 1);
-
-        assert!(items[0].is_strikeout);
-    }
-
-    #[test]
-    fn list_bullet_is_not_a_strikeout() {
-        for marker in ["•", "-", "*"] {
-            let mut items = vec![item(marker, 100.0, 500.0, 6.0, 10.0)];
-            let lines = vec![hline(99.0, 107.0, 503.0)];
-
-            mark_underlined_items(&mut items, &[], &lines, 1);
-
-            assert!(!items[0].is_strikeout, "marker {marker:?}");
-        }
-    }
-
-    #[test]
-    fn snug_rule_marks_adjacent_split_runs_as_strikeout() {
-        let mut items = vec![
-            item("deleted", 100.0, 500.0, 40.0, 10.0),
-            item("text", 142.0, 500.0, 25.0, 10.0),
-        ];
-        let lines = vec![hline(99.0, 168.0, 503.0)];
-
-        mark_underlined_items(&mut items, &[], &lines, 1);
-
-        assert!(items.iter().all(|item| item.is_strikeout));
-    }
-
-    #[test]
-    fn snug_rule_groups_split_runs_with_baseline_drift() {
-        let mut items = vec![
-            item("deleted", 100.0, 500.0, 40.0, 10.0),
-            item("text", 142.0, 498.0, 25.0, 10.0),
-        ];
-        let lines = vec![hline(99.0, 168.0, 503.0)];
-
-        mark_underlined_items(&mut items, &[], &lines, 1);
-
-        assert!(items.iter().all(|item| item.is_strikeout));
-    }
-
-    #[test]
-    fn snug_rule_keeps_inline_superscript_in_strike_owner() {
-        let mut items = vec![
-            item("deleted", 100.0, 500.0, 40.0, 10.0),
-            item("2", 140.5, 503.0, 4.0, 6.0),
-        ];
-        let lines = vec![hline(99.0, 145.0, 503.0)];
-
-        mark_underlined_items(&mut items, &[], &lines, 1);
-
-        assert!(items.iter().all(|item| item.is_strikeout));
-    }
-
-    #[test]
-    fn snug_rule_keeps_inline_subscript_in_strike_owner() {
-        let mut items = vec![
-            item("deleted", 100.0, 500.0, 40.0, 10.0),
-            item("2", 140.5, 497.0, 4.0, 6.0),
-        ];
-        let lines = vec![hline(99.0, 145.0, 503.0)];
-
-        mark_underlined_items(&mut items, &[], &lines, 1);
-
-        assert!(items.iter().all(|item| item.is_strikeout));
-    }
-
     #[test]
     fn baseline_rule_marks_underline_not_strikeout() {
         let mut items = vec![item("underlined", 100.0, 500.0, 60.0, 10.0)];

src/extractor/xobjects.rs

@@ -8,9 +8,8 @@ use lopdf::{Document, Encoding, Object, ObjectId};
 use std::collections::HashMap;
 
 use super::fonts::{
-    build_font_encodings, build_font_widths, build_type3_scales, compute_string_width_ts,
-    extract_text_from_operand, get_font_file2_obj_num, get_operand_bytes, CMapDecisionCache,
-    FontStyleCache,
+    build_font_encodings, build_font_widths, compute_string_width_ts, extract_text_from_operand,
+    get_font_file2_obj_num, get_operand_bytes, CMapDecisionCache, FontStyleCache,
 };
 use super::{get_number, image_bbox_from_ctm, multiply_matrices};
 
@@ -167,7 +166,6 @@ fn extract_form_xobject_text_inner(
 
     // Build font width info for the form
     let font_widths = build_font_widths(doc, &form_fonts);
-    let type3_scales = build_type3_scales(doc, &form_fonts);
 
     // Build font base names and ToUnicode refs for the form
     let mut font_base_names: HashMap<String, String> = HashMap::new();
@@ -415,8 +413,7 @@ fn extract_form_xobject_text_inner(
                         &font_widths,
                     ) {
                         let combined = multiply_matrices(&text_matrix, &ctm);
-                        let rendered_size = effective_font_size(current_font_size, &combined)
-                            * type3_scales.get(&current_font).copied().unwrap_or(1.0);
+                        let rendered_size = effective_font_size(current_font_size, &combined);
                         let (x, y) = (combined[4], combined[5]);
                         let width = if let Some(font_info) = font_widths.get(&current_font) {
                             if let Some(raw_bytes) = get_operand_bytes(&op.operands[0]) {
@@ -575,8 +572,7 @@ fn extract_form_xobject_text_inner(
                         }
                         if !sub_items.is_empty() {
                             let combined = multiply_matrices(&text_matrix, &ctm);
-                            let rendered_size = effective_font_size(current_font_size, &combined)
-                                * type3_scales.get(&current_font).copied().unwrap_or(1.0);
+                            let rendered_size = effective_font_size(current_font_size, &combined);
                             let base_font = font_base_names
                                 .get(&current_font)
                                 .map(|s| s.as_str())

src/lib.rs

@@ -50,7 +50,7 @@ pub use detector::{
 };
 pub use extractor::{
     extract_text, extract_text_with_positions, extract_text_with_positions_mem,
-    extract_text_with_positions_pages, extract_text_with_positions_pages_with_password,
+    extract_text_with_positions_pages,
 };
 pub use markdown::{
     to_markdown, to_markdown_from_items, to_markdown_from_items_with_rects,
@@ -1090,12 +1090,7 @@ pub fn extract_tables_in_regions_mem(
                     candidates.push(candidate);
                 }
             }
-            let detected = tables::detect_tables_with_page_width(
-                &matched,
-                base_font_size,
-                false,
-                items.map_or(1.0, |items| tables::content_width(items)),
-            );
+            let detected = tables::detect_tables(&matched, base_font_size, false);
             if let Some(candidate) = detected
                 .iter()
                 .find_map(|t| evaluate(TableCandidateSource::Heuristic, t))
@@ -5657,7 +5652,6 @@ fn compute_layout_complexity(
 
         // Check for side-by-side layout
         let owned_items: Vec<types::TextItem> = page_items.iter().map(|i| (*i).clone()).collect();
-        let page_content_width = tables::content_width(&owned_items);
         let bands = markdown::split_side_by_side(&owned_items);
 
         let band_ranges: Vec<(f32, f32)> = if bands.is_empty() {
@@ -5708,12 +5702,7 @@ fn compute_layout_complexity(
                 break;
             }
             // Heuristic fallback for borderless tables
-            let heuristic_tables = tables::detect_tables_with_page_width(
-                &band_items,
-                base_size,
-                false,
-                page_content_width,
-            );
+            let heuristic_tables = tables::detect_tables(&band_items, base_size, false);
             if has_data_table(&heuristic_tables) {
                 found_table = true;
                 break;

src/markdown/analysis.rs

@@ -171,74 +171,6 @@ pub(crate) fn is_toc_marker_heading(text: &str) -> bool {
 /// equation and absent from name-plus-number headings. A bare trailing colon
 /// is NOT a fragment signal either: real headings frequently end with colons
 /// ("Procedure:", "Steps for Using the Microscope:").
-/// True when the line opens with a section number ("3.", "2.1.4", "IV)").
-///
-/// Mirrors the acceptance of `heading::parse_numbering` rather than the
-/// stricter `convert::starts_with_section_number`, which deliberately
-/// requires two components because it bypasses isolation checks. Here a
-/// single "1." counts: numbering is independent evidence of a heading, and
-/// `heading.rs` applies its numbered-prefix allowance *after* consulting
-/// `is_heading_fragment`, so without this exemption a numbered
-/// sentence-case heading would be vetoed before that allowance can run.
-fn starts_with_numbering_prefix(t: &str) -> bool {
-    let Some(first) = t.split_whitespace().next() else {
-        return false;
-    };
-    let has_delimiter = first.ends_with(['.', ')', ':']);
-    let token = first.trim_end_matches(['.', ')', ':']);
-    if token.is_empty() {
-        return false;
-    }
-    let parts: Vec<&str> = token.split('.').collect();
-    let decimal = parts
-        .iter()
-        .all(|p| !p.is_empty() && p.len() <= 3 && p.chars().all(|c| c.is_ascii_digit()));
-    if decimal {
-        // "1." / "2.1." carry a delimiter; "2.3 Title" is written without
-        // one, so a multi-component number is accepted bare. A bare single
-        // number ("3 apples") is not — that is ordinary prose.
-        return has_delimiter || parts.len() >= 2;
-    }
-    // Roman numerals go through the heading parser's own grammar so the two
-    // agree: uppercase I/V/X/L/C only, at most 8 characters. A looser rule
-    // here would exempt markers the parser rejects — "iv)" or "d)" from an
-    // alphabetical list — letting an ordinary list item bypass the veto and
-    // reach heading promotion.
-    //
-    // A delimiter is also required: a bare leading "I" is the pronoun far
-    // more often than a section number.
-    has_delimiter && crate::markdown::heading::roman_value(token).is_some()
-}
-
-/// True when the line reads as a title rather than a sentence: every
-/// content word (ignoring minor words) starts uppercase. Used to spare real
-/// headings from the dangling-verb veto — "Bond Yields" is a section title,
-/// "the method yields" is a stranded clause, and only the casing tells them
-/// apart.
-fn looks_title_case(t: &str) -> bool {
-    const MINOR: &[&str] = &[
-        "a", "an", "the", "of", "and", "or", "for", "to", "in", "on", "at", "by", "with", "from",
-        "as", "is", "are", "that", "than", "into",
-    ];
-    let mut content = 0usize;
-    let mut capitalized = 0usize;
-    for w in t.split_whitespace() {
-        let cleaned: String = w.chars().filter(|c| c.is_alphabetic()).collect();
-        if cleaned.is_empty() {
-            continue;
-        }
-        if MINOR.contains(&cleaned.to_lowercase().as_str()) {
-            continue;
-        }
-        content += 1;
-        if cleaned.chars().next().is_some_and(char::is_uppercase) {
-            capitalized += 1;
-        }
-    }
-    // A single content word ("Yields") is a title by default.
-    content == 0 || capitalized == content
-}
-
 pub(crate) fn is_heading_fragment(text: &str) -> bool {
     let t = text.trim_end();
 
@@ -312,133 +244,9 @@ pub(crate) fn is_heading_fragment(text: &str) -> bool {
     if t.ends_with(':') && t.split_whitespace().any(is_equation_number) {
         return true;
     }
-
-    // Dangling clause: a stranded sentence lead-in ends on a relational
-    // verb with no terminal punctuation — "Note that the exact error equals"
-    // left ahead of its formula when a phantom table dissolved.
-    //
-    // Gated on the line reading as prose rather than a title. Case is the
-    // discriminator the trailing word alone cannot provide: a heading is
-    // title case ("Bond Yields", "The Method Yields") while a stranded
-    // lead-in is sentence case ("the method yields"). Without this gate the
-    // veto eats real headings — "Bond Yields", "Crop Yields" and any wrapped
-    // title-case heading the preprocessor failed to merge.
-    if !t.ends_with(['.', '!', '?', ':', ';', ')', ']'])
-        && !looks_title_case(t)
-        && !starts_with_numbering_prefix(t)
-    {
-        if let Some(last) = t.split_whitespace().next_back() {
-            let word: String = last
-                .trim_matches(|c: char| !c.is_alphanumeric())
-                .to_lowercase();
-            // Relational verbs only, and only those with no common noun
-            // sense. "yields" was dropped for exactly that reason: "Bond
-            // Yields" is a real section title. Function words, copulas and
-            // auxiliaries were measured and rejected outright — a heading
-            // that wraps across lines ends on those, and suppressing them
-            // destroyed real IRS Publication 17 headings.
-            const DANGLING_TAIL: &[&str] =
-                &["equals", "denotes", "implies", "satisfies", "signifies"];
-            if DANGLING_TAIL.contains(&word.as_str()) {
-                return true;
-            }
-        }
-    }
     false
 }
 
-#[cfg(test)]
-mod fragment_heading_tests {
-    use super::is_heading_fragment;
-
-    #[test]
-    fn dangling_tail_marks_stranded_clause() {
-        // opendataloader 01030000000144: left behind when a phantom table
-        // dissolved, ahead of its formula on the next line.
-        assert!(is_heading_fragment("Note that the exact error equals"));
-        assert!(is_heading_fragment("The remainder term satisfies"));
-        assert!(is_heading_fragment("we conclude that the sum equals"));
-    }
-
-    #[test]
-    fn real_headings_survive() {
-        assert!(!is_heading_fragment("Introduction"));
-        assert!(!is_heading_fragment("Error Analysis"));
-        assert!(!is_heading_fragment("Materials and Methods"));
-        assert!(!is_heading_fragment("Results"));
-        assert!(!is_heading_fragment("3.2 Richardson Extrapolation"));
-        assert!(!is_heading_fragment("Discussion and Conclusions"));
-        // Terminal punctuation means the clause is complete.
-        assert!(!is_heading_fragment("What is a Derivative?"));
-        assert!(!is_heading_fragment("Procedure:"));
-        assert!(!is_heading_fragment("Note that this is important."));
-    }
-
-    #[test]
-    fn title_case_headings_ending_in_a_verb_survive() {
-        // "yields" is also a plural noun; these are real section titles.
-        assert!(!is_heading_fragment("Bond Yields"));
-        assert!(!is_heading_fragment("Crop Yields"));
-        assert!(!is_heading_fragment("Dividend Yields"));
-        assert!(!is_heading_fragment("Yields"));
-        // A wrapped title-case heading whose first line ends on a listed
-        // verb must survive even if the preprocessor failed to merge it.
-        assert!(!is_heading_fragment("The Theorem Implies"));
-        assert!(!is_heading_fragment("What This Denotes"));
-    }
-
-    #[test]
-    fn numbered_sentence_case_headings_survive() {
-        // heading.rs consults is_heading_fragment BEFORE applying its
-        // numbered-prefix allowance, so the veto must not pre-empt it.
-        assert!(!is_heading_fragment("1. What the model implies"));
-        assert!(!is_heading_fragment("2.3 How the estimator satisfies"));
-        assert!(!is_heading_fragment("IV) What this denotes"));
-        // Without numbering the same wording is still a stranded clause.
-        assert!(is_heading_fragment("What the model implies"));
-        // A bare leading number or pronoun is prose, not numbering.
-        assert!(is_heading_fragment("3 apples and what that implies"));
-        assert!(is_heading_fragment("I think the model implies"));
-        // Markers heading::parse_numbering rejects must not be exempted
-        // either, or an ordinary list item bypasses the veto: lowercase
-        // roman, alphabetical markers, and over-long tokens.
-        assert!(is_heading_fragment("iv) the estimator satisfies"));
-        assert!(is_heading_fragment("d) the value implies"));
-        // Unsupported character (M is outside the parser's I/V/X/L/C set).
-        assert!(is_heading_fragment("MMMM. the value implies"));
-        // Over-long token: nine valid characters, so this exercises the
-        // 8-character bound rather than the character set.
-        assert!(is_heading_fragment("IIIIIIIII. the value implies"));
-        // Eight is still within the bound and stays exempt.
-        assert!(!is_heading_fragment("IIIIIIII. What this implies"));
-        // Uppercase roman within the parser's grammar is still exempt.
-        assert!(!is_heading_fragment("IV. What this denotes"));
-        assert!(!is_heading_fragment("XII) What this implies"));
-    }
-
-    #[test]
-    fn wrapped_headings_are_not_fragments() {
-        // A heading that wraps across lines ends on a function word. These
-        // are real headings from IRS Publication 17 and must survive.
-        assert!(!is_heading_fragment("Casualty and"));
-        assert!(!is_heading_fragment("Rule 10. You Must Be at"));
-        assert!(!is_heading_fragment("Higher Standard Deduction for"));
-        assert!(!is_heading_fragment("Qualifying Child of"));
-        assert!(!is_heading_fragment("When Can I Withdraw or"));
-        // Copulas and auxiliaries also end real wrapped headings.
-        assert!(!is_heading_fragment("Rule 15. Your AGI Must Be"));
-        assert!(!is_heading_fragment("What Medical Expenses Are"));
-        assert!(!is_heading_fragment("Rule 13. You Must Have"));
-        assert!(!is_heading_fragment("When Can a Roth IRA Be"));
-    }
-
-    #[test]
-    fn dangling_check_is_case_insensitive() {
-        // All-caps is not sentence case, so the veto must not fire there.
-        assert!(!is_heading_fragment("THE REMAINDER EQUALS"));
-    }
-}
-
 /// Compute the Y-gap threshold for paragraph break detection.
 ///
 /// Instead of using a fixed multiple of base_size (which fails for double-spaced

src/markdown/heading.rs

@@ -127,9 +127,7 @@ fn visual_style(line: &TextLine) -> Option<VisualStyle> {
     })
 }
 
-/// Shared with `analysis::starts_with_numbering_prefix` so the veto
-/// exemption and the heading parser agree on what a roman numeral is.
-pub(super) fn roman_value(token: &str) -> Option<u32> {
+fn roman_value(token: &str) -> Option<u32> {
     if token.is_empty() || token.len() > 8 {
         return None;
     }

src/markdown/mod.rs

@@ -1035,8 +1035,8 @@ pub(crate) fn to_markdown_from_items_with_rects_and_lines(
     context: MarkdownDocumentContext<'_>,
 ) -> String {
     use crate::tables::{
-        content_width, detect_tables_from_lines, detect_tables_from_rects,
-        detect_tables_from_struct_tree, detect_tables_with_page_width, try_build_rect_guided_table,
+        detect_tables, detect_tables_from_lines, detect_tables_from_rects,
+        detect_tables_from_struct_tree, try_build_rect_guided_table,
     };
     use crate::types::ItemType;
 
@@ -1144,7 +1144,6 @@ pub(crate) fn to_markdown_from_items_with_rects_and_lines(
     for page in pages {
         let group = page_groups.get(&page).unwrap();
         let page_items: Vec<TextItem> = group.iter().map(|(_, item)| (*item).clone()).collect();
-        let page_content_width = content_width(&page_items);
 
         // Chart-bar regions: bar charts drawn as filled rects read as cell
         // rects or aligned text and get gridded into phantom tables. Their
@@ -1429,12 +1428,7 @@ pub(crate) fn to_markdown_from_items_with_rects_and_lines(
                     // table can share the prose anchors. Reject only candidates
                     // whose cells prove they are parallel prose fragments.
                     let reject_parallel_prose = chart_prose_columns && !was_split;
-                    let tables = detect_tables_with_page_width(
-                        subset_items,
-                        base_size,
-                        false,
-                        page_content_width,
-                    );
+                    let tables = detect_tables(subset_items, base_size, false);
                     for table in tables {
                         if reject_parallel_prose && is_parallel_prose_table(&table) {
                             log::debug!(
@@ -1615,12 +1609,7 @@ pub(crate) fn to_markdown_from_items_with_rects_and_lines(
             // and reject chart-page prose candidates individually below.
             let skip_body_font =
                 merged_retry_skips_body_font(detected_columns, !chart_regions.is_empty());
-            let heuristic_tables = detect_tables_with_page_width(
-                &chart_free,
-                base_size,
-                skip_body_font,
-                page_content_width,
-            );
+            let heuristic_tables = detect_tables(&chart_free, base_size, skip_body_font);
             for table in &heuristic_tables {
                 if !chart_regions.is_empty() && is_parallel_prose_table(table) {
                     log::debug!(

src/tables/grid.rs

@@ -436,8 +436,7 @@ pub(crate) fn recover_header_row(
         .iter()
         .enumerate()
         .filter(|(_, item)| {
-            !item.is_strikeout
-                && item.font_size > small_font_threshold
+            item.font_size > small_font_threshold
                 && item.y > first_row_y
                 && item.y <= first_row_y + row_gap_limit
         })
@@ -805,31 +804,6 @@ mod tests {
         assert_eq!(table.rows.len(), rows_before);
     }
 
-    #[test]
-    fn test_recover_header_row_skips_strikeout_candidates() {
-        let mut old_col1 = make_item("Old Col1", 100.0, 520.0, 12.0);
-        old_col1.is_strikeout = true;
-        let mut old_col2 = make_item("Old Col2", 200.0, 520.0, 12.0);
-        old_col2.is_strikeout = true;
-        let all_items = vec![
-            old_col1,
-            old_col2,
-            make_item("A", 100.0, 500.0, 8.0),
-            make_item("B", 200.0, 500.0, 8.0),
-        ];
-        let mut table = Table {
-            columns: vec![100.0, 200.0],
-            rows: vec![500.0, 480.0],
-            cells: vec![vec!["A".into(), "B".into()], vec!["C".into(), "D".into()]],
-            item_indices: vec![2, 3],
-            kind: TableKind::Data,
-        };
-
-        recover_header_row(&mut table, &all_items, 9.0);
-        assert_eq!(table.rows.len(), 2);
-        assert_eq!(table.cells[0], vec!["A", "B"]);
-    }
-
     #[test]
     fn test_recover_header_row_too_far_above() {
         let all_items = vec![

src/tables/mod.rs

@@ -12,9 +12,7 @@ mod grid;
 pub mod structured;
 
 pub use detect_heuristic::detect_tables;
-pub(crate) use detect_heuristic::{
-    content_width, detect_tables_with_page_width, is_table_of_contents,
-};
+pub(crate) use detect_heuristic::is_table_of_contents;
 pub use detect_lines::detect_tables_from_lines;
 pub(crate) use detect_lines::detect_vector_grid_tables_from_lines;
 pub(crate) use detect_rects::cluster_rects;

src/types.rs

@@ -148,17 +148,14 @@ impl TextLine {
         self.text_with_formatting(false, false, false)
     }
 
-    /// Get text with optional bold/italic/decorative markdown formatting.
-    ///
-    /// `format_decorations` enables both geometrically detected source
-    /// decorations: underline (`<u>`) and strikeout (`<s>`).
+    /// Get text with optional bold/italic/underline markdown formatting
     pub fn text_with_formatting(
         &self,
         format_bold: bool,
         format_italic: bool,
-        format_decorations: bool,
+        format_underline: bool,
     ) -> String {
-        if !format_bold && !format_italic && !format_decorations {
+        if !format_bold && !format_italic && !format_underline {
             return self.text_plain();
         }
 
@@ -168,7 +165,6 @@ impl TextLine {
         let mut current_bold = false;
         let mut current_italic = false;
         let mut current_underline = false;
-        let mut current_strikeout = false;
 
         for (i, item) in self.items.iter().enumerate() {
             let text = item.text.as_str();
@@ -194,16 +190,13 @@ impl TextLine {
             // we push text_trimmed below (which strips it).
             let has_leading_space = text.starts_with(' ');
 
-            // Check for style changes. Source decorations are exclusive:
-            // `<u>`/`<s>` content stays free of `**`/`*` markers — consumers
-            // (and the eval harnesses this feeds) match tag content literally,
-            // and mixed nesting breaks that. A struck-and-underlined item is
-            // emitted as struck text because deletion is the stronger semantic
-            // distinction in redline documents.
-            let item_strikeout = format_decorations && item.is_strikeout;
-            let item_underline = format_decorations && item.is_underline && !item_strikeout;
-            let item_bold = format_bold && item.is_bold && !item_underline && !item_strikeout;
-            let item_italic = format_italic && item.is_italic && !item_underline && !item_strikeout;
+            // Check for style changes. Underline is exclusive: `<u>` content
+            // stays free of `**`/`*` markers — consumers (and the eval
+            // harnesses this feeds) match the tag content literally, and
+            // mixed `<u>**x**</u>` nesting breaks that.
+            let item_underline = format_underline && item.is_underline;
+            let item_bold = format_bold && item.is_bold && !item_underline;
+            let item_italic = format_italic && item.is_italic && !item_underline;
 
             // Close previous styles if they change
             if current_italic && !item_italic {
@@ -218,10 +211,6 @@ impl TextLine {
                 result.push_str("</u>");
                 current_underline = false;
             }
-            if current_strikeout && !item_strikeout {
-                result.push_str("</s>");
-                current_strikeout = false;
-            }
 
             // Add space: either from spacing logic or preserved from item text
             if needs_space || (has_leading_space && !result.is_empty() && !result.ends_with(' ')) {
@@ -233,10 +222,6 @@ impl TextLine {
                 result.push_str("<u>");
                 current_underline = true;
             }
-            if item_strikeout && !current_strikeout {
-                result.push_str("<s>");
-                current_strikeout = true;
-            }
             if item_bold && !current_bold {
                 result.push_str("**");
                 current_bold = true;
@@ -259,9 +244,6 @@ impl TextLine {
         if current_underline {
             result.push_str("</u>");
         }
-        if current_strikeout {
-            result.push_str("</s>");
-        }
 
         result
     }
@@ -327,88 +309,3 @@ impl TextLine {
             || space_already_exists)
     }
 }
-
-#[cfg(test)]
-mod formatting_tests {
-    use super::{ItemType, TextItem, TextLine};
-
-    fn item(text: &str, x: f32, width: f32, strikeout: bool) -> TextItem {
-        TextItem {
-            text: text.to_string(),
-            x,
-            y: 100.0,
-            width,
-            height: 12.0,
-            font: "F1".to_string(),
-            font_size: 12.0,
-            page: 1,
-            is_bold: false,
-            is_italic: false,
-            is_underline: false,
-            is_strikeout: strikeout,
-            item_type: ItemType::Text,
-            mcid: None,
-        }
-    }
-
-    fn line(items: Vec<TextItem>) -> TextLine {
-        TextLine {
-            items,
-            y: 100.0,
-            page: 1,
-            adaptive_threshold: 0.1,
-        }
-    }
-
-    #[test]
-    fn formatting_emits_semantic_strikeout() {
-        let line = line(vec![item("deleted", 10.0, 42.0, true)]);
-
-        assert_eq!(
-            line.text_with_formatting(true, true, true),
-            "<s>deleted</s>"
-        );
-    }
-
-    #[test]
-    fn formatting_closes_strikeout_before_live_text() {
-        let line = line(vec![
-            item("keep", 10.0, 24.0, false),
-            item("remove", 40.0, 42.0, true),
-            item("keep", 88.0, 24.0, false),
-        ]);
-
-        assert_eq!(
-            line.text_with_formatting(true, true, true),
-            "keep <s>remove</s> keep"
-        );
-    }
-
-    #[test]
-    fn formatting_coalesces_adjacent_struck_items() {
-        let line = line(vec![
-            item("deleted", 10.0, 42.0, true),
-            item("words", 58.0, 30.0, true),
-        ]);
-
-        assert_eq!(
-            line.text_with_formatting(true, true, true),
-            "<s>deleted words</s>"
-        );
-    }
-
-    #[test]
-    fn strikeout_takes_precedence_over_other_styles() {
-        let mut decorated = item("deleted", 10.0, 42.0, true);
-        decorated.is_bold = true;
-        decorated.is_italic = true;
-        decorated.is_underline = true;
-        let line = line(vec![decorated]);
-
-        assert_eq!(
-            line.text_with_formatting(true, true, true),
-            "<s>deleted</s>"
-        );
-        assert_eq!(line.text(), "deleted");
-    }
-}

tests/integration_tests.rs

@@ -1310,18 +1310,6 @@ fn test_snapshot_2013_app2() {
     assert_snapshot("2013-app2");
 }
 
-/// First two pages of Shannon's "A Mathematical Theory of Communication"
-/// (1998 dvips 5.58 → Distiller 3 retypesetting). Canonical legacy-TeX PDF:
-/// non-embedded base-14 fonts with no /Widths (exercises the built-in AFM
-/// metrics fallback), Type3 PK bitmap math fonts with FontMatrix
-/// [1 0 0 -1 0 0] (exercises visual-size scaling), a two-line embedded drop
-/// cap, indent-only paragraph breaks, and display math that must not be
-/// detected as tables or headings.
-#[test]
-fn test_snapshot_shannon_entropy() {
-    assert_snapshot("shannon-entropy-p1-2");
-}
-
 // ============================================================================
 // Pages Needing OCR Tests
 // ============================================================================

tests/snapshots/shannon-entropy-p1-2.md

@@ -1,43 +0,0 @@
-Reprinted with corrections from *The Bell System Technical Journal,* Vol. 27, pp. 379–423, 623–656, July, October, 1948.
-
-## A Mathematical Theory of Communication
-
-### By C. E. SHANNON
-
-INTRODUCTION
-
-HE recent development of various methods of modulation such as PCM and PPM which exchange
-
-# Tbandwidth for signal-to-noise ratio has intensified the interest in a general theory of communication. A
-
-basis for such a theory is contained in the important papers of Nyquist¹ and Hartley² on this subject. In the present paper we will extend the theory to include a number of new factors, in particular the effect of noise in the channel, and the savings possible due to the statistical structure of the original message and due to the nature of the final destination of the information. The fundamental problem of communication is that of reproducing at one point either exactly or ap- proximately a message selected at another point. Frequently the messages have *meaning*; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one *selected from a set* of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design. If the number of messages in the set is finite then this number or any monotonic function of this number can be regarded as a measure of the information produced when one message is chosen from the set, all choices being equally likely. As was pointed out by Hartley the most natural choice is the logarithmic function. Although this definition must be generalized considerably when we consider the influence of the statistics of the message and when we have a continuous range of messages, we will in all cases use an essentially logarithmic measure. The logarithmic measure is more convenient for various reasons:
-
-1. It is practically more useful. Parameters of engineering importance such as time, bandwidth, number of relays, etc., tend to vary linearly with the logarithm of the number of possibilities. For example, adding one relay to a group doubles the number of possible states of the relays. It adds 1 to the base 2 logarithm of this number. Doubling the time roughly squares the number of possible messages, or doubles the logarithm, etc.
-2. It is nearer to our intuitive feeling as to the proper measure. This is closely related to (1) since we in- tuitively measures entities by linear comparison with common standards. One feels, for example, that two punched cards should have twice the capacity of one for information storage, and two identical channels twice the capacity of one for transmitting information.
-3. It is mathematically more suitable. Many of the limiting operations are simple in terms of the loga- rithm but would require clumsy restatement in terms of the number of possibilities. The choice of a logarithmic base corresponds to the choice of a unit for measuring information. If the
-base 2 is used the resulting units may be called binary digits, or more briefly *bits,* a word suggested by
-
-J. W. Tukey. A device with two stable positions, such as a relay or a flip-flop circuit, can store one bit of information. *N* such devices can store*N* bits, since the total number of possible states is 2
-*N* and log₂2 *N* = *N*. If the base 10 is used the units may be called decimal digits. Since
-
-log₂*M* = log₁₀*M*= log₁₀2 = 3:32 log₁₀*M*;
-
-1 Nyquist, H., “Certain Factors Affecting Telegraph Speed,” *Bell System Technical Journal,* April 1924, p. 324; “Certain Topics in Telegraph Transmission Theory,” *A.I.E.E. Trans.,* v. 47, April 1928, p. 617. 2 Hartley, R. V. L., “Transmission of Information,” *Bell System Technical Journal,* July 1928, p. 535.
-
-INFORMATION SOURCE TRANSMITTER RECEIVER DESTINATION
-
-SIGNAL RECEIVED SIGNAL MESSAGE MESSAGE
-
-NOISE SOURCE
-
-Fig. 1 — Schematic diagram of a general communication system.
-
-a decimal digit is about 3 13 bits. A digit wheel on a desk computing machine has ten stable positions and therefore has a storage capacity of one decimal digit. In analytical work where integration and differentiation are involved the base *e* is sometimes useful. The resulting units of information will be called natural units. Change from the base *a* to base *b* merely requires multiplication by log*ba*. By a communication system we will mean a system of the type indicated schematically in Fig. 1. It consists of essentially five parts:
-
-1. An *information source* which produces a message or sequence of messages to be communicated to the receiving terminal. The message may be of various types: (a) A sequence of letters as in a telegraph of teletype system; (b) A single function of time *f* (*t*) as in radio or telephony; (c) A function of time and other variables as in black and white television — here the message may be thought of as a function *f* (*x*; *y*;*t*) of two space coordinates and time, the light intensity at point (*x*; *y*) and time *t* on a pickup tube plate; (d) Two or more functions of time, say *f* (*t*), *g*(*t*), *h*(*t*) — this is the case in “three- dimensional” sound transmission or if the system is intended to service several individual channels in multiplex; (e) Several functions of several variables — in color television the message consists of three functions *f* (*x*; *y*;*t*), *g*(*x*; *y*;*t*), *h*(*x*; *y*;*t*) defined in a three-dimensional continuum — we may also think of these three functions as components of a vector field defined in the region — similarly, several black and white television sources would produce “messages” consisting of a number of functions of three variables; (f) Various combinations also occur, for example in television with an associated audio channel.
-2. A *transmitter* which operates on the message in some way to produce a signal suitable for trans- mission over the channel. In telephony this operation consists merely of changing sound pressure into a proportional electrical current. In telegraphy we have an encoding operation which produces a sequence of dots, dashes and spaces on the channel corresponding to the message. In a multiplex PCM system the different speech functions must be sampled, compressed, quantized and encoded, and finally interleaved properly to construct the signal. Vocoder systems, television and frequency modulation are other examples of complex operations applied to the message to obtain the signal.
-3. The *channel* is merely the medium used to transmit the signal from transmitter to receiver. It may be a pair of wires, a coaxial cable, a band of radio frequencies, a beam of light, etc.
-4. The *receiver* ordinarily performs the inverse operation of that done by the transmitter, reconstructing the message from the signal.
-5. The *destination* is the person (or thing) for whom the message is intended. We wish to consider certain general problems involving communication systems. To do this it is first
-necessary to represent the various elements involved as mathematical entities, suitably idealized from their
-