2f9390e4bf39f00edb30a71b85cafdd4f019f974
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Abimael MartellandClaude Opus 4.6
|
44092bcc9e |
feat(tables): borderless table detection improvements (#17)
* feat(tables): improve heuristic detection for borderless wrapped-cell tables Three changes to the body-font heuristic detector: 1. Adaptive Y-gap in find_table_regions_strict: use median qualifying-row spacing × 3 instead of fixed 25pt. Tables with wrapped cells have larger gaps between qualifying rows (those with 3+ X-clusters). 2. Y-only region filtering: use full X range when collecting region items. The strict X bounds from qualifying rows excluded continuation lines in wrapped cells, starving find_column_boundaries of items. 3. Merged-band retry: when split_side_by_side splits a page into bands but no band produces a table, retry heuristic detection on all items merged as a single band. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(tables): gap-histogram column detection for small tables + lower avg_cells Two changes to fix PDF 045 (borderless table with narrow "No." column): 1. Extend gap-histogram column threshold to small tables: when the gap between within-column jitter and between-column spacing is >10pt (unambiguous bimodal signal), use the detected threshold even with fewer than 500 items. Previously only triggered for dense tables. 2. Lower BodyFont avg_cells_per_row minimum from 2.5 to 2.0 to handle tables with wrapped multi-line cells where continuation lines have only 1 filled cell. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tables): trim empty outer columns + relax partial H-line validation - Rect detection: trim empty first/last columns instead of rejecting the whole table. Rect edges often extend beyond text boundaries. - Line detection: accept tables with 6+ partial horizontal lines (>15% width) when <3 full-spanning lines exist. Handles tables with column-level separators instead of full-width rules. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(tables): cell-rect fallback for tables with variable-width backgrounds When rect clustering produces a grid that fails validation (empty interior columns from variable-width cell backgrounds), fall through to a new strategy: use rect Y-edges for row boundaries and text X-position clustering for columns. This handles tables like the opendataloader-bench 088-090 comparison tables where each cell has its own background rect at different widths. Also widen failed-cluster hint width cap for large clusters (≥30 rects) to allow page-spanning table regions. TEDS score on opendataloader-bench: 0.300 → 0.353. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tables): relax vertical line spanning validation for partial borders Accept tables with 4+ partial vertical lines (>10% table height) when fewer than 2 span >30%. Handles tables like opendataloader-bench 053 with column-level vertical separators that don't extend the full height. TEDS: 0.353 → 0.377 on opendataloader-bench. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tables): lower cell-rect density threshold + add validation logging - Lower cell-rect density minimum from 25% to 15% to accept sparser tables with decorative backgrounds (fixes 147). - Add debug logging to all heuristic validation paths for diagnosability. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(tables): relax body-font validations for text-only and 2-column tables Four fixes closing 71% of the TEDS gap vs opendataloader: 1. Validation 7 (table-like content): bypass numeric content requirement for tables with 3+ columns that passed all structural checks. Text-only tables (category lists, program descriptions) are legitimate. 2. Qualifying row threshold: lower from 3+ to 2+ X-clusters per row. Enables 2-column body-font table detection (fixes 166). 3. Row-stripe max cell length: raise from 500 to 2000 for 3+ column tables. Tables with paragraph descriptions in one column are valid (fixes 121). 4. Row-stripe empty-column trimming: apply the same outer-column trim as grid detection (fixes 121 column-0 rejection). TEDS: 0.377 → 0.438 on opendataloader-bench (gap: -0.056 vs odl). TEDS=0 docs: 14 → 9. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(tables): enable 2-column body-font tables + lower all minimums - Lower BodyFont minimum columns from 3 to 2 in detect_table_in_region - Lower BodyFont minimum rows from 3 to 2 - Lower avg_cells_per_row minimum from 2.0 to 1.5 (handles wrapped cells in 2-column tables) - Apply empty-outer-column trimming to row-stripe detection (not just grid) TEDS: 0.438 → 0.468 on opendataloader-bench (gap: -0.027 vs odl). TEDS=0 docs: 9 → 8. 86% of original gap closed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(tables): text-based row fallback + fix 120 flow-chart and 188 leaderboard Three changes that push TEDS past opendataloader: 1. Cell-rect Y-edge fallback: when rects have too few Y-edges for row structure, derive rows from text Y-position clustering within the rect bounding box. Fixes flow-chart tables (120) and column-header- only rects (188). 2. Lower cell-rect minimum from 20 to 6 rects to catch smaller tables. 3. Relax validation 1 (first-column presence) from 50% to 25% of rows. Tables with wrapped model names have continuation lines without first column content. TEDS: 0.468 → 0.508 on opendataloader-bench. Now BEATS opendataloader (0.508 vs 0.494, gap=+0.014). TEDS=0 docs: 8 → 5. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(tables): wrapped-cell continuation row merging Merge rows that have fewer filled cells than the header row into the previous row. Handles wrapped multi-line cells where text overflow creates extra rows (e.g., "Direct" + "communications" → "Direct communications"). Conditions: fewer filled cells than header, more than previous row had, not a data row (numeric), not a short subheader label. TEDS: 0.508 → 0.522 on opendataloader-bench (now +0.028 vs odl). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tables): tighten cell-rect validation + fix continuation-row merging Cell-rect false positives: - Raise density threshold back to 25% (from 15%) - Add max cell length check (500 chars) to reject paragraph content - Reject disproportionate grids (>20 rows, <4 cols) Continuation-row merging: - Wide tables (5+ cols): only merge rows with ≤50% header cells - Narrow tables (2-4 cols): merge rows with fewer cells than header - Prevents merging normal data rows in large tables (6_KE_Chart) while keeping wrapped-cell merging for narrow tables (178) TEDS: 0.498 on opendataloader-bench (still +0.004 vs odl). pdf-evals: 191/192 passed, 0 regressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Abimael MartellandClaude Opus 4.6
|
8c4181434f |
fix(text): handle Tc/Tw character and word spacing (#11)
* fix(text): handle Tc/Tw character and word spacing in text width computation PDFs using Tc (character spacing) and Tw (word spacing) operators for text justification had words incorrectly split across TextItems. The computed advance width didn't account for these spacing parameters, causing spurious spaces mid-word (e.g. "deve lopers" instead of "developers"). - Add Tc/Tw operator handling and graphics state save/restore - Incorporate char_spacing and word_spacing into compute_string_width_ts - Add adaptive merge threshold: tighter for lowercase→lowercase junctions, wider before joining punctuation - Add unit tests for Tc/Tw width computation and merge behavior Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(fonts): add large Tc width computation test Verifies that large character spacing values are applied in full without any artificial cap, matching PDF spec behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(text): guard against Tc/Tw-inflated widths in merge and join paths Two targeted fixes to prevent character-spacing (Tc) and word-spacing (Tw) inflation from causing data quality regressions: 1. should_join_items: reject large negative gaps (< -font_size) that arise when Tc/Tw inflate item widths past adjacent items. Fixes FY_2015 merged numbers (e.g. "239.696.0" → "239.69 6.0"). 2. merge_text_items: cap effective width for gap computation when Tw inflates space-containing items beyond 0.85× font_size per char. Prevents column-level gaps from collapsing into merge range, recovering table detection for Baldwin-Edwards and similar PDFs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Abimael MartellandClaude Opus 4.6
|
258210a821 |
fix(postprocess): collapse consecutive spaces in extracted text
OCR text layers and some PDF producers emit trailing spaces on each
text item, which combine with gap-based joining to produce double
spaces ("Vice President"). Now collapses runs of 2+ spaces to single
space within lines, preserving leading indentation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
||
|
Abimael MartellandClaude Opus 4.6
|
80a3b81ff9 |
perf(tables): remove cosmetic padding from markdown tables
Drop column-width padding from table output. Saves ~37% bytes on table-heavy PDFs. Markdown renders identically — padding was purely cosmetic. Optimizes token efficiency for AI agent consumers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Abimael MartellandClaude Opus 4.6
|
28313e1f2d |
test: Add snapshot regression tests with PDF fixtures
Add 5 stripped public-domain PDF fixtures and golden markdown snapshots for CI regression testing. Fix non-deterministic output caused by HashMap iteration order in font stats, table heuristics, and rect clustering by adding deterministic tie-breaking. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |