* feat(tables): improve heuristic detection for borderless wrapped-cell tables
Three changes to the body-font heuristic detector:
1. Adaptive Y-gap in find_table_regions_strict: use median qualifying-row
spacing × 3 instead of fixed 25pt. Tables with wrapped cells have
larger gaps between qualifying rows (those with 3+ X-clusters).
2. Y-only region filtering: use full X range when collecting region items.
The strict X bounds from qualifying rows excluded continuation lines
in wrapped cells, starving find_column_boundaries of items.
3. Merged-band retry: when split_side_by_side splits a page into bands
but no band produces a table, retry heuristic detection on all items
merged as a single band.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): gap-histogram column detection for small tables + lower avg_cells
Two changes to fix PDF 045 (borderless table with narrow "No." column):
1. Extend gap-histogram column threshold to small tables: when the gap
between within-column jitter and between-column spacing is >10pt
(unambiguous bimodal signal), use the detected threshold even with
fewer than 500 items. Previously only triggered for dense tables.
2. Lower BodyFont avg_cells_per_row minimum from 2.5 to 2.0 to handle
tables with wrapped multi-line cells where continuation lines have
only 1 filled cell.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): trim empty outer columns + relax partial H-line validation
- Rect detection: trim empty first/last columns instead of rejecting
the whole table. Rect edges often extend beyond text boundaries.
- Line detection: accept tables with 6+ partial horizontal lines
(>15% width) when <3 full-spanning lines exist. Handles tables
with column-level separators instead of full-width rules.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): cell-rect fallback for tables with variable-width backgrounds
When rect clustering produces a grid that fails validation (empty
interior columns from variable-width cell backgrounds), fall through
to a new strategy: use rect Y-edges for row boundaries and text
X-position clustering for columns. This handles tables like the
opendataloader-bench 088-090 comparison tables where each cell has
its own background rect at different widths.
Also widen failed-cluster hint width cap for large clusters (≥30 rects)
to allow page-spanning table regions.
TEDS score on opendataloader-bench: 0.300 → 0.353.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): relax vertical line spanning validation for partial borders
Accept tables with 4+ partial vertical lines (>10% table height) when
fewer than 2 span >30%. Handles tables like opendataloader-bench 053
with column-level vertical separators that don't extend the full height.
TEDS: 0.353 → 0.377 on opendataloader-bench.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): lower cell-rect density threshold + add validation logging
- Lower cell-rect density minimum from 25% to 15% to accept sparser
tables with decorative backgrounds (fixes 147).
- Add debug logging to all heuristic validation paths for diagnosability.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): relax body-font validations for text-only and 2-column tables
Four fixes closing 71% of the TEDS gap vs opendataloader:
1. Validation 7 (table-like content): bypass numeric content requirement
for tables with 3+ columns that passed all structural checks. Text-only
tables (category lists, program descriptions) are legitimate.
2. Qualifying row threshold: lower from 3+ to 2+ X-clusters per row.
Enables 2-column body-font table detection (fixes 166).
3. Row-stripe max cell length: raise from 500 to 2000 for 3+ column
tables. Tables with paragraph descriptions in one column are valid
(fixes 121).
4. Row-stripe empty-column trimming: apply the same outer-column trim
as grid detection (fixes 121 column-0 rejection).
TEDS: 0.377 → 0.438 on opendataloader-bench (gap: -0.056 vs odl).
TEDS=0 docs: 14 → 9.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): enable 2-column body-font tables + lower all minimums
- Lower BodyFont minimum columns from 3 to 2 in detect_table_in_region
- Lower BodyFont minimum rows from 3 to 2
- Lower avg_cells_per_row minimum from 2.0 to 1.5 (handles wrapped cells
in 2-column tables)
- Apply empty-outer-column trimming to row-stripe detection (not just grid)
TEDS: 0.438 → 0.468 on opendataloader-bench (gap: -0.027 vs odl).
TEDS=0 docs: 9 → 8. 86% of original gap closed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): text-based row fallback + fix 120 flow-chart and 188 leaderboard
Three changes that push TEDS past opendataloader:
1. Cell-rect Y-edge fallback: when rects have too few Y-edges for row
structure, derive rows from text Y-position clustering within the
rect bounding box. Fixes flow-chart tables (120) and column-header-
only rects (188).
2. Lower cell-rect minimum from 20 to 6 rects to catch smaller tables.
3. Relax validation 1 (first-column presence) from 50% to 25% of rows.
Tables with wrapped model names have continuation lines without first
column content.
TEDS: 0.468 → 0.508 on opendataloader-bench.
Now BEATS opendataloader (0.508 vs 0.494, gap=+0.014).
TEDS=0 docs: 8 → 5.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tables): wrapped-cell continuation row merging
Merge rows that have fewer filled cells than the header row into the
previous row. Handles wrapped multi-line cells where text overflow
creates extra rows (e.g., "Direct" + "communications" → "Direct
communications").
Conditions: fewer filled cells than header, more than previous row had,
not a data row (numeric), not a short subheader label.
TEDS: 0.508 → 0.522 on opendataloader-bench (now +0.028 vs odl).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tables): tighten cell-rect validation + fix continuation-row merging
Cell-rect false positives:
- Raise density threshold back to 25% (from 15%)
- Add max cell length check (500 chars) to reject paragraph content
- Reject disproportionate grids (>20 rows, <4 cols)
Continuation-row merging:
- Wide tables (5+ cols): only merge rows with ≤50% header cells
- Narrow tables (2-4 cols): merge rows with fewer cells than header
- Prevents merging normal data rows in large tables (6_KE_Chart)
while keeping wrapped-cell merging for narrow tables (178)
TEDS: 0.498 on opendataloader-bench (still +0.004 vs odl).
pdf-evals: 191/192 passed, 0 regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(text): handle Tc/Tw character and word spacing in text width computation
PDFs using Tc (character spacing) and Tw (word spacing) operators for text
justification had words incorrectly split across TextItems. The computed
advance width didn't account for these spacing parameters, causing spurious
spaces mid-word (e.g. "deve lopers" instead of "developers").
- Add Tc/Tw operator handling and graphics state save/restore
- Incorporate char_spacing and word_spacing into compute_string_width_ts
- Add adaptive merge threshold: tighter for lowercase→lowercase junctions,
wider before joining punctuation
- Add unit tests for Tc/Tw width computation and merge behavior
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(fonts): add large Tc width computation test
Verifies that large character spacing values are applied in full
without any artificial cap, matching PDF spec behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(text): guard against Tc/Tw-inflated widths in merge and join paths
Two targeted fixes to prevent character-spacing (Tc) and word-spacing
(Tw) inflation from causing data quality regressions:
1. should_join_items: reject large negative gaps (< -font_size) that
arise when Tc/Tw inflate item widths past adjacent items. Fixes
FY_2015 merged numbers (e.g. "239.696.0" → "239.69 6.0").
2. merge_text_items: cap effective width for gap computation when Tw
inflates space-containing items beyond 0.85× font_size per char.
Prevents column-level gaps from collapsing into merge range,
recovering table detection for Baldwin-Edwards and similar PDFs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
OCR text layers and some PDF producers emit trailing spaces on each
text item, which combine with gap-based joining to produce double
spaces ("Vice President"). Now collapses runs of 2+ spaces to single
space within lines, preserving leading indentation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Drop column-width padding from table output. Saves ~37% bytes on
table-heavy PDFs. Markdown renders identically — padding was purely
cosmetic. Optimizes token efficiency for AI agent consumers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tables with very wide cells (e.g. 230+ chars from merged schedule columns)
caused massive whitespace padding in every row. Capping alignment at 40
characters reduces output size significantly (franklin: 77KB→24KB,
Closed-Business: 571KB→502KB) without affecting content.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rects spanning multiple grid rows (e.g. classification labels) now have
their text consolidated into the first sub-row via propagate_merged_cells().
Also prevent continuation-row merge from folding body data into the header.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 5 stripped public-domain PDF fixtures and golden markdown snapshots
for CI regression testing. Fix non-deterministic output caused by
HashMap iteration order in font stats, table heuristics, and rect
clustering by adding deterministic tie-breaking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
