fix(extractor): detect two-column layout when the page also has a table

[mohojojo]

Problem

detect_columns (src/extractor/layout.rs) gates both its fallback column-detection strategies — relative-valley and XY-cut — behind !page_has_table:

if valleys.is_empty() && page_items.len() >= 100 && !page_has_table {

The intent (per the comment) is that a table's inter-column gaps can look like gutters in the projection histogram. But the side effect is that any page which has a detected table and genuine two-column prose loses column detection entirely and collapses to single-column, Y-interleaved reading order.

This is common in real documents. A fund factsheet, for example, has left/right text columns sitting above a returns table. When the absolute-valley pass finds no empty gutter (justified text fills it), the relative-valley fallback is skipped because a table is present, so the two columns get read line-by-line interleaved:

**ÁLTALÁNOS INFORMÁCIÓK PIACI ÖSSZEFOGLALÓ:** Alapkezelő: Marketprog Asset
Management Zrt. Az alap HUF sorozatának árfolyama áprilisban 2,8%-kal emelkedett,
EUR sorozata Letétkezelő: Unicredit Bank Zrt. 3,6%-kal. Vezető forgalmazó: ...

(left-column labels shredded into right-column sentences).

Root cause, confirmed

With RUST_LOG=debug, the same page yields opposite results depending on the page_has_table argument:

page 1: 0 valleys found but none passed validation      # page_has_table=true  → 1 column
page 1: 2 columns detected (boundaries: [294.4543])      # page_has_table=false → correct split
columns_have_prose: col [3..294]   ratio=0.87 avg_items=2.9
columns_have_prose: col [294..591] ratio=0.84 avg_items=1.9
page 1: relative valley detection found 2 columns

The relative-valley path finds the correct split at x=294 and both sides pass columns_have_prose — it's only the page_has_table gate that suppresses it.

Fix

The relative-valley path already has its own table defense: columns_have_prose rejects splits whose sides look like tables/forms (avg_items > 3.5 per line, low prose fill). That guard — not the page-level page_has_table flag — is the right protection, so the relative-valley fallback now runs regardless of tables.

The XY-cut fallback has no such prose guard, so it stays gated on !page_has_table to avoid reading a table's widest column gap as a page gutter.

Verification

• New regression test relative_valley_detects_columns_even_with_table_on_page: two-column justified prose on a page flagged page_has_table = true must still split into two columns. Fails before this change (returns 1 column), passes after.
• Full suite green: cargo test (all existing tests unchanged), cargo fmt --all -- --check, cargo clippy -- -D warnings.
• Ran an unrelated 461-document corpus through pdf2md before/after: only 4 outputs changed, all corrections to the interleaving bug; the other 457 are byte-identical (the columns_have_prose guard keeps the change scoped to genuine two-column pages).

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Fixes two-column detection on pages that also contain tables and stops justified prose from being misidentified as wide text tables. Reading order is corrected and phantom tables are avoided.

• Bug Fixes
  • Run relative‑valley fallback in detect_columns even when page_has_table is true; rely on columns_have_prose to reject table‑like splits, and keep XY‑cut gated behind !page_has_table.
  • Strengthen is_paragraph_content: detect near‑full rows of lowercase, multi‑word fragments as prose to avoid treating justified paragraphs as N‑column tables; real numeric/label tables remain unaffected.
  • Add/refresh regression tests: unit test for the prose heuristic and a snapshot (marketprog-prestige-two-column) that now reads columns correctly and removes the phantom table.

^{Written for commit 0ac0edb. Summary will update on new commits.}

[mohojojo]

Added a real-document regression fixture: tests/fixtures/marketprog-prestige-two-column.pdf (a two-column fund factsheet with a returns table on the same page — the exact layout this fixes) plus its snapshot tests/snapshots/marketprog-prestige-two-column.md and test_snapshot_marketprog_prestige_two_column. The snapshot encodes the corrected reading order (left column ÁLTALÁNOS INFORMÁCIÓK → BEFEKTETÉSI POLITIKA contiguous, preceding the right column PIACI ÖSSZEFOGLALÓ) and fails if the interleaving regresses. Verified it fails on main and passes with the fix.

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[mohojojo]

Good catch — fixed in 0ac0edb rather than blessing the bad output.

Root cause: has_table_like_content bypasses the numeric-content check for any 3+ column table (to allow legitimate text-only tables like category lists). Justified paragraph text abuses this: the disclaimer's inter-word gaps align into vertical "columns", so a two-line paragraph is detected as an 8-column table — splitting one sentence into cells and re-emitting Tájékoztatója…, exactly the duplication/reordering you flagged.

Fix: is_paragraph_content now also recognizes this shape via a prose-row signal — a near-full row whose cells are mostly lowercase mid-sentence fragments with at least one multi-word cell. Two+ such rows ⇒ paragraph, so it's rejected as a table. Numeric or capitalized-label rows never match, so real tables are untouched (the stats table on the very same page still renders).

The snapshot is regenerated: the disclaimer now flows as a single paragraph, no phantom table, no duplication. Added unit tests justified_prose_split_across_columns_is_paragraph and numeric_stats_table_is_not_paragraph.

Verified on a 461-document corpus: removes justified-prose phantom tables from 52 documents and destroys no real data table (spot-checked the biggest deltas — every one was market-commentary or a garbled prose+data merge being correctly de-tabled, with the underlying data preserved as text).

[cubic-dev-ai]

2 issues found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/snapshots/marketprog-prestige-two-column.md">

<violation number="1" location="tests/snapshots/marketprog-prestige-two-column.md:27">
P2: The updated snapshot still blesses malformed prose with missing spaces, so the regression test will accept unreadable output for this paragraph. The extractor's spacing should be corrected and the snapshot regenerated rather than encoding these concatenations as the expected result.</violation>
</file>

<file name="src/tables/detect_heuristic.rs">

<violation number="1" location="src/tables/detect_heuristic.rs:1385">
P2: The new prose-row gate in validation 8 can reject legitimate wide text tables and flatten them to plain text. `has_table_like_content` (validation 7) deliberately bypasses the numeric-content check for any 3+ column table so that genuine text-only tables (category lists, program descriptions, glossaries) are kept. But this new `prose_rows >= 2` check then rejects any wide table where ≥2 rows are near-full, ≥60% lowercase-starting, and contain at least one 3+-word cell. A real multi-column text table whose rows are fully populated with lowercase multi-word cells therefore passes validation 7 and is then dropped as 'prose' — a structural regression for valid tables. The existing unit tests only cover the 8-column disclaimer (should reject) and a numeric stats table (should keep); neither exercises a legitimate full-width lowercase text table, so this false-positive path is untested. Consider gating the new heuristic on the table lacking genuine data cells (numbers/dates/units) so text rows beside real data are preserved, and/or requiring multi-word cells across a majority of the row rather than a single cell.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The updated snapshot still blesses malformed prose with missing spaces, so the regression test will accept unreadable output for this paragraph. The extractor's spacing should be corrected and the snapshot regenerated rather than encoding these concatenations as the expected result.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/snapshots/marketprog-prestige-two-column.md, line 27:

<comment>The updated snapshot still blesses malformed prose with missing spaces, so the regression test will accept unreadable output for this paragraph. The extractor's spacing should be corrected and the snapshot regenerated rather than encoding these concatenations as the expected result.</comment>

<file context>
@@ -24,13 +24,7 @@ Nincs ilyen eszköz a portfólióban **Lejárat szerinti megoszlás:**
-|Tájékoztatója forgalmazási helyeken. A befektetési alap forgalmazásával (vétel, tartás, eladás) kapcsolatos költségek az alap kezelési szabályzatában ésa forgalmazási helyeken megismerhetők.|tartalmazza,|mely a mindenkor|érvényes|kondíciós|listákkal|együtt megtalálható|a|
-
-Tájékoztatója tartalmazza, mely a mindenkor érvényes kondíciós listákkal együtt megtalálható a
+A múltbeli hozamoknem jelentenekgaranciát az alap jövőbeli teljesítményére nézve. Jelen hirdetésnem minősül ajánlattételnek vagy befektetési tanácsadásnak. A befektetés részletes feltételeit az Alap Tájékoztatója tartalmazza, mely a mindenkor érvényes kondíciós listákkal együtt megtalálható a forgalmazási helyeken. A befektetési alap forgalmazásával (vétel, tartás, eladás) kapcsolatos költségek az <u>alap kezelési szabályzatában ésa forgalmazási helyeken megismerhetők.</u>
 
 **KOCKÁZATI MUTATÓK AZ ELMÚLT 12 HÓNAPRA:** Az alap heti hozamokból számolt évesített szórása: 4,77 % A benchmark heti hozamokból számolt évesített szórása: 0,04 % **BEFEKTETÉSI HORIZONT:** A javasolt minimális befektetési idő:
</file context>

[cubic-dev-ai]

P2: The new prose-row gate in validation 8 can reject legitimate wide text tables and flatten them to plain text. has_table_like_content (validation 7) deliberately bypasses the numeric-content check for any 3+ column table so that genuine text-only tables (category lists, program descriptions, glossaries) are kept. But this new prose_rows >= 2 check then rejects any wide table where ≥2 rows are near-full, ≥60% lowercase-starting, and contain at least one 3+-word cell. A real multi-column text table whose rows are fully populated with lowercase multi-word cells therefore passes validation 7 and is then dropped as 'prose' — a structural regression for valid tables. The existing unit tests only cover the 8-column disclaimer (should reject) and a numeric stats table (should keep); neither exercises a legitimate full-width lowercase text table, so this false-positive path is untested. Consider gating the new heuristic on the table lacking genuine data cells (numbers/dates/units) so text rows beside real data are preserved, and/or requiring multi-word cells across a majority of the row rather than a single cell.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/tables/detect_heuristic.rs, line 1385:

<comment>The new prose-row gate in validation 8 can reject legitimate wide text tables and flatten them to plain text. `has_table_like_content` (validation 7) deliberately bypasses the numeric-content check for any 3+ column table so that genuine text-only tables (category lists, program descriptions, glossaries) are kept. But this new `prose_rows >= 2` check then rejects any wide table where ≥2 rows are near-full, ≥60% lowercase-starting, and contain at least one 3+-word cell. A real multi-column text table whose rows are fully populated with lowercase multi-word cells therefore passes validation 7 and is then dropped as 'prose' — a structural regression for valid tables. The existing unit tests only cover the 8-column disclaimer (should reject) and a numeric stats table (should keep); neither exercises a legitimate full-width lowercase text table, so this false-positive path is untested. Consider gating the new heuristic on the table lacking genuine data cells (numbers/dates/units) so text rows beside real data are preserved, and/or requiring multi-word cells across a majority of the row rather than a single cell.</comment>

<file context>
@@ -1360,6 +1359,36 @@ fn is_paragraph_content(cells: &[Vec<String>]) -> bool {
+                .filter(|c| c.chars().next().is_some_and(char::is_lowercase))
+                .count();
+            let has_multiword = row_filled.iter().any(|c| c.split_whitespace().count() >= 3);
+            lowercase_start * 5 >= row_filled.len() * 3 && has_multiword
+        })
+        .count();
</file context>

Suggested change

	lowercase_start * 5 >= row_filled.len() * 3 && has_multiword
	// Only treat as prose if the row carries no genuine table data,
	// so real tables with numbers/dates/units aren't flattened.
	let has_data_cell = row_filled.iter().any(|c| looks_like_table_data(c));
	if has_data_cell {
	return false;
	}
	lowercase_start * 5 >= row_filled.len() * 3 && has_multiword