Skip to content

JOIN schema change, retyping and capture audits on both tables; the generative tier draws truncates (J5b, closes #1110 with #1163) - #1171

Merged
mmmries merged 2 commits into
salesforce-misc:mainfrom
mmmries:feat/issue-1110-join-schema-audits
Oct 19, 2026
Merged

mmmries merged 2 commits into
salesforce-misc:mainfrom
mmmries:feat/issue-1110-join-schema-audits

Conversation

@mmmries

@mmmries mmmries commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

⚠️ This PR changes docs/known-correctness-gaps.md: gap 23 (a space-padded value truncated into a widened varchar copy) now also lists a JOIN's GROUP BY and ON copies, what that truncation does to a JOIN group, and that self_check detects it on a JOIN target. Gap 16's planned work now says self_check already compares JOIN targets (J7a, #1166).

Summary

J5b of #1110, stacked on #1163 (J5a). A JOIN (behind TRELLIS_DEV_GROUP_BASIS=1) now goes through schema change, retyping and the capture and row-security audits on both of its tables, the way an aggregate does on its one table. J5a covered per-side TRUNCATE and the cell recompute. The generative JOIN tier now draws TRUNCATE on either table.

Several of these paths already treated a JOIN as reading both tables, through #1107's catalog work, so they needed no change:

  • the capture pass's pauses, through readers_of: a column dropped, renamed or re-keyed on either table, an ON pair that no longer matches, row-level security, a subscription, a hierarchy;
  • define's and resume's checks on both tables;
  • the drain's halt closure.

This PR closes four gaps that were left.

Changes

Copies (defs::copies). A JOIN used to record no typed copies at all. As a result, a widened column on either table was never followed: varchar(10) → varchar(40) left the target, P/M/T and J at varchar(10), and every longer value failed its page. Likewise, integer → bigint under SUM(items.qty) left the target column bigint with no pause. join_copies now records these:

  • the target's GROUP BY columns;
  • the hidden tables' columns typed from a side: T's group columns, P's and M's group and join-key columns, and J's join key. These are exact types with collation, as hidden_tables_ddl creates them.
  • the target's other fields, typed by join::field_pg_type over both tables' live columns.

The existing machinery then handles them:

  • retype_in_place re-types catalog-only widenings in place, including the hidden tables, and requests the release of keys held with 22001/22003. The release goes through J5a's join::release.
  • An outgrown column pauses the JOIN.
  • A resume re-types the target. It doesn't re-type the hidden tables, because it drops them and creates them again at the live types. J keeps every key it has seen, so an ALTER of J after an ON pair narrows could fail on a key whose rows are gone, and would refuse the resume.

copies::inputs (the drift memo's fingerprint set) now includes the joined table and the hidden tables, so a change to the joined table isn't skipped.

Evaluated-as check (capture::baseline::respond). When a column changes type without firing a trigger and the definition would now evaluate it as a different value type, the definition pauses. That check compared only the FROM table's recorded types. It now also compares the joined table's recorded types (join_columns). The JOIN apply renders the joined table's images by those recorded types, so timestamp → timestamptz under COUNT(items.shipped) used to go unpaused.

Capture audit (staging::capture_audit::read_tables). self_check's capture audit listed only the source and relationship to-sides. A disabled trigger, row-level security or a subscription on a JOIN's joined table was never reported, and the comparison ran anyway. The joined table is now audited.

Column quarantine. No code change. The column fuse and its cascade are 1-1-only (ADR-0003). A JOIN evaluates in SQL and raises no EvalError to attribute, and column_dependents skips non-1-1 readers. So a JOIN is treated exactly as an aggregate is, which is what #1110 asks for.

Generative JOIN tier. JoinOpKind::Truncate (weight 1 in 101) is drawn on either table in half the cases. A truncating case issues each truncated table from one lane (concurrent_plan's whole-table rule), so the other half keeps several lanes per table. The hand-built fast case truncates each table once. The TODO(#1110) notes and the suite doc are updated.

Docs. known-correctness-gaps.md (gaps 16 and 23), transforms.md (the widening row, including its RESUME remedy, and the JOIN row's after-define column), type-support.md, ADR-0014, 01-capture-by-triggers.md, and the defs::join module doc. These describe the JOIN's copies.

Verification

New tests. Each fails with the src/ change reverted and passes with it:

  • defs_join_define::a_widened_column_of_either_table_is_retyped_in_place_or_pauses_the_join: GROUP BY columns on both tables and a C-collated ON pair widened together are re-typed in place in the target, P, M, J and T, collation kept, with no pause. Then qty going bigint pauses, naming public.by_code.qty, and the resume re-types it to numeric. Before the fix: left: "character varying(10)".
  • defs_join_define::a_joined_column_retyped_to_another_value_type_pauses_the_join: timestamp → timestamptz under a COUNT pauses the JOIN on either table. Before the fix, the joined table didn't pause (left: WaitingToBackfill).
  • join_schema_changes::a_catalog_only_widening_of_either_table_is_retyped_in_place_on_a_live_join: a live LEFT JOIN with varchar(5) group and ON columns widened to varchar(20).
    • One too-long value per table is held before the pass.
    • The pass re-types all nine columns, leaves the JOIN live, and releases both keys.
    • The target equals the oracle, and does again after longer values are written.
    • Before the fix: left: "character varying(5)".
  • join_self_check::a_capture_fault_on_either_table_is_reported_before_the_comparison: capture triggers disabled on either table are reported as capture faults on that table, and nothing is compared. Before the fix, the joined table's faults were missed and 20 keys were compared.
    Row-level security and a subscription on the joined table are reported by the same per-table loop. tests/row_security.rs covers that check itself, as a non-superuser role. No JOIN-specific RLS test was added.
  • defs_join_define::a_join_resume_recreates_its_hidden_tables_without_re_typing_them (added in review): J holds a key past integer range, both ON columns narrow from bigint to integer, then PAUSE/RESUME. The resume completes and J is recreated as integer, empty. Without the fix the resume ends paused with trellis.join_1_j.code ... integer out of range ... DROP TRANSFORM by_code and define it again.
  • generate::join unit test: truncates on both tables, the empty-table fallback, and no id reuse.
  • join_convergence's fast cases, now with truncates.

J5a already covers the acceptance scenarios (J1 family 10 and the mixed-page barrier) in join_interleavings.rs.

Generative sweep: logbook 20261018T223932-a-sweep. PROPTEST_CASES=6 … join_convergence -- --ignored under MemoryMax=8G: 6 cases, 11 truncates across both tables, all converged and all live.

verify: green, passed=3602 failed=0 ignored=40 (tree 874292933c, rebased on main 03c68db)

Closes #1110 (with #1163). Stacked on #1163; rebase after it merges. A JOIN rebuild after a rewriting DDL still waits for the JOIN build (#1109).

🤖 Generated with Claude Code

mmmries and others added 2 commits October 19, 2026 08:25
…part of salesforce-misc#1110)

A JOIN behind the group-basis knob already paused on a lost, renamed or
re-keyed column, row-level security, a subscription or a hierarchy on
either table (salesforce-misc#1107's readers_of). Four paths still saw only the FROM
table, or nothing:

- defs::copies recorded no copies for a JOIN, so a widened column on
  either table was never followed. A catalog-only varchar widening left
  the target and P/M/J/T narrow, so every longer value failed its page.
  A rewriting widening under SUM() never paused. join_copies now records
  the target's GROUP BY columns, the hidden tables' side-typed columns
  (exact type and collation) and the field columns. Retype-in-place, the
  retype releases (through J5a's join::release), the outgrown pause and
  the resume's re-type all apply to them. copies::inputs fingerprints the
  joined table and the hidden tables for the drift memo.
- capture::baseline's evaluated-as check compared only the FROM table's
  recorded types, so the joined table's recorded join_columns went stale
  silently. It now checks both.
- self_check's capture audit listed only the source and relationship
  to-sides. It now audits the joined table too.
- Column quarantine needs no change: the column fuse is 1-1-only, and a
  JOIN is treated as an aggregate is.

The generative JOIN tier now draws TRUNCATE on either table, in half the
cases. Docs list the JOIN's copies, and gap 23 covers them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g them first (salesforce-misc#1110)

J5b made a JOIN's hidden-table columns typed copies, so a resume asked the
staging worker to ALTER P, M, J and T before complete_resume drops and
recreates them anyway. J keeps every key it has seen, so after an ON pair
narrows (bigint to integer) a key whose rows are gone fails the ALTER, and
the resume ended with "DROP TRANSFORM and define it again" where recreating J
succeeds. It also rewrote P (up to the fact table's size) under ACCESS
EXCLUSIVE just to drop it. Both resume paths now leave JoinHidden copies to
the recreate. Gap 16's and gap 23's self_check lines now count the JOIN
audit J7a (salesforce-misc#1166) landed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

#1091 J5: JOIN TRUNCATE, recompute, quarantine and schema change on either side

1 participant