Repository navigation
JOIN schema change, retyping and capture audits on both tables; the generative tier draws truncates (J5b, closes #1110 with #1163) - #1171
Merged
mmmries merged 2 commits intoOct 19, 2026
Conversation
…part of salesforce-misc#1110) A JOIN behind the group-basis knob already paused on a lost, renamed or re-keyed column, row-level security, a subscription or a hierarchy on either table (salesforce-misc#1107's readers_of). Four paths still saw only the FROM table, or nothing: - defs::copies recorded no copies for a JOIN, so a widened column on either table was never followed. A catalog-only varchar widening left the target and P/M/J/T narrow, so every longer value failed its page. A rewriting widening under SUM() never paused. join_copies now records the target's GROUP BY columns, the hidden tables' side-typed columns (exact type and collation) and the field columns. Retype-in-place, the retype releases (through J5a's join::release), the outgrown pause and the resume's re-type all apply to them. copies::inputs fingerprints the joined table and the hidden tables for the drift memo. - capture::baseline's evaluated-as check compared only the FROM table's recorded types, so the joined table's recorded join_columns went stale silently. It now checks both. - self_check's capture audit listed only the source and relationship to-sides. It now audits the joined table too. - Column quarantine needs no change: the column fuse is 1-1-only, and a JOIN is treated as an aggregate is. The generative JOIN tier now draws TRUNCATE on either table, in half the cases. Docs list the JOIN's copies, and gap 23 covers them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g them first (salesforce-misc#1110) J5b made a JOIN's hidden-table columns typed copies, so a resume asked the staging worker to ALTER P, M, J and T before complete_resume drops and recreates them anyway. J keeps every key it has seen, so after an ON pair narrows (bigint to integer) a key whose rows are gone fails the ALTER, and the resume ended with "DROP TRANSFORM and define it again" where recreating J succeeds. It also rewrote P (up to the fact table's size) under ACCESS EXCLUSIVE just to drop it. Both resume paths now leave JoinHidden copies to the recreate. Gap 16's and gap 23's self_check lines now count the JOIN audit J7a (salesforce-misc#1166) landed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs/known-correctness-gaps.md: gap 23 (a space-padded value truncated into a widenedvarcharcopy) now also lists aJOIN'sGROUP BYandONcopies, what that truncation does to aJOINgroup, and thatself_checkdetects it on aJOINtarget. Gap 16's planned work now saysself_checkalready comparesJOINtargets (J7a, #1166).Summary
J5b of #1110, stacked on #1163 (J5a). A
JOIN(behindTRELLIS_DEV_GROUP_BASIS=1) now goes through schema change, retyping and the capture and row-security audits on both of its tables, the way an aggregate does on its one table. J5a covered per-sideTRUNCATEand the cell recompute. The generative JOIN tier now drawsTRUNCATEon either table.Several of these paths already treated a
JOINas reading both tables, through #1107's catalog work, so they needed no change:readers_of: a column dropped, renamed or re-keyed on either table, anONpair that no longer matches, row-level security, a subscription, a hierarchy;This PR closes four gaps that were left.
Changes
Copies (
defs::copies). AJOINused to record no typed copies at all. As a result, a widened column on either table was never followed:varchar(10)→varchar(40)left the target,P/M/TandJatvarchar(10), and every longer value failed its page. Likewise,integer→bigintunderSUM(items.qty)left the target columnbigintwith no pause.join_copiesnow records these:GROUP BYcolumns;T's group columns,P's andM's group and join-key columns, andJ's join key. These are exact types with collation, ashidden_tables_ddlcreates them.join::field_pg_typeover both tables' live columns.The existing machinery then handles them:
retype_in_placere-types catalog-only widenings in place, including the hidden tables, and requests the release of keys held with22001/22003. The release goes through J5a'sjoin::release.JOIN.Jkeeps every key it has seen, so anALTERofJafter anONpair narrows could fail on a key whose rows are gone, and would refuse the resume.copies::inputs(the drift memo's fingerprint set) now includes the joined table and the hidden tables, so a change to the joined table isn't skipped.Evaluated-as check (
capture::baseline::respond). When a column changes type without firing a trigger and the definition would now evaluate it as a different value type, the definition pauses. That check compared only theFROMtable's recorded types. It now also compares the joined table's recorded types (join_columns). TheJOINapply renders the joined table's images by those recorded types, sotimestamp→timestamptzunderCOUNT(items.shipped)used to go unpaused.Capture audit (
staging::capture_audit::read_tables).self_check's capture audit listed only the source and relationship to-sides. A disabled trigger, row-level security or a subscription on aJOIN's joined table was never reported, and the comparison ran anyway. The joined table is now audited.Column quarantine. No code change. The column fuse and its cascade are 1-1-only (ADR-0003). A
JOINevaluates in SQL and raises noEvalErrorto attribute, andcolumn_dependentsskips non-1-1 readers. So aJOINis treated exactly as an aggregate is, which is what #1110 asks for.Generative JOIN tier.
JoinOpKind::Truncate(weight 1 in 101) is drawn on either table in half the cases. A truncating case issues each truncated table from one lane (concurrent_plan's whole-table rule), so the other half keeps several lanes per table. The hand-built fast case truncates each table once. TheTODO(#1110)notes and the suite doc are updated.Docs.
known-correctness-gaps.md(gaps 16 and 23),transforms.md(the widening row, including itsRESUMEremedy, and theJOINrow's after-define column),type-support.md, ADR-0014,01-capture-by-triggers.md, and thedefs::joinmodule doc. These describe theJOIN's copies.Verification
New tests. Each fails with the
src/change reverted and passes with it:defs_join_define::a_widened_column_of_either_table_is_retyped_in_place_or_pauses_the_join:GROUP BYcolumns on both tables and aC-collatedONpair widened together are re-typed in place in the target,P,M,JandT, collation kept, with no pause. Thenqtygoingbigintpauses, namingpublic.by_code.qty, and the resume re-types it tonumeric. Before the fix:left: "character varying(10)".defs_join_define::a_joined_column_retyped_to_another_value_type_pauses_the_join:timestamp→timestamptzunder aCOUNTpauses theJOINon either table. Before the fix, the joined table didn't pause (left: WaitingToBackfill).join_schema_changes::a_catalog_only_widening_of_either_table_is_retyped_in_place_on_a_live_join: a liveLEFT JOINwithvarchar(5)group andONcolumns widened tovarchar(20).JOINlive, and releases both keys.left: "character varying(5)".join_self_check::a_capture_fault_on_either_table_is_reported_before_the_comparison: capture triggers disabled on either table are reported as capture faults on that table, and nothing is compared. Before the fix, the joined table's faults were missed and 20 keys were compared.Row-level security and a subscription on the joined table are reported by the same per-table loop.
tests/row_security.rscovers that check itself, as a non-superuser role. NoJOIN-specific RLS test was added.defs_join_define::a_join_resume_recreates_its_hidden_tables_without_re_typing_them(added in review):Jholds a key pastintegerrange, bothONcolumns narrow frombiginttointeger, thenPAUSE/RESUME. The resume completes andJis recreated asinteger, empty. Without the fix the resume ends paused withtrellis.join_1_j.code ... integer out of range ... DROP TRANSFORM by_code and define it again.generate::joinunit test: truncates on both tables, the empty-table fallback, and no id reuse.join_convergence's fast cases, now with truncates.J5a already covers the acceptance scenarios (J1 family 10 and the mixed-page barrier) in
join_interleavings.rs.Generative sweep: logbook
20261018T223932-a-sweep.PROPTEST_CASES=6 … join_convergence -- --ignoredunderMemoryMax=8G: 6 cases, 11 truncates across both tables, all converged and alllive.verify: green, passed=3602 failed=0 ignored=40 (tree 874292933c, rebased on main 03c68db)Closes #1110 (with #1163). Stacked on #1163; rebase after it merges. A JOIN rebuild after a rewriting DDL still waits for the JOIN build (#1109).
🤖 Generated with Claude Code