[mmmries]

Part of #1100 (G2b). This PR also carries the one-commit test fix from #1158 (main's lib tests do not compile until it merges); that commit drops out of the diff once #1158 is in.

Summary

Two pieces of #1100's scope for the group-basis aggregate path, which is still behind the dev knob (TRELLIS_DEV_GROUP_BASIS). With the knob off, nothing changes: no docs change, and the ledger's tombstone GC runs as before.

• Zero-group GC (ADR-0002 R4). A group whose partials all reach zero loses its visible row, but its hidden row stayed in agg_<id>, so a definition whose groups churn (insert-then-delete per group key) grew its hidden table without bound. Research: a group-level basis instead of the per-row ledger for invertible aggregates with no relationships (#897) #1043's JOIN model ran 16x slower with thousands of such rows on hot keys. The maintenance loop's tombstone pass (staging::retire::collect_tombstones) now also deletes, from every hidden table, each zero group that either has no basis or whose stamp is at or below the contiguous drained prefix.

  • Claiming. Rows are claimed FOR UPDATE SKIP LOCKED in key order, in batches of 5,000, up to 4 batches per table per pass. The ledger's batch sizes and backlog signal are reused. Each batch is a short transaction that takes the hidden table's ROW EXCLUSIVE lock NOWAIT, so a concurrent drop skips the table.
  • Finding zero groups cheaply. __zero is a stored generated column (__n = 0 and every __count_i = 0 and every __sum_i = 0) with a partial index on the group columns where __zero. The GC runs under the ledger GC's enable_seqscan/enable_bitmapscan = off, so it walks that index. __zero only flips when a group empties or refills, so an apply that keeps a group non-zero still changes no indexed column and stays HOT. An index whose predicate tested the partials directly would have made every partial update non-HOT.
  • Any non-zero partial keeps the group. n = 0 with a sum still owed is never collected (Research: a group-level basis instead of the per-row ledger for invertible aggregates with no relationships (#897) #1043's RV.gc-n0).
  • __stamp is nullable, with no default. A future absolute write (#1091 G3: Group-basis build and resume (chunked variant B and the one-GROUP-BY build) #1101/#1091 G4: Group-basis TRUNCATE, recompute producers and quarantine #1102) that sets a basis without a stamp keeps its group. Before, it would have been collected at stamp 0.
  • No build skip, unlike the ledger. A recompute or build locks its groups before its snapshot (R1). One that loses a group to the GC inserts it afresh and writes its own basis. A page whose lock statement meets a row the GC is deleting waits for the GC's commit, and ON CONFLICT then inserts the group fresh.

• Plan control on the apply statement (Leaf correction's Re-derive page can plan a per-key nested loop when the root lookups find nothing (found in #1043's model) #1085's concern, J2's full-scan plans).

  • The hidden-row update … from t where u.ctid = t.__hc ran with enable_nestloop off. That made it a hash join over a sequential scan of the whole hidden table, under the page's group locks. It now also filters u.ctid = any(array(select __hc from t where <nonzero>)), which gives a TID scan.
  • The group lookup's NULL branch used x.g is not distinct from alias.g, which no index serves. So every page naming a group with a NULL scanned the hidden table, and the target in the visible-delete lookup, whole. It is now one indexed probe per pattern of NULL columns: equality on the non-null columns, is null on the others, behind a one-time filter. This covers up to 4 GROUP BY columns (15 probes). Past that it keeps the old form.
  • enable_seqscan is now off for the statement, alongside enable_nestloop (both SET LOCAL, reset after). The statement reaches real tables only through the group identity index or by ctid. Without the setting, an un-analyzed or freshly grown table could be planned as a scan. The staging temp table has no index, so it is still scanned sequentially.

• Planner order / truncate barrier: no change, confirmed. Group basis: source TRUNCATE, the group recompute, and quarantine release (G4, part of #1102) #1144's review found the barrier already holds:

  • The seal gate never fences a segment past an unfenced predecessor (seal::seal_phase1's gate, pinned by tests/sealing.rs seal_gate_blocks_when_the_predecessor_fence_is_not_yet_published).
  • next_claimable_segments never returns a segment past the lowest undrained truncate-bearing one. It returns that segment only once every segment below it has drained.

  So segments are planned in order and the two-sided barrier holds at claim. Nothing is duplicated here.

JOIN's P/M apply (J3) isn't on this branch's base. When it lands, its group statements should get the same treatment: TID-array update and per-NULL-pattern probes. Its zero cells need this GC extended to join_<id>_p/_m, whose __stamp is still not null default 0 on that branch and has to become nullable the same way.

G4 (#1102, merged as #1144) sets __basis and __stamp in its recompute's one statement (coalesce(max(seg_seq), 0) beside pg_current_snapshot(), after its lock statement), which is what the GC's soundness needs. Rebased onto it:

• hidden_table_ddl keeps this PR's nullable __stamp with no default, __zero and the partial index. No insert relied on G4's default 0: the lock statement inserts only group columns (no basis, so a zero group from it is collectable), and the recompute always writes __stamp.
• apply_sql keeps G4's image_columns parameter and shared visible_tail_sql, with this PR's TID-array hidden update. group_lookup keeps the per-NULL-pattern probes, which G4's recompute lookups use too.
• recompute_sql's hidden update gets the same TID-array form (u.ctid = any(array(select r.__hc from r)) and u.ctid = r.__hc). Its source lookup keeps is not distinct from.
• The recompute (recompute_locked_groups) doesn't run under PLAN_SETTINGS, so enable_seqscan = off never applies to its source GROUP BY. Only the apply statement and explain_apply take it.
• Name the group-basis staging temp table after its shape, so a pooled connection never reuses an old-shape table (part of #1142) #1157's unit test the_staging_table_name_follows_its_shape (and its sibling) called GroupTargetPlan::new without G4's source_pk argument, so cargo test --lib didn't compile on main. This PR passes an empty source_pk there.

G6's self_check (staging/self_check_groups.rs) reads only the partial columns, never __basis, __stamp or __zero, and it treats a group missing from the hidden table as all zeros. A GC'd zero group therefore compares equal, and its test harness never runs the GC. This PR updates that file's comment that the hidden table keeps a zero group's row. explain_apply builds its plan with #1157's shape-hashed staging table name, as pages do.

Plan evidence

Measured on a scratch PG 17 cluster with shared_buffers = 1GB and a hidden table of 1M groups. Each page touched 4,000 groups × 5 transactions. Times are three runs, EXPLAIN ANALYZE.

Apply statement form	Execution time
G2a (u.ctid = t.__hc, hash join + Seq Scan on 1M rows)	181–186 ms
+ u.ctid = any(array(select __hc from t)) (Tid Scan)	86–88 ms
+ also enable_seqscan = off	87–94 ms (same plan)
Alternative: the group lookup as one g = any(array(...)) index scan + hash join instead of the lateral probes	81–86 ms

I rejected the last alternative: it gains about 5%, and doesn't extend to multi-column groups or the NULL patterns. After the change, the remaining cost is the lateral index probes (about 30 ms) and the visible upsert's 4,000 conflict arbitrations (about 23 ms).

Verification

• tests/group_basis_interleavings.rs adds the following. All are deterministic, with no polling (Flaky CI: a_composite_key_backfill_chunks_a_boundary_inside_a_group_exactly_once times out under runner load #297).

  • r4_the_gc_bounds_a_hidden_table_whose_groups_churn: 5 rounds of filling and emptying 1,000 new groups. Each GC collects exactly the 1,000 zero groups, and the hidden table goes back to its one live group.
  • r4_a_group_with_a_nonzero_partial_stays (Research: a group-level basis instead of the per-row ledger for invertible aggregates with no relationships (#897) #1043 RV.gc-n0): after out-of-order pages, the group sits at n = 0, sum = -2, and the GC keeps it.
  • r4_a_zero_group_with_a_basis_stays_until_the_prefix_passes_its_stamp (Research: a group-level basis instead of the per-row ledger for invertible aggregates with no relationships (#897) #1043 R.recreate-gc-race): a later segment has drained, and the GC still keeps the group while C1's segment hasn't. C1 is then refused, and the group goes once the prefix passes its stamp.
  • r4_a_basis_without_a_stamp_keeps_its_group.
  • r4_the_gc_skips_a_group_a_page_holds: the GC returns 0 while a frozen page holds the zero group, and the page then refills it.
  • r4_a_page_meeting_a_group_the_gc_is_deleting_inserts_it_afresh: the other order. A GC delete is held open when a page refilling the group starts; the page's lock statement queues on it, and once the GC commits it inserts the group afresh, with no basis, and applies its change.
  • every_pattern_of_null_group_columns_finds_its_group: two GROUP BY columns, all four NULL patterns, fills, moves and empties, checked against Postgres after every page.
  • the_apply_statement_reads_no_table_whole: a plan-shape test that the hidden update is a Tid Scan on agg_* whose TID condition is ctid = ANY(...), and that nothing is a Seq Scan on agg*.
  • the_zero_group_gc_walks_its_index: a plan-shape test that the GC uses an Index Scan on agg_*_g_idx and has no Seq Scan.

• The test-side recompute now writes __stamp (the newest segment, read in the same statement as the snapshot, R5).

• Before/after:

  • On the G2a apply SQL with G2a's settings, the_apply_statement_reads_no_table_whole fails with Seq Scan on agg_1 u under the hash join. With enable_seqscan = off alone, the G2a form gets a disabled nested loop probing each hidden row by ctid on the test's tiny tables, so the test also requires the ctid = ANY condition; reverting only the update form fails it.
  • Reverting the group lookup to the is not distinct from form fails the plan test (Seq Scan on agg_1 x_1, Seq Scan on agg x_3). Dropping the all-NULL pattern's probe fails every_pattern_of_null_group_columns_finds_its_group (the (NULL, NULL) group goes missing).
  • Removing skip locked from the GC makes r4_the_gc_skips_a_group_a_page_holds hang behind the frozen page.
  • Mutating the GC predicate to ignore the stamp fails the two basis tests.
  • Mutating it to __n = 0 alone fails those two and r4_a_group_with_a_nonzero_partial_stays.
  • The basis test also covers the early_tombstone_gc plant: there the stamp equals the highest drained segment.

• Checked by hand on a scratch PG 17 cluster: with the generated __zero and its partial index, 1,000 updates that keep groups non-zero were all HOT (n_tup_hot_upd 1000/1000, the same as a table with no partial index). With an index predicate on the partials directly, none were HOT (0/1000). The GC's index scan keeps __zero AND (__basis IS NULL OR __stamp <= $1) as a filter under LockRows, so EPQ re-checks it on a row a page refilled after the GC's snapshot.

• Also checked by hand: with 5 GROUP BY columns (past the per-pattern limit), the lookup falls back to is not distinct from behind its one-time filter, and pages with every mix of NULLs stay exact. With the partial index dropped, the GC skips the table and pages are unaffected.

• verify (after the rebase onto main at aaa970d, with G4 and Name the group-basis staging temp table after its shape, so a pooled connection never reuses an old-shape table (part of #1142) #1157): VERIFIED: tree f0198e11f0 green, passed=3489 failed=0 ignored=38.

• Benchmarks: logbook 20261017T212057-a-bench. The runs are on tmpfs, so they measure CPU only. Each probe ran 8 workers at 400k rows/s offered for 20 s, and all three configurations ran in the same session:

  • ledger: the ledger control;
  • G2a: the G2a-state control, a detached worktree at 5a9b2f1b with --group-basis;
  • G2b: this commit with --group-basis.

  Contamination. Other lanes' cargo test runs and a full verify overlapped the first series (about 21:28–21:50), and the bench script flagged nearly every probe contended. The first G2a-state runs also crashed at 40,000 groups on a /tmp DSM quota error (infra: Disk quota exceeded / Resource temporarily unavailable resizing pg_dynshmem). I therefore re-ran every cell that wasn't clean, one probe at a time and up to 4 tries each, until the script reported it uncontended. The table uses only uncontended probes.

  • Re-run cells: all of G2a; G2b group-contention at 10, 400, 4,000 and 40,000 and fold-in at 40,000; ledger group-contention at 10, 400 and 40,000 and fold-in at 40,000.

  • Kept from the first series (already clean): ledger group-contention at 4,000 and fold-in at 400 and 4,000; G2b fold-in at 400 and 4,000.

  • The 1M-group probes ran in the second re-run series: G2a was clean on its first try, and G2b on its second (its first was flagged contended, at 49.9 µs/row).

  • Group-contention and fold-in-ratio 10 at 40,000 groups are the same workload: 400,000 rows/s over 40,000 groups. They are two samples of one shape, not two shapes.

  All probes drained with the oracle OK, except the G2a probe at 1M groups, which didn't drain.

  Scenario	Groups	Ledger rows/s	G2a rows/s	G2b rows/s	CPU µs/row G2a → G2b	WAL B/row G2a → G2b
  group-contention	10	198k	310k	323k	24.9 → 24.3	445 → 445
  group-contention	400	195k	232k	233k	26.1 → 26.0	454 → 454
  group-contention	4,000	183k	199k	219k	26.6 → 26.2	489 → 490
  group-contention	40,000	112k	96k	90k	34.7 → 34.9	715 → 712
  fold-in-ratio 1000	400	196k	238k	232k	25.9 → 25.8	454 → 454
  fold-in-ratio 100	4,000	190k	208k	213k	26.2 → 26.2	491 → 491
  fold-in-ratio 10	40,000	121k	90k	89k	35.7 → 34.6	720 → 713
  group-contention	1,000,000	—	did not drain (7.95M of 8M applied)	drained, 67k folded/s overall	55.1 → 50.7	1,656 → 1,577

  The 40,000-group case doesn't move. G2a measured 88k vs. the ledger's 115k. Here it is 90k (G2b) vs. 96k (G2a) vs. 112k (ledger) in group-contention, and 89k vs. 90k vs. 121k in fold-in. The hidden table at 40,000 groups is small enough that the whole-table hash join was a minor cost there. The gap is lock serialization: every page locks most groups. Row-lock waiters average 1.9–2.1 backends on the group basis vs. 1.1–1.5 on the ledger, and insert … on conflict (the lock statement) is the top row-lock statement. Closing it needs shorter lock holds or fewer groups per page, not plan control. I suggest filing it as its own issue if the default flip needs parity at that shape.

  • The plan fix shows up where the hidden table is large. At 1M groups, G2a didn't finish draining within the probe's grace window, while G2b drained, at about 8% less CPU per row. That is one clean sample each.
  • The GC's write-path costs (the __zero generated column, and the partial index that changes only on empty/refill transitions) don't show: CPU and WAL per row are unchanged at every size.

Reviewer focus

• collect_zero_groups in staging/retire.rs and collect_zero_groups_sql in staging/group_basis.rs:
  • the predicate __zero and (__basis is null or __stamp <= $1);
  • order by <index cols> limit $2 for update skip locked (LockRows sits below the Limit, so skipped rows don't count against it);
  • EPQ re-checking __zero on a row a page refilled after the GC's snapshot.
• The "no build skip" argument above. It holds only if #1091 G3: Group-basis build and resume (chunked variant B and the one-GROUP-BY build) #1101/#1091 G4: Group-basis TRUNCATE, recompute producers and quarantine #1102's absolute writes lock first and set __stamp in their snapshot's statement (R1, R5). The module doc's "Zero groups" section states that contract.
• group_lookup's per-NULL-pattern branches: the guard is exact (is null / is not null per column), so at most one branch matches.

🤖 Generated with Claude Code