[mmmries]

Part of #1113 (J8a).

Summary

The generative and benchmark tiers for JOIN transforms, first part:

• Generative JOIN tier (generative/tests/join_convergence.rs, generative/src/generate/join.rs): two-table programs (ja, jb) with one to three LEFT/INNER JOIN … GROUP BY definitions (single- or two-column key; grouped by the joined side as J2's S1, by both sides as S2, or by the FROM side; COUNT/SUM/AVG on either side, an expression argument, composed fields), and insert/update/delete streams on both tables with NULL keys, groups and values. Four key shapes: spread (12 keys), hot_key, fan_out (3 keys) and wide (64 keys, bursts always past the 256-row split threshold). Issued through the concurrent runner (bursts of 40-600 ops over 2-4 lanes, 2-6 drain workers, 20-200 ms seals), so writes to both tables race into the same batches and onto the same keys. The oracle is the definition's own JOIN … GROUP BY query, compared byte for byte (oracle::render_join_select); every definition must still be live at the end. The property is #[ignore]d like the others; run it with cargo test -p generative --test join_convergence -- --ignored. Two fast tests run in verify: a hand-built case end to end, and a red control (a corrupted target cell diverges).
• bench join (benchmark/src/streaming/join.rs): Research: JOIN transforms (FROM a [LEFT|INNER] JOIN b ON … GROUP BY …) kept exact without per-row state, as the replacement for relationships #1091 J2's scenario through the real engine, each page one commit, cost = write time + time until a token taken after it converges: D1 (2,000 facts/key) and D2 (10 facts/key), S1 and S2, a-side pages (J2's 30/40/15/15 mix in 200-change groups) and b-side pages (UPDATE gb, unmatched→matched inserts, matched→unmatched deletes), and J2's hot-key run (8 writers on 10 keys, with and without a concurrent dimension writer). The same-session control is J2's B0: next to a, the application keeps a per-fact copy a0 carrying gb/y in the writing transaction (J2's B0 apply, rewriting every fact of a key on a dimension change) and Trellis aggregates a0 both ways, every page applied to every variant in a rotating order. Each dataset ends with oracle checks and a cross-database check (each JOIN target equals the B0 target with its GROUP BY); a summary line says whether D1 kept J2's ordering S1 < S2 < B0 per page kind. Runnable as .claude/scripts/bench join ….

Not in this part: TRUNCATE of either side (halts the JOIN until J5 #1110; TODO in the generator and test doc), rows written before define (the JOIN build is J4 #1109; the run waits for every definition to go live over empty tables before its first op, ConcurrentPlan::settle_after_install). Nightly is disabled project-wide, so no workflow change; the tier runs through the same entry as the other properties.

Design notes

• The generative crate's JOIN unreachables are replaced where the tier needs them: render_definition renders a JOIN, ManualBackend::snapshot reads a JOIN target by its GROUP BY columns, oracle::check routes a JOIN to its own query with no evaluator leg, coverage counts it as Join and doesn't read jb.y as a relationship path.
• ConcurrentPlan::settle_after_install (off for every other tier) is the one change to the shared runner.
• The oracle shares no evaluation code with the engine: it renders the definition's own JOIN … GROUP BY from the AST. The only shared piece is the literal renderer (0 as 0::numeric in COALESCE(SUM(jb.y), 0)).
• Why wide exists. With 12 keys or fewer, every page of a split batch names each key on both sides, so the P and M cell locks order concurrent pages anyway and the J lock is never the only thing between them. J3's mutation checks, run against this tier: dropping the LEFT flip term or using P_new for P_old diverges on the first case (and in the hand-built case); dropping the J lock did not diverge in 60 cases of the three dense shapes, even with a 30 or 200 ms sleep widening the race. With wide it diverges by the eighth case, and the unmutated engine passes 80 cases (26 wide).

Verification

Generative tier (logbook ids):

id	run	result
20261018T001407-b-sweep	smoke, 4 cases	ok, 1,209 ops
20261018T030141-b-sweep	300 cases (bursts 40-200)	ok, no divergence, 110,326 ops; 99 fan-out / 104 hot-key / 97 spread; all 12 definition shapes; 1,321 batches sealed, 0 split across workers (bursts never reached the 256-row split threshold)
20261018T041104-b-sweep	60 cases after widening bursts to 40-600	ok, no divergence, 31,316 ops; 32 batches split, all claimed by 2+ workers (up to 6), in 27 of 60 cases
20261018T050035-b-sweep	review: J3's three mutations against the tier, then 80 cases with wide on the real engine	mutations as above; ok, no divergence, 42,017 ops, 26 wide cases, 52 batches split across workers in 41 of 80 cases

Plus two fast tests in verify: a hand-built case converges, and a corrupted target cell diverges (red control).

bench join (logbook ids 20261018T001431-b-bench tmpfs smoke at 100k facts; 20261018T025034-b-bench failed in the hot-key phase on "too many clients", fixed by running only the measured variant's client there; 20261018T033430-b-bench the run below plus the 1-worker diagnostic). bench --disk join, 2M facts, 8 drain workers, 10 ms seals, all oracle and cross-database checks passed. Medians, page_s:

D1	S1	S2	B0	J2's ordering kept
a-side page (20k changes)	0.50 s	7.27 s	2.63 s	no (S2 > B0)
b-side UPDATE gb (20 rows)	0.031 s	1.25 s	1.89 s	yes
b-side insert (unmatched→matched)	0.026 s	1.84 s	8.49 s	yes
b-side delete (matched→unmatched)	0.028 s	1.56 s	6.11 s	yes

D2 (10 facts/key): a-side S1 1.14 s, S2 3.41 s, B0 4.66 s. b-side S1 0.05-0.06 s, S2 0.09-0.17 s, B0 0.06-0.59 s. Hot keys (8 writers, 160k changes on 10 keys), converged changes/s: S1 10.6k, S2 8.0k, B0 49.7k without a dimension writer. With one: S1 16.0k, S2 6.4k, B0 3.0k (B0's own rewrites deadlocked 53 times against the fact writers, retried).

D1's ordering is reproduced on the b-side but not on the a-side. S2's a-side page costs 2.14 s with one drain worker but 7.27 s with eight (same pages, a separate run). B0 wasn't run with one worker, so whether S2 < B0 holds at one worker is open; against B0's 8-worker 2.63 s, S2's 1-worker page is 1.2x faster, where J2 measured 3.6x. Filed as #1155 (Requires Design Decision, Next Release): a JOIN over few keys gets slower with more drain workers. The bench reports the ordering per page kind (d1_ordering in its summary line). It is a measurement, so it doesn't fail the run.

How to read these numbers:

• B0 keeps both aggregates (S1's and S2's GROUP BY) over a0, as J2's B0 did, while each JOIN variant keeps one. That favours S1 and S2, so the a-side inversion is not caused by the control. The b-side margins are partly B0 paying for two targets.
• The hot-key run is not J2's measurement 5: that one modelled the apply statements in single sessions (351k S1, 67k S2 changes/s, no B0 run). Here the whole engine is measured from first write to convergence, and both JOINs are 5-6x behind B0. The hot-key runs used 8 workers only, so they don't show whether that gap scales with workers the way #1091 JOIN apply: a-side pages slow down as drain workers are added (D1 S2: 7.3 s with 8 workers, 2.1 s with 1) #1155's a-side does.
• D2 does not keep J2's D2 ordering on UPDATE gb pages: B0 0.060 s, S1 0.062 s, S2 0.088 s (J2: S1 0.07 s < S2 0.64 s < B0 1.0 s). The acceptance criterion is D1 only.
• b-side pages of S1 cost 0.03 s, which is close to the 10 ms seal plus the 2 ms poll. They are an upper bound.

verify: VERIFIED, passed=3558 failed=0 ignored=40.

🤖 Generated with Claude Code