Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy by adriangb · Pull Request #24859 · apache/datafusion

adriangb · 2026-09-01T19:07:15Z

Which issue does this PR close?

No existing issue. We found this during an investigation of a production out of memory. We can file one if you want a changelog entry.

Rationale for this change

Reproduce it

Run this in datafusion-cli. It writes 4,000,000 rows in 500,000 groups, with 2,000,000 distinct (g, x) pairs.

COPY (
  SELECT
    value % 500000 AS g,
    'id-' || CAST(value % 2000000 AS VARCHAR) AS x
  FROM generate_series(1, 4000000)
) TO 'repro.parquet' STORED AS PARQUET;

CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION 'repro.parquet';

EXPLAIN FORMAT INDENT SELECT g, count(*), count(DISTINCT x) FROM t GROUP BY g;

On main the rule leaves the aggregate alone:

Projection: t.g, count(Int64(1)) AS count(*), count(DISTINCT t.x)
  Aggregate: groupBy=[[t.g]], aggr=[[count(Int64(1)), count(DISTINCT t.x)]]
    TableScan: t projection=[g, x]

Now put the CREATE EXTERNAL TABLE statement and the query into run.sql, and watch the process:

/usr/bin/time -l datafusion-cli -f run.sql     # macOS; use time -v on Linux

On main this machine reports 16.4 GiB peak RSS and 4.8 s for 500,000 output rows. The count(*) is the only reason the rule stops. Delete it and main rewrites the query.

What this PR changes for that query

The rule now accepts the count(*) and rewrites the aggregate:

Projection: t.g, CASE WHEN sum(alias2) IS NOT NULL THEN sum(alias2) ELSE Int64(0) END AS count(*), count(alias1) AS count(DISTINCT t.x)
  Aggregate: groupBy=[[t.g]], aggr=[[sum(alias2), count(alias1)]]
    Aggregate: groupBy=[[t.g, t.x AS alias1]], aggr=[[count(Int64(1)) AS alias2]]
      TableScan: t projection=[g, x]

The same run reports 335 MiB peak RSS and 0.07 s. The results are identical.

Read that 50x with care. It is the number on main today, and main does not carry #24857. #24857 removes a per-group pre-allocation of about 33.8 KiB from the path this rewrite exists to avoid. Stacked on #24857 the same comparison gives 1.82x, not 50x. The table below gives both columns.

datafusion.execution.parquet.schema_force_view_types defaults to true, so x above arrives as Utf8View.

Why the rewrite helps here

SingleDistinctToGroupBy rewrites AGG(DISTINCT x) into a two phase group by. The rule accepts a non-distinct sum, min or max next to the distinct aggregate. It rejects a non-distinct count. One count(*) is therefore enough to keep the unrewritten plan.

We hit this in production. A query of this shape drove a process to 10.98 GB and to death.

The rewrite is not free. Every other aggregate moves down into the inner group by. That group by holds one row for each (group, distinct value) pair, not one row for each group. Each aggregate then keeps its state at that finer grain. The rewrite pays for this cost when it takes the distinct aggregate off GroupsAccumulatorAdapter. The adapter keeps one boxed Accumulator for each group, which is the expensive shape.

count(DISTINCT x) has a specialized GroupsAccumulator for each integer type, and for no other type. An integer distinct count never reaches the adapter, so the rewrite buys nothing for it.

The measurements

The harness reports the peak MemoryPool reservation for SELECT g, count(*), count(DISTINCT x) FROM t GROUP BY g over 4,000,000 rows. The first column is today's main. The second column is the same measurement on a tree that also carries #24857.

groups	distinct argument	on `main`	with #24857
500,000	`BIGINT`	1.14x worse	1.14x worse
500,000	`Utf8`	15.1x better	1.67x better
500,000	`Utf8View`	44.1x better	1.82x better
2,000	`BIGINT`	1.33x worse	1.33x worse
2,000	`Utf8`	1.04x worse	1.11x better
2,000	`Utf8View`	1.18x better	1.30x better

The second column is the honest one, and it is much smaller than the first. The absolute saving in the motivating cell falls from 13.4 GiB to 181 MiB, which is about 75 times less. An earlier version of this description claimed 55x. That number is an artifact of a pre-allocation that goes away when #24857 lands.

Two notes on the table.

The type labels in the earlier description were wrong. A string column that comes from a Parquet file is Utf8View, not Utf8, so the rows labelled VARCHAR were Utf8View rows. The two types are far apart on main. At 500,000 groups the unrewritten arm costs 4.00 GiB for Utf8 and 13.67 GiB for Utf8View.

#24857 also closes the hole at 2,000 groups, and for an unexpected reason. The unrewritten arm gets larger there, from 261 MiB to 303 MiB. #24857 removes the pre-allocation, and it also makes size() report the real hashbrown allocation. At 2,000 distinct values per group the honest accounting is larger than the pre-allocation it removes.

Where the rewrite stops paying

There is no crossover in group count for a string argument. The peak of the rewritten arm follows the number of distinct pairs, which does not change with the group count. The peak of the unrewritten arm grows with the group count.

The crossover is in the density of distinct values. It arrives only when every row holds a distinct value. There the residual Utf8 loss is 1.06x to 1.08x, and Utf8View does not cross at all.

The gate is a proxy

The gate asks which accumulator the distinct aggregate gets. That is not the true discriminator.

What decides the outcome is the cost per distinct value on each side. The rewrite materializes one hash table row for each distinct (group keys, x) pair, which is about 48 bytes, plus one accumulator slot for each companion aggregate at that grain. The rewrite wins when the unrewritten accumulator costs more than that for each value. It loses when the accumulator costs less.

The proxy and the true discriminator agree for count(DISTINCT x), which is the only case this PR opens. They disagree elsewhere, and the pre-existing arms of this rule carry that disagreement:

sum(DISTINCT int_col) and avg(DISTINCT int_col) have no distinct groups accumulator. They still regress 3.15x, at 71.0 MiB unrewritten against 223.4 MiB rewritten, over 4,000,000 rows in 2,000 groups.
min(DISTINCT x) is worse. min(DISTINCT x) is the same value as min(x), and min_max correctly ignores is_distinct. The unrewritten plan holds one scalar for each group, and the rewrite builds a 4,000,000 row hash table. We measured up to 1113x more peak memory.

That regression predates this PR, and this PR does not extend it. The gate keeps every one of those functions out of the path this PR adds. We report the broader regression upstream separately. A cost model is out of scope here.

What changes are included in this PR?

This PR allows a non-distinct count next to the distinct aggregate, when the distinct aggregate reports that it has no specialized GroupsAccumulator for its argument types.

count is the one supported function whose outer phase is a different function. The inner group by counts the rows of each (group, distinct value) partition. The outer phase adds those partial counts with sum, because a count over a group is the sum of the counts of any partition of that group.

Two details follow from that substitution.

count and sum come from the session function registry, as replace_distinct_aggregate already does for first_value. The rewrite fires only for that exact count, compared by identity and not by name. A session without a registry, or with its own count, keeps the previous behaviour.

count returns a non-null 0 over an empty input, and sum of no rows returns NULL. An aggregate without a GROUP BY reaches that case. The inner aggregate emits no rows and the outer aggregate still emits one row, so SELECT count(*), count(DISTINCT x) FROM empty would return NULL, 0 instead of 0, 0. The projection selects CASE WHEN sum(alias) IS NOT NULL THEN sum(alias) ELSE 0 END. That restores the 0, and it keeps the type and the nullability that count had.

FILTER and ORDER BY still block the rewrite.

The gate

An optimizer rule has no AccumulatorArgs to ask AggregateUDFImpl::groups_accumulator_supported with. datafusion-optimizer also cannot depend on datafusion-functions-aggregate to read count's list of types. datafusion/optimizer/Cargo.toml names the intended way out:

If you want to add special handling for a specific function, use the methods on the ScalarUDFImpl or AggregateUDFImpl traits (or add a new method to those traits).

So this PR adds AggregateUDFImpl::groups_accumulator_supported_for_types(&[DataType], is_distinct) -> Option<bool>. Count is the only implementor. Count::groups_accumulator_supported now delegates to it, so there is one list of supported types and not two that can drift.

The default is None, which means the implementation does not answer the question. None is not a third answer, and a caller must not read it as either Some(true) or Some(false). This rule rewrites only on Some(false), which is the one answer that reports a call on the adapter. An aggregate that answers None, which today is every aggregate except count, keeps the previous behaviour.

The gate covers only the count this PR adds. A plan that already qualifies through a non-distinct sum, min or max is rewritten as before, over any distinct argument type. A narrower rule there would change plans that this repository has always rewritten, and nothing measured here calls for that change.

Files touched beyond the rule itself:

datafusion/expr/src/udaf.rs and datafusion/functions-aggregate/src/count.rs: the new trait method and its one implementation.
datafusion/sqllogictest/test_files/single_distinct_to_groupby.slt: the new coverage described below.
datafusion/substrait/tests/cases/roundtrip_logical_plan.rs: aggregate_distinct_with_having now builds its session without this rule, so it keeps round tripping the plan shape that the test was written for. Its distinct argument is a DECIMAL, so the gate does not exclude it.

No existing snapshot in the repository changes.

What is the testing strategy for this PR?

single_distinct_to_groupby.slt asserts every result twice, once under datafusion.optimizer.max_passes = 0 and once under the default, with identical expected blocks. A null-handling error or a type error therefore appears as a result mismatch, and not only as a plan difference.

The table carries the same values in a VARCHAR column and in an INT column, which are the two sides of the gate. The file asserts both sides. The VARCHAR distinct rewrites with a count next to it. The INT distinct does not. The INT distinct still rewrites when it qualifies through sum. The results agree with the unoptimized plan in every case.

The file also asserts that an aggregate which answers None stays unrewritten. sum(DISTINCT v) and min(DISTINCT v) next to a count(*) both keep the plan they have on main.

The remaining coverage is count(*) against count(1) against count(col), grouped and ungrouped, a group whose distinct column is entirely NULL, a group with NULLs in both the distinct column and the summed column, empty input in three shapes, HAVING with ORDER BY on the rewritten count, and the production join shape.

The unit tests of the rule cover both sides of the gate directly. They also cover the None answer twice: once with sum(DISTINCT b), and once with a test aggregate that leaves the new method at its default.

Benchmarks

Q22 is the only ClickBench query whose plan this PR can change, and under the gate it no longer does. Its count(DISTINCT "UserID") is over an Int64. Q9 (RegionID, SUM, COUNT(*), AVG, COUNT(DISTINCT UserID)) still bails out, because AVG disqualifies it. Q8, Q10, Q11 and Q13 have a lone distinct aggregate and main already rewrites them. The gate does not touch them.

Latency. Measured on clickbench_partitioned (100 files, about 100M rows) before the gate, when Q22 did change. A run-level A/B cannot resolve a change this small. A comparison of the base binary against itself with compare.py reported 9 queries faster, 28 slower and 6 unchanged, with swings up to 1.58x. A paired comparison over 40 repetitions, with the 42 unchanged queries as an in-experiment control, put Q22 at -2.03%, 95% CI [-5.48%, +1.39%]. There is no measurable latency difference, and this setup resolves effects of about 3% and no better. That agrees with #11360, which found the removal of the whole rule to be a wash.

Memory. A memory-limited run (DATAFUSION_RUNTIME_MEMORY_LIMIT: 4G) of the ungated version produced the gate. The peak MemoryPool reservation of Q22 went from 3.1 MiB to 7.3 MiB, which is +132.2%, and Q22 was the only plan that changed. That is the integer case above. The base plan already had PrimitiveDistinctCountGroupsAccumulator, and the rewrite replaced it with an inner group by on (SearchPhrase, UserID) that also carries min("URL") and min("Title") at that grain.

Re-benchmarked with the gate, same benchmark and same 4G limit, three independent runs from one trigger against the same merge base (1, 2, 3):

Run	Q22 base	Q22 changed	Change
1	4.2 MiB	3.1 MiB	-24.4%
2	3.0 MiB	2.6 MiB	-13.8%
3	4.8 MiB	3.0 MiB	-37.3%

The +132.2% is gone. The plan of Q22 is now the base plan byte for byte, so the two sides differ only by measurement noise. Do not read any of it as an improvement. The noise on a 3 MiB high-water mark reported to 0.1 MiB is large, and the base side alone spans 3.0 MiB to 4.8 MiB across three runs of identical code. For scale, Q17 (GROUP BY "UserID", "SearchPhrase", no DISTINCT, so this PR cannot change its plan) moved +6.3%, +26.7% and +6.9% on a 2 GiB reservation in those same three runs. No query moved consistently in the regressing direction.

Those runs are on d510dd4. The commits after it change a doc comment, the treatment of an unanswered gate question, and tests. No ClickBench query holds a distinct sum, min, max or avg, so none of those commits can change a ClickBench plan.

Are there any user-facing changes?

There is no public API change and no change to query results. AggregateUDFImpl gains one method with a default, which is not a breaking change for implementors.

Plans for SELECT ..., count(...), count(DISTINCT x) ... GROUP BY ... change shape when x has no specialized GroupsAccumulator. EXPLAIN output for that shape therefore differs, and such queries should use less memory. Read the size of that win from the second column of the table above, and not from the first.

The size of the win also depends on the group count, because the cost of the adapter is per group. The rule has always had that property. This PR does not change it, and the optimizer has no group cardinality estimate to gate on.

`SingleDistinctToGroupBy` rewrites `AGG(DISTINCT x)` into a two phase group by, which is what keeps a high cardinality distinct off the one-accumulator-per-group path in `GroupsAccumulatorAdapter`. The rule tolerated a non-distinct `sum`, `min` or `max` next to the distinct aggregate but bailed out on `count`, so the very common `count(*), count(DISTINCT x) ... GROUP BY` shape kept the unrewritten plan and its memory profile. Allow a non-distinct `count` as well. `count` is the one supported function whose outer phase is a different function: the inner group by counts the rows of each `(group, distinct value)` partition and the outer phase adds those partial counts up with `sum`, since count over a group is the sum of the counts of any partition of that group. Two details follow from that substitution: - `count` and `sum` are resolved from the session function registry and the rewrite only fires when the aggregate is that exact `count`, so a session without a registry or with its own `count` is left alone. - `count` returns a non-null 0 over an empty input while `sum` of no rows is NULL, which is reachable for an aggregate with no group by. The projection selects `CASE WHEN sum(alias) IS NOT NULL THEN sum(alias) ELSE 0 END`, which restores the 0 and keeps the column's type and nullability as `count` had them. The new sqllogictest file asserts every result twice, once with the optimizer disabled and once with it enabled, over data with NULL and all-NULL distinct values, an empty input, and `count(*)` versus `count(col)` versus `count(1)`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`aggregate_distinct_with_having` round trips `SELECT a, count(distinct b) ... HAVING count(b) > 100` through substrait and asserts the plan comes back displaying identically. It passed only because the non-distinct `count` made `SingleDistinctToGroupBy` bail out, so the plan had no aliases in it. With the rule now allowing that `count`, the query is rewritten and the assertion fails. The failure is a pre-existing substrait gap rather than anything specific to this query: substrait carries no names for an aggregate's grouping and measure expressions, so the consumer derives them from the expressions themselves and the `alias1` and `alias2` names the rule introduces are lost. Any plan the rule rewrites fails the same way, including the plain `SELECT a, count(distinct b) FROM data GROUP BY a, c` that this change does not touch. Remove the rule from the session used by this one test, so it keeps covering the un-rewritten aggregate it was written for instead of depending on the rule bailing out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

codecov-commenter · 2026-09-01T19:15:44Z

Codecov Report

❌ Patch coverage is 84.48753% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.15%. Comparing base (da89c7c) to head (d3dbc09).
⚠️ Report is 30 commits behind head on main.

Files with missing lines	Patch %	Lines
...fusion/optimizer/src/single_distinct_to_groupby.rs	84.95%	18 Missing and 30 partials ⚠️
datafusion/expr/src/udaf.rs	65.21%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24859      +/-   ##
==========================================
+ Coverage   81.60%   82.15%   +0.54%     
==========================================
  Files        1123     1123              
  Lines      408898   419189   +10291     
  Branches   408898   419189   +10291     
==========================================
+ Hits       333670   344368   +10698     
+ Misses      55625    55482     -143     
+ Partials    19603    19339     -264

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

adriangb · 2026-09-01T20:47:58Z

run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 4G

adriangbot · 2026-09-01T20:50:53Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5500225568-2072-x9l4r 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/single-distinct-to-groupby-allow-count (f47c045) to da89c7c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
env:
  DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-01T21:12:08Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing claude/single-distinct-to-groupby-allow-count (f47c045) to da89c7c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
env:
  DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ claude_single-distinct-to-groupby-allow-count ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │    1.38 ms │                                       1.22 ms │ +1.12x faster │
│ QQuery 1  │   12.34 ms │                                      11.86 ms │     no change │
│ QQuery 2  │   37.35 ms │                                      36.73 ms │     no change │
│ QQuery 3  │   33.25 ms │                                      31.14 ms │ +1.07x faster │
│ QQuery 4  │  265.05 ms │                                     221.44 ms │ +1.20x faster │
│ QQuery 5  │  276.49 ms │                                     269.41 ms │     no change │
│ QQuery 6  │    1.28 ms │                                       1.28 ms │     no change │
│ QQuery 7  │   13.28 ms │                                      13.09 ms │     no change │
│ QQuery 8  │  337.55 ms │                                     330.68 ms │     no change │
│ QQuery 9  │  452.80 ms │                                     447.81 ms │     no change │
│ QQuery 10 │   69.67 ms │                                      69.38 ms │     no change │
│ QQuery 11 │   80.87 ms │                                      80.40 ms │     no change │
│ QQuery 12 │  265.72 ms │                                     266.16 ms │     no change │
│ QQuery 13 │  978.95 ms │                                     959.23 ms │     no change │
│ QQuery 14 │  281.40 ms │                                     287.72 ms │     no change │
│ QQuery 15 │  266.83 ms │                                     260.92 ms │     no change │
│ QQuery 16 │ 1228.08 ms │                                    1196.31 ms │     no change │
│ QQuery 17 │  916.26 ms │                                     890.67 ms │     no change │
│ QQuery 18 │ 2497.04 ms │                                    2464.68 ms │     no change │
│ QQuery 19 │   28.04 ms │                                      30.09 ms │  1.07x slower │
│ QQuery 20 │  528.22 ms │                                     524.86 ms │     no change │
│ QQuery 21 │  516.39 ms │                                     512.66 ms │     no change │
│ QQuery 22 │  980.18 ms │                                     974.68 ms │     no change │
│ QQuery 23 │ 3061.15 ms │                                    3010.47 ms │     no change │
│ QQuery 24 │   42.00 ms │                                      41.44 ms │     no change │
│ QQuery 25 │  110.73 ms │                                     109.66 ms │     no change │
│ QQuery 26 │   42.25 ms │                                      41.12 ms │     no change │
│ QQuery 27 │  511.60 ms │                                     509.48 ms │     no change │
│ QQuery 28 │ 2913.69 ms │                                    2885.57 ms │     no change │
│ QQuery 29 │   41.01 ms │                                      41.29 ms │     no change │
│ QQuery 30 │  298.78 ms │                                     296.46 ms │     no change │
│ QQuery 31 │  273.81 ms │                                     284.70 ms │     no change │
│ QQuery 32 │ 3254.62 ms │                                    3309.85 ms │     no change │
│ QQuery 33 │ 2515.41 ms │                                    2534.95 ms │     no change │
│ QQuery 34 │ 2659.40 ms │                                    2562.43 ms │     no change │
│ QQuery 35 │  280.03 ms │                                     278.34 ms │     no change │
│ QQuery 36 │   65.80 ms │                                      65.81 ms │     no change │
│ QQuery 37 │   35.17 ms │                                      35.47 ms │     no change │
│ QQuery 38 │   39.97 ms │                                      40.54 ms │     no change │
│ QQuery 39 │  133.36 ms │                                     130.60 ms │     no change │
│ QQuery 40 │   13.78 ms │                                      13.90 ms │     no change │
│ QQuery 41 │   13.60 ms │                                      13.63 ms │     no change │
│ QQuery 42 │   12.82 ms │                                      13.08 ms │     no change │
└───────────┴────────────┴───────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                            ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                            │ 26387.39ms │
│ Total Time (claude_single-distinct-to-groupby-allow-count)   │ 26101.22ms │
│ Average Time (HEAD)                                          │   613.66ms │
│ Average Time (claude_single-distinct-to-groupby-allow-count) │   607.01ms │
│ Queries Faster                                               │          3 │
│ Queries Slower                                               │          1 │
│ Queries with No Change                                       │         39 │
│ Queries with Failure                                         │          0 │
└──────────────────────────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                   HEAD ┃ claude_single-distinct-to-groupby-allow-count ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │           1.38 / 4.40 ±5.86 / 16.12 ms │                  1.22 / 3.94 ±5.36 / 14.66 ms │ +1.12x faster │
│ QQuery 1  │         12.34 / 12.80 ±0.27 / 13.14 ms │                11.86 / 12.01 ±0.12 / 12.20 ms │ +1.07x faster │
│ QQuery 2  │         37.35 / 37.99 ±0.55 / 38.93 ms │                36.73 / 36.90 ±0.12 / 37.05 ms │     no change │
│ QQuery 3  │         33.25 / 34.03 ±0.82 / 35.47 ms │                31.14 / 31.35 ±0.14 / 31.54 ms │ +1.09x faster │
│ QQuery 4  │      265.05 / 269.97 ±4.66 / 276.23 ms │             221.44 / 231.96 ±8.16 / 245.31 ms │ +1.16x faster │
│ QQuery 5  │     276.49 / 288.95 ±13.08 / 309.95 ms │             269.41 / 274.36 ±4.16 / 280.38 ms │ +1.05x faster │
│ QQuery 6  │            1.28 / 1.41 ±0.20 / 1.81 ms │                   1.28 / 1.44 ±0.24 / 1.91 ms │     no change │
│ QQuery 7  │         13.28 / 13.35 ±0.06 / 13.47 ms │                13.09 / 13.20 ±0.07 / 13.29 ms │     no change │
│ QQuery 8  │     337.55 / 351.39 ±10.86 / 361.65 ms │           330.68 / 414.89 ±159.42 / 733.70 ms │  1.18x slower │
│ QQuery 9  │     452.80 / 476.14 ±19.03 / 503.53 ms │             447.81 / 454.59 ±6.52 / 466.88 ms │     no change │
│ QQuery 10 │         69.67 / 75.80 ±7.42 / 87.62 ms │                69.38 / 72.74 ±4.97 / 82.59 ms │     no change │
│ QQuery 11 │         80.87 / 81.53 ±0.43 / 82.17 ms │                80.40 / 81.73 ±2.17 / 86.04 ms │     no change │
│ QQuery 12 │      265.72 / 273.17 ±6.19 / 281.39 ms │             266.16 / 269.76 ±3.69 / 275.76 ms │     no change │
│ QQuery 13 │      978.95 / 984.79 ±4.26 / 992.10 ms │            959.23 / 972.59 ±11.09 / 989.55 ms │     no change │
│ QQuery 14 │      281.40 / 289.20 ±5.44 / 297.13 ms │            287.72 / 311.18 ±15.57 / 333.71 ms │  1.08x slower │
│ QQuery 15 │      266.83 / 271.93 ±2.95 / 275.50 ms │            260.92 / 274.45 ±10.73 / 292.43 ms │     no change │
│ QQuery 16 │  1228.08 / 1270.69 ±27.81 / 1314.41 ms │         1196.31 / 1231.12 ±19.58 / 1251.73 ms │     no change │
│ QQuery 17 │     916.26 / 944.27 ±23.43 / 979.60 ms │            890.67 / 931.33 ±22.36 / 957.69 ms │     no change │
│ QQuery 18 │  2497.04 / 2549.71 ±59.07 / 2659.37 ms │        2464.68 / 2619.87 ±114.28 / 2736.65 ms │     no change │
│ QQuery 19 │         28.04 / 29.82 ±2.46 / 34.68 ms │                30.09 / 30.96 ±0.56 / 31.67 ms │     no change │
│ QQuery 20 │      528.22 / 535.30 ±4.58 / 540.87 ms │            524.86 / 548.63 ±18.95 / 574.17 ms │     no change │
│ QQuery 21 │      516.39 / 520.17 ±3.16 / 525.39 ms │             512.66 / 517.24 ±4.36 / 525.02 ms │     no change │
│ QQuery 22 │      980.18 / 986.55 ±4.83 / 992.72 ms │            974.68 / 988.63 ±9.98 / 1003.70 ms │     no change │
│ QQuery 23 │  3061.15 / 3176.07 ±88.50 / 3302.98 ms │         3010.47 / 3045.11 ±30.25 / 3090.12 ms │     no change │
│ QQuery 24 │         42.00 / 42.34 ±0.44 / 43.20 ms │                41.44 / 43.22 ±3.08 / 49.38 ms │     no change │
│ QQuery 25 │      110.73 / 116.59 ±7.38 / 130.21 ms │             109.66 / 113.03 ±3.80 / 120.17 ms │     no change │
│ QQuery 26 │         42.25 / 43.27 ±0.79 / 44.67 ms │                41.12 / 41.64 ±0.66 / 42.93 ms │     no change │
│ QQuery 27 │      511.60 / 515.42 ±3.73 / 522.37 ms │             509.48 / 514.21 ±4.78 / 521.97 ms │     no change │
│ QQuery 28 │  2913.69 / 2942.96 ±24.29 / 2976.11 ms │         2885.57 / 2943.07 ±47.48 / 3016.49 ms │     no change │
│ QQuery 29 │        41.01 / 52.70 ±10.75 / 69.66 ms │                41.29 / 46.47 ±8.04 / 62.43 ms │ +1.13x faster │
│ QQuery 30 │      298.78 / 305.77 ±4.66 / 313.00 ms │            296.46 / 317.23 ±30.89 / 378.34 ms │     no change │
│ QQuery 31 │      273.81 / 287.28 ±7.37 / 294.32 ms │             284.70 / 291.53 ±4.80 / 296.70 ms │     no change │
│ QQuery 32 │  3254.62 / 3304.71 ±50.88 / 3397.00 ms │        3309.85 / 3514.92 ±160.53 / 3712.98 ms │  1.06x slower │
│ QQuery 33 │ 2515.41 / 2697.29 ±162.83 / 2974.36 ms │         2534.95 / 2607.56 ±78.80 / 2758.34 ms │     no change │
│ QQuery 34 │ 2659.40 / 2764.19 ±119.47 / 2950.04 ms │         2562.43 / 2636.57 ±48.78 / 2684.60 ms │     no change │
│ QQuery 35 │      280.03 / 288.16 ±6.99 / 297.41 ms │            278.34 / 289.88 ±10.71 / 308.26 ms │     no change │
│ QQuery 36 │         65.80 / 72.10 ±3.65 / 75.62 ms │                65.81 / 69.11 ±2.82 / 72.45 ms │     no change │
│ QQuery 37 │         35.17 / 35.91 ±0.46 / 36.61 ms │               35.47 / 44.91 ±17.07 / 79.01 ms │  1.25x slower │
│ QQuery 38 │       39.97 / 54.20 ±24.14 / 102.34 ms │                40.54 / 43.23 ±1.55 / 45.17 ms │ +1.25x faster │
│ QQuery 39 │      133.36 / 140.02 ±4.66 / 147.49 ms │             130.60 / 134.20 ±3.25 / 140.32 ms │     no change │
│ QQuery 40 │         13.78 / 14.17 ±0.25 / 14.52 ms │                13.90 / 14.46 ±0.38 / 15.06 ms │     no change │
│ QQuery 41 │         13.60 / 13.82 ±0.18 / 14.12 ms │                13.63 / 14.37 ±0.88 / 16.03 ms │     no change │
│ QQuery 42 │         12.82 / 18.13 ±9.98 / 38.07 ms │                13.08 / 14.22 ±1.63 / 17.42 ms │ +1.27x faster │
└───────────┴────────────────────────────────────────┴───────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                            ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                            │ 27188.49ms │
│ Total Time (claude_single-distinct-to-groupby-allow-count)   │ 27063.85ms │
│ Average Time (HEAD)                                          │   632.29ms │
│ Average Time (claude_single-distinct-to-groupby-allow-count) │   629.39ms │
│ Queries Faster                                               │          8 │
│ Queries Slower                                               │          4 │
│ Queries with No Change                                       │         31 │
│ Queries with Failure                                         │          0 │
└──────────────────────────────────────────────────────────────┴────────────┘

Memory Pool Peaks

Peak MemoryPool reservation per query — what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with DATAFUSION_RUNTIME_MEMORY_LIMIT set.

Base: da89c7c (merge-base) | Changed: claude/single-distinct-to-groupby-allow-count

clickbench_partitioned

Query	Base	Changed	Change
Query 0	0 B	0 B	0.0%
Query 1	104 B	104 B	+0.0%
Query 2	936 B	936 B	+0.0%
Query 3	312 B	312 B	+0.0%
Query 4	757.7 MiB	741.3 MiB	-2.2%
Query 5	1.2 GiB	1.2 GiB	-0.4%
Query 6	0 B	0 B	0.0%
Query 7	40.1 MiB	60.2 MiB	+50.1%
Query 8	873.1 MiB	852.4 MiB	-2.4%
Query 9	551.7 MiB	551.7 MiB	+0.0%
Query 10	107.6 MiB	111.0 MiB	+3.2%
Query 11	113.2 MiB	115.9 MiB	+2.4%
Query 12	1.3 GiB	1.3 GiB	-0.2%
Query 13	998.5 MiB	1.1 GiB	+8.5%
Query 14	1.3 GiB	1.3 GiB	-1.3%
Query 15	1.1 GiB	1.2 GiB	+1.9%
Query 16	1.9 GiB	1.7 GiB	-10.3%
Query 17	2.0 GiB	1.9 GiB	-1.2%
Query 18	2.0 GiB	2.0 GiB	-2.9%
Query 19	0 B	0 B	0.0%
Query 20	104 B	104 B	+0.0%
Query 21	3.3 MiB	3.3 MiB	-0.0%
Query 22	3.1 MiB	7.3 MiB	+132.2%
Query 23	26.8 MiB	25.7 MiB	-4.0%
Query 24	58.8 MiB	58.6 MiB	-0.4%
Query 25	170.3 MiB	173.2 MiB	+1.7%
Query 26	59.9 MiB	60.3 MiB	+0.6%
Query 27	2.2 MiB	2.2 MiB	+0.0%
Query 28	1.5 GiB	1.4 GiB	-3.7%
Query 29	624 B	624 B	+0.0%
Query 30	698.5 MiB	707.1 MiB	+1.2%
Query 31	1.5 GiB	1.5 GiB	+3.2%
Query 32	926.9 MiB	966.9 MiB	+4.3%
Query 33	2.0 GiB	2.1 GiB	+5.2%
Query 34	2.1 GiB	2.1 GiB	-2.6%
Query 35	612.0 MiB	622.3 MiB	+1.7%
Query 36	113.6 MiB	111.1 MiB	-2.2%
Query 37	6.9 MiB	6.9 MiB	+0.0%
Query 38	5.2 MiB	5.2 MiB	-0.4%
Query 39	297.5 MiB	297.8 MiB	+0.1%
Query 40	2.0 MiB	1.8 MiB	-8.3%
Query 41	3.1 MiB	3.1 MiB	+0.0%
Query 42	1.6 MiB	1.6 MiB	+5.1%

Pool accounting vs. process RSS

Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.

Benchmark	Side	Max pool peak	Peak RSS	Gap	RSS / pool
clickbench_partitioned	base (`da89c7c (merge-base)`)	2.1 GiB	8.7 GiB	6.6 GiB	4.1×
clickbench_partitioned	changed (`claude/single-distinct-to-groupby-allow-count`)	2.1 GiB	8.6 GiB	6.4 GiB	4.0×

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	140.0s
Peak memory	8.7 GiB
Avg memory	5.3 GiB
CPU user	1382.5s
CPU sys	131.6s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	140.0s
Peak memory	8.6 GiB
Avg memory	5.1 GiB
CPU user	1376.6s
CPU sys	132.7s
Peak spill	0 B

File an issue against this benchmark runner

The two grouped `COUNT(DISTINCT <string>)` memory limit tests only reach the per group accumulators while `single_distinct_aggregation_to_group_by` declines to rewrite the query. They leant on `count(*)` for that, which the rule rejects only because `count` is missing from the `sum`/`min`/`max` allow list. apache#24859 proposes adding `count` to that list, which would rewrite the query, remove the accumulators, and leave both tests passing at any memory limit while still looking like they test something. Aggregate `avg(payload)` over a new `Int64` column instead. `avg` cannot be added to that list: the rule re-aggregates its own partial results over the deduplicated inner group by, and averaging per group averages of different sizes gives the wrong answer. That is why ClickBench Q9 keeps its distinct aggregate under apache#24859. Verified from the physical plan with apache#24859 cherry-picked on top of this branch: the `avg` query still plans as `aggr=[count(DISTINCT t.value), avg(t.payload)]`, while the `count(*)` query becomes `aggr=[count(alias1), sum(alias2)]` over an inner `GROUP BY group_key, value`, and drops from needing ~1.9 MB to ~0.9 MB. Re-swept both thresholds against the base commit. `Utf8` needs ~35.5 MB before and ~1.9 MB after; `Utf8View` needs ~123 MB before and ~2.5 MB after, so the 8 MB and 16 MB limits keep at least 4x margin on each side and are unchanged.

A memory limited ClickBench run contradicted the memory rationale for the case it measures. Q22 is `SELECT "SearchPhrase", MIN("URL"), MIN("Title"), COUNT(*), COUNT(DISTINCT "UserID") ... GROUP BY "SearchPhrase"`, the only plan the change touched, and its peak memory pool reservation went from 3.1 MiB to 7.3 MiB, up 132.2%, with neighbouring queries moving by a few percent either way. `UserID` is an `Int64`, and `Count::groups_accumulator_supported` returns true for every integer type and false for everything else. So the base plan already had `PrimitiveDistinctCountGroupsAccumulator` and never went near `GroupsAccumulatorAdapter`. The rewrite replaced a compact vectorized accumulator with an inner group by on `(SearchPhrase, UserID)`, which also carries `min("URL")` and `min("Title")` at that much finer grain, and bought nothing back. Measured locally on 4M rows, 500k groups and 2M distinct pairs, peak pool reservation for `SELECT g, count(*), count(DISTINCT x) FROM t GROUP BY g`: x BIGINT 135 MiB unrewritten, 205 MiB rewritten x VARCHAR 13.6 GiB unrewritten, 255 MiB rewritten Peak RSS for the VARCHAR pair was 18.0 GiB against 433 MiB, so the pool figure is real memory rather than an accounting artifact. The rewrite pays exactly when the distinct argument would otherwise land on the adapter. `datafusion-optimizer` cannot depend on `datafusion-functions-aggregate` to read that list of types, and has no `AccumulatorArgs` to ask `groups_accumulator_supported` with. Its `Cargo.toml` names the way out, so this adds `AggregateUDFImpl::groups_accumulator_supported_for_types`, defaulting to false as `groups_accumulator_supported` does. `Count` is the only implementor and its physical method now delegates to it, so the two cannot disagree. The gate covers only the `count` this branch added. A plan that already qualified through a non-distinct `sum`, `min` or `max` is rewritten exactly as before, over any distinct argument type. The ClickBench snapshot returns to its base form, so no existing snapshot in the repository changes. `single_distinct_to_groupby.slt` now carries the same values in a `VARCHAR` and an `INT` column and asserts both sides of the gate, still under both `datafusion.optimizer.max_passes = 0` and the default.

adriangb · 2026-09-02T18:56:54Z

run benchmark clickbench_partitioned clickbench_partitioned clickbench_partitioned

env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 4G

adriangbot · 2026-09-02T18:59:41Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5514755290-2103-9pm4r 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/single-distinct-to-groupby-allow-count (d510dd4) to da89c7c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
env:
  DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-02T18:59:50Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5514755290-2104-xv7bw 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/single-distinct-to-groupby-allow-count (d510dd4) to da89c7c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
env:
  DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-02T19:00:08Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5514755290-2105-kkpnq 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/single-distinct-to-groupby-allow-count (d510dd4) to da89c7c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
env:
  DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-02T19:20:17Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing claude/single-distinct-to-groupby-allow-count (d510dd4) to da89c7c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
env:
  DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ claude_single-distinct-to-groupby-allow-count ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │    1.22 ms │                                       1.21 ms │     no change │
│ QQuery 1  │   11.84 ms │                                      11.85 ms │     no change │
│ QQuery 2  │   37.10 ms │                                      36.43 ms │     no change │
│ QQuery 3  │   31.32 ms │                                      31.30 ms │     no change │
│ QQuery 4  │  227.95 ms │                                     223.17 ms │     no change │
│ QQuery 5  │  278.44 ms │                                     273.19 ms │     no change │
│ QQuery 6  │    1.28 ms │                                       1.27 ms │     no change │
│ QQuery 7  │   13.60 ms │                                      13.41 ms │     no change │
│ QQuery 8  │  334.97 ms │                                     336.15 ms │     no change │
│ QQuery 9  │  462.84 ms │                                     455.82 ms │     no change │
│ QQuery 10 │   70.72 ms │                                      70.27 ms │     no change │
│ QQuery 11 │   82.51 ms │                                      81.12 ms │     no change │
│ QQuery 12 │  271.21 ms │                                     270.32 ms │     no change │
│ QQuery 13 │  990.69 ms │                                     971.51 ms │     no change │
│ QQuery 14 │  285.44 ms │                                     284.86 ms │     no change │
│ QQuery 15 │  271.87 ms │                                     263.20 ms │     no change │
│ QQuery 16 │ 1224.39 ms │                                    1207.65 ms │     no change │
│ QQuery 17 │  938.17 ms │                                     928.40 ms │     no change │
│ QQuery 18 │ 2525.15 ms │                                    2495.36 ms │     no change │
│ QQuery 19 │   28.34 ms │                                      27.90 ms │     no change │
│ QQuery 20 │  525.20 ms │                                     523.29 ms │     no change │
│ QQuery 21 │  517.57 ms │                                     515.53 ms │     no change │
│ QQuery 22 │  988.34 ms │                                     983.42 ms │     no change │
│ QQuery 23 │ 3019.56 ms │                                    3020.27 ms │     no change │
│ QQuery 24 │   41.26 ms │                                      41.15 ms │     no change │
│ QQuery 25 │  109.96 ms │                                     110.13 ms │     no change │
│ QQuery 26 │   42.24 ms │                                      41.22 ms │     no change │
│ QQuery 27 │  512.82 ms │                                     522.42 ms │     no change │
│ QQuery 28 │ 2944.09 ms │                                    2916.45 ms │     no change │
│ QQuery 29 │   41.01 ms │                                      41.20 ms │     no change │
│ QQuery 30 │  303.22 ms │                                     300.73 ms │     no change │
│ QQuery 31 │  289.32 ms │                                     282.18 ms │     no change │
│ QQuery 32 │ 3284.67 ms │                                    3283.17 ms │     no change │
│ QQuery 33 │ 2556.04 ms │                                    2592.71 ms │     no change │
│ QQuery 34 │ 2630.88 ms │                                    2647.50 ms │     no change │
│ QQuery 35 │  338.93 ms │                                     284.82 ms │ +1.19x faster │
│ QQuery 36 │   69.03 ms │                                      64.59 ms │ +1.07x faster │
│ QQuery 37 │   36.77 ms │                                      35.69 ms │     no change │
│ QQuery 38 │   42.58 ms │                                      40.14 ms │ +1.06x faster │
│ QQuery 39 │  152.46 ms │                                     133.46 ms │ +1.14x faster │
│ QQuery 40 │   16.37 ms │                                      14.15 ms │ +1.16x faster │
│ QQuery 41 │   15.40 ms │                                      13.77 ms │ +1.12x faster │
│ QQuery 42 │   13.40 ms │                                      13.30 ms │     no change │
└───────────┴────────────┴───────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                            ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                            │ 26580.17ms │
│ Total Time (claude_single-distinct-to-groupby-allow-count)   │ 26405.69ms │
│ Average Time (HEAD)                                          │   618.14ms │
│ Average Time (claude_single-distinct-to-groupby-allow-count) │   614.09ms │
│ Queries Faster                                               │          6 │
│ Queries Slower                                               │          0 │
│ Queries with No Change                                       │         37 │
│ Queries with Failure                                         │          0 │
└──────────────────────────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                   HEAD ┃ claude_single-distinct-to-groupby-allow-count ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │           1.22 / 4.03 ±5.52 / 15.07 ms │                  1.21 / 4.07 ±5.62 / 15.31 ms │     no change │
│ QQuery 1  │         11.84 / 12.04 ±0.13 / 12.20 ms │                11.85 / 12.04 ±0.14 / 12.18 ms │     no change │
│ QQuery 2  │         37.10 / 37.32 ±0.21 / 37.70 ms │                36.43 / 36.77 ±0.27 / 37.14 ms │     no change │
│ QQuery 3  │         31.32 / 31.96 ±0.75 / 33.37 ms │                31.30 / 32.04 ±0.72 / 33.31 ms │     no change │
│ QQuery 4  │      227.95 / 230.00 ±2.10 / 233.78 ms │             223.17 / 225.87 ±1.68 / 227.65 ms │     no change │
│ QQuery 5  │      278.44 / 279.03 ±0.47 / 279.59 ms │             273.19 / 277.34 ±4.14 / 282.77 ms │     no change │
│ QQuery 6  │            1.28 / 1.42 ±0.22 / 1.86 ms │                   1.27 / 1.42 ±0.23 / 1.88 ms │     no change │
│ QQuery 7  │         13.60 / 13.73 ±0.09 / 13.86 ms │                13.41 / 14.00 ±0.38 / 14.52 ms │     no change │
│ QQuery 8  │      334.97 / 339.76 ±4.12 / 345.24 ms │           336.15 / 422.42 ±164.81 / 751.89 ms │  1.24x slower │
│ QQuery 9  │      462.84 / 473.37 ±5.98 / 479.47 ms │             455.82 / 466.55 ±9.97 / 482.30 ms │     no change │
│ QQuery 10 │         70.72 / 72.05 ±1.05 / 73.56 ms │                70.27 / 74.76 ±8.10 / 90.96 ms │     no change │
│ QQuery 11 │         82.51 / 85.86 ±6.16 / 98.18 ms │                81.12 / 82.61 ±1.13 / 83.95 ms │     no change │
│ QQuery 12 │      271.21 / 274.38 ±3.28 / 279.68 ms │             270.32 / 274.99 ±3.16 / 280.26 ms │     no change │
│ QQuery 13 │     990.69 / 997.89 ±5.99 / 1005.99 ms │           971.51 / 994.19 ±14.38 / 1009.62 ms │     no change │
│ QQuery 14 │      285.44 / 291.48 ±3.21 / 294.20 ms │             284.86 / 288.27 ±3.12 / 292.14 ms │     no change │
│ QQuery 15 │      271.87 / 274.95 ±3.34 / 281.34 ms │             263.20 / 272.39 ±8.06 / 287.09 ms │     no change │
│ QQuery 16 │  1224.39 / 1253.92 ±19.39 / 1281.87 ms │         1207.65 / 1240.15 ±23.07 / 1267.82 ms │     no change │
│ QQuery 17 │    938.17 / 964.06 ±24.48 / 1003.91 ms │             928.40 / 937.91 ±7.10 / 949.12 ms │     no change │
│ QQuery 18 │  2525.15 / 2581.99 ±30.34 / 2607.28 ms │        2495.36 / 2605.08 ±169.51 / 2941.99 ms │     no change │
│ QQuery 19 │         28.34 / 28.90 ±0.58 / 29.94 ms │                27.90 / 28.46 ±0.48 / 29.07 ms │     no change │
│ QQuery 20 │      525.20 / 536.89 ±7.11 / 544.22 ms │             523.29 / 528.63 ±7.76 / 543.93 ms │     no change │
│ QQuery 21 │      517.57 / 520.31 ±1.95 / 522.69 ms │             515.53 / 523.04 ±6.77 / 532.36 ms │     no change │
│ QQuery 22 │   988.34 / 1001.63 ±12.13 / 1017.49 ms │           983.42 / 994.83 ±12.13 / 1016.41 ms │     no change │
│ QQuery 23 │  3019.56 / 3035.98 ±16.50 / 3065.28 ms │         3020.27 / 3049.11 ±17.13 / 3071.25 ms │     no change │
│ QQuery 24 │         41.26 / 42.45 ±1.80 / 46.03 ms │                41.15 / 42.19 ±1.24 / 44.58 ms │     no change │
│ QQuery 25 │      109.96 / 112.39 ±3.13 / 118.54 ms │             110.13 / 111.02 ±0.57 / 111.82 ms │     no change │
│ QQuery 26 │         42.24 / 42.97 ±0.69 / 43.93 ms │                41.22 / 41.58 ±0.27 / 42.02 ms │     no change │
│ QQuery 27 │     512.82 / 526.02 ±11.16 / 539.89 ms │             522.42 / 528.21 ±4.27 / 533.77 ms │     no change │
│ QQuery 28 │  2944.09 / 2971.20 ±20.57 / 3005.46 ms │         2916.45 / 2947.55 ±27.99 / 2989.42 ms │     no change │
│ QQuery 29 │         41.01 / 43.94 ±5.16 / 54.25 ms │                41.20 / 44.30 ±4.62 / 53.34 ms │     no change │
│ QQuery 30 │      303.22 / 310.81 ±9.03 / 326.79 ms │             300.73 / 309.74 ±7.10 / 321.89 ms │     no change │
│ QQuery 31 │      289.32 / 298.20 ±6.15 / 308.15 ms │            282.18 / 291.67 ±11.42 / 308.58 ms │     no change │
│ QQuery 32 │  3284.67 / 3322.65 ±27.89 / 3350.85 ms │         3283.17 / 3347.84 ±43.18 / 3408.73 ms │     no change │
│ QQuery 33 │  2556.04 / 2627.15 ±45.76 / 2678.81 ms │         2592.71 / 2646.43 ±49.60 / 2715.34 ms │     no change │
│ QQuery 34 │ 2630.88 / 2838.41 ±161.17 / 3093.67 ms │         2647.50 / 2735.70 ±59.70 / 2816.97 ms │     no change │
│ QQuery 35 │     338.93 / 354.55 ±13.22 / 373.40 ms │             284.82 / 296.84 ±9.47 / 307.76 ms │ +1.19x faster │
│ QQuery 36 │         69.03 / 76.47 ±6.95 / 89.31 ms │                64.59 / 71.83 ±5.20 / 77.62 ms │ +1.06x faster │
│ QQuery 37 │         36.77 / 37.43 ±0.54 / 38.26 ms │                35.69 / 37.36 ±1.53 / 39.68 ms │     no change │
│ QQuery 38 │        42.58 / 54.55 ±21.43 / 97.34 ms │                40.14 / 40.46 ±0.28 / 40.99 ms │ +1.35x faster │
│ QQuery 39 │      152.46 / 156.61 ±3.70 / 162.37 ms │            133.46 / 154.18 ±24.35 / 202.03 ms │     no change │
│ QQuery 40 │         16.37 / 17.84 ±1.91 / 21.58 ms │                14.15 / 14.62 ±0.32 / 15.05 ms │ +1.22x faster │
│ QQuery 41 │         15.40 / 15.89 ±0.50 / 16.58 ms │                13.77 / 14.16 ±0.31 / 14.61 ms │ +1.12x faster │
│ QQuery 42 │         13.40 / 13.91 ±0.30 / 14.23 ms │                13.30 / 13.43 ±0.10 / 13.54 ms │     no change │
└───────────┴────────────────────────────────────────┴───────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                            ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                            │ 27206.37ms │
│ Total Time (claude_single-distinct-to-groupby-allow-count)   │ 27076.07ms │
│ Average Time (HEAD)                                          │   632.71ms │
│ Average Time (claude_single-distinct-to-groupby-allow-count) │   629.68ms │
│ Queries Faster                                               │          5 │
│ Queries Slower                                               │          1 │
│ Queries with No Change                                       │         37 │
│ Queries with Failure                                         │          0 │
└──────────────────────────────────────────────────────────────┴────────────┘

Memory Pool Peaks

Peak MemoryPool reservation per query — what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with DATAFUSION_RUNTIME_MEMORY_LIMIT set.

Base: da89c7c (merge-base) | Changed: claude/single-distinct-to-groupby-allow-count

clickbench_partitioned

Query	Base	Changed	Change
Query 0	0 B	0 B	0.0%
Query 1	104 B	104 B	+0.0%
Query 2	936 B	936 B	+0.0%
Query 3	312 B	312 B	+0.0%
Query 4	761.5 MiB	758.3 MiB	-0.4%
Query 5	1.2 GiB	1.1 GiB	-4.1%
Query 6	0 B	0 B	0.0%
Query 7	50.2 MiB	70.1 MiB	+39.7%
Query 8	869.5 MiB	883.8 MiB	+1.6%
Query 9	593.9 MiB	551.7 MiB	-7.1%
Query 10	107.5 MiB	114.9 MiB	+6.8%
Query 11	111.9 MiB	117.8 MiB	+5.3%
Query 12	1.3 GiB	1.3 GiB	+1.2%
Query 13	1018.6 MiB	1014.4 MiB	-0.4%
Query 14	1.3 GiB	1.3 GiB	-2.1%
Query 15	1.2 GiB	1.2 GiB	-0.2%
Query 16	1.7 GiB	2.0 GiB	+14.8%
Query 17	1.8 GiB	1.9 GiB	+6.3%
Query 18	1.8 GiB	1.7 GiB	-4.6%
Query 19	0 B	0 B	0.0%
Query 20	104 B	104 B	+0.0%
Query 21	3.3 MiB	3.3 MiB	-0.6%
Query 22	4.2 MiB	3.1 MiB	-24.4%
Query 23	27.3 MiB	28.4 MiB	+4.1%
Query 24	59.3 MiB	60.8 MiB	+2.5%
Query 25	178.2 MiB	177.9 MiB	-0.2%
Query 26	61.5 MiB	60.2 MiB	-2.2%
Query 27	2.4 MiB	2.2 MiB	-9.1%
Query 28	1.5 GiB	1.5 GiB	+1.5%
Query 29	624 B	624 B	+0.0%
Query 30	724.4 MiB	733.8 MiB	+1.3%
Query 31	1.5 GiB	1.5 GiB	-3.1%
Query 32	928.7 MiB	927.5 MiB	-0.1%
Query 33	2.1 GiB	2.1 GiB	-2.0%
Query 34	2.0 GiB	2.2 GiB	+6.4%
Query 35	596.8 MiB	593.9 MiB	-0.5%
Query 36	124.8 MiB	117.7 MiB	-5.7%
Query 37	6.3 MiB	6.9 MiB	+10.0%
Query 38	5.6 MiB	5.7 MiB	+0.7%
Query 39	298.3 MiB	297.8 MiB	-0.2%
Query 40	1.8 MiB	1.7 MiB	-5.9%
Query 41	3.1 MiB	3.1 MiB	+0.0%
Query 42	1.9 MiB	1.6 MiB	-11.1%

Pool accounting vs. process RSS

Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.

Benchmark	Side	Max pool peak	Peak RSS	Gap	RSS / pool
clickbench_partitioned	base (`da89c7c (merge-base)`)	2.1 GiB	8.5 GiB	6.4 GiB	4.0×
clickbench_partitioned	changed (`claude/single-distinct-to-groupby-allow-count`)	2.2 GiB	8.7 GiB	6.5 GiB	4.0×

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	140.0s
Peak memory	8.5 GiB
Avg memory	4.9 GiB
CPU user	1369.2s
CPU sys	136.3s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	140.0s
Peak memory	8.7 GiB
Avg memory	5.1 GiB
CPU user	1369.4s
CPU sys	135.5s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-09-02T19:20:19Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing claude/single-distinct-to-groupby-allow-count (d510dd4) to da89c7c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
env:
  DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ claude_single-distinct-to-groupby-allow-count ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │    1.27 ms │                                       1.22 ms │     no change │
│ QQuery 1  │   12.22 ms │                                      11.62 ms │     no change │
│ QQuery 2  │   38.04 ms │                                      36.38 ms │     no change │
│ QQuery 3  │   32.62 ms │                                      30.96 ms │ +1.05x faster │
│ QQuery 4  │  227.33 ms │                                     220.88 ms │     no change │
│ QQuery 5  │  274.86 ms │                                     270.90 ms │     no change │
│ QQuery 6  │    1.28 ms │                                       1.25 ms │     no change │
│ QQuery 7  │   13.55 ms │                                      13.08 ms │     no change │
│ QQuery 8  │  332.26 ms │                                     327.53 ms │     no change │
│ QQuery 9  │  446.62 ms │                                     457.33 ms │     no change │
│ QQuery 10 │   69.37 ms │                                      69.51 ms │     no change │
│ QQuery 11 │   82.09 ms │                                      80.21 ms │     no change │
│ QQuery 12 │  269.35 ms │                                     266.19 ms │     no change │
│ QQuery 13 │  966.91 ms │                                     966.01 ms │     no change │
│ QQuery 14 │  284.98 ms │                                     319.46 ms │  1.12x slower │
│ QQuery 15 │  276.84 ms │                                     312.31 ms │  1.13x slower │
│ QQuery 16 │ 1238.76 ms │                                    1256.69 ms │     no change │
│ QQuery 17 │  898.65 ms │                                     940.45 ms │     no change │
│ QQuery 18 │ 2455.27 ms │                                    2464.09 ms │     no change │
│ QQuery 19 │   29.77 ms │                                      27.66 ms │ +1.08x faster │
│ QQuery 20 │  517.52 ms │                                     510.92 ms │     no change │
│ QQuery 21 │  518.26 ms │                                     509.53 ms │     no change │
│ QQuery 22 │  977.48 ms │                                     977.16 ms │     no change │
│ QQuery 23 │ 3000.63 ms │                                    2958.54 ms │     no change │
│ QQuery 24 │   40.79 ms │                                      41.28 ms │     no change │
│ QQuery 25 │  109.83 ms │                                     108.64 ms │     no change │
│ QQuery 26 │   41.31 ms │                                      41.13 ms │     no change │
│ QQuery 27 │  508.08 ms │                                     508.58 ms │     no change │
│ QQuery 28 │ 2907.16 ms │                                    2980.89 ms │     no change │
│ QQuery 29 │   41.41 ms │                                      41.08 ms │     no change │
│ QQuery 30 │  302.75 ms │                                     297.41 ms │     no change │
│ QQuery 31 │  273.33 ms │                                     280.18 ms │     no change │
│ QQuery 32 │ 3282.50 ms │                                    3248.08 ms │     no change │
│ QQuery 33 │ 2590.55 ms │                                    2520.60 ms │     no change │
│ QQuery 34 │ 2692.59 ms │                                    2580.73 ms │     no change │
│ QQuery 35 │  277.90 ms │                                     300.20 ms │  1.08x slower │
│ QQuery 36 │   67.22 ms │                                      70.81 ms │  1.05x slower │
│ QQuery 37 │   35.86 ms │                                      38.17 ms │  1.06x slower │
│ QQuery 38 │   40.22 ms │                                      42.64 ms │  1.06x slower │
│ QQuery 39 │  130.00 ms │                                     159.50 ms │  1.23x slower │
│ QQuery 40 │   13.96 ms │                                      16.06 ms │  1.15x slower │
│ QQuery 41 │   13.59 ms │                                      15.23 ms │  1.12x slower │
│ QQuery 42 │   13.39 ms │                                      14.62 ms │  1.09x slower │
└───────────┴────────────┴───────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                            ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                            │ 26348.35ms │
│ Total Time (claude_single-distinct-to-groupby-allow-count)   │ 26335.75ms │
│ Average Time (HEAD)                                          │   612.75ms │
│ Average Time (claude_single-distinct-to-groupby-allow-count) │   612.46ms │
│ Queries Faster                                               │          2 │
│ Queries Slower                                               │         10 │
│ Queries with No Change                                       │         31 │
│ Queries with Failure                                         │          0 │
└──────────────────────────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                   HEAD ┃ claude_single-distinct-to-groupby-allow-count ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │           1.27 / 4.22 ±5.73 / 15.68 ms │                  1.22 / 3.95 ±5.39 / 14.73 ms │ +1.07x faster │
│ QQuery 1  │         12.22 / 12.58 ±0.25 / 12.87 ms │                11.62 / 12.05 ±0.27 / 12.45 ms │     no change │
│ QQuery 2  │         38.04 / 38.46 ±0.31 / 38.92 ms │                36.38 / 36.55 ±0.24 / 37.01 ms │     no change │
│ QQuery 3  │         32.62 / 33.85 ±1.02 / 35.31 ms │                30.96 / 31.38 ±0.61 / 32.58 ms │ +1.08x faster │
│ QQuery 4  │      227.33 / 229.65 ±1.63 / 232.07 ms │             220.88 / 225.64 ±3.37 / 231.19 ms │     no change │
│ QQuery 5  │      274.86 / 277.89 ±1.70 / 279.65 ms │             270.90 / 274.70 ±3.88 / 281.59 ms │     no change │
│ QQuery 6  │            1.28 / 1.46 ±0.21 / 1.85 ms │                   1.25 / 1.39 ±0.22 / 1.82 ms │     no change │
│ QQuery 7  │         13.55 / 13.67 ±0.10 / 13.85 ms │                13.08 / 13.21 ±0.09 / 13.36 ms │     no change │
│ QQuery 8  │      332.26 / 340.41 ±6.04 / 347.47 ms │             327.53 / 335.83 ±5.09 / 341.12 ms │     no change │
│ QQuery 9  │      446.62 / 459.03 ±7.80 / 466.14 ms │            457.33 / 467.33 ±11.39 / 488.93 ms │     no change │
│ QQuery 10 │         69.37 / 70.42 ±0.88 / 72.00 ms │                69.51 / 70.92 ±1.98 / 74.83 ms │     no change │
│ QQuery 11 │         82.09 / 82.50 ±0.30 / 82.94 ms │                80.21 / 81.06 ±0.87 / 82.12 ms │     no change │
│ QQuery 12 │      269.35 / 271.44 ±2.11 / 275.24 ms │             266.19 / 272.74 ±3.97 / 278.22 ms │     no change │
│ QQuery 13 │      966.91 / 975.94 ±6.10 / 982.32 ms │           966.01 / 986.04 ±21.18 / 1026.61 ms │     no change │
│ QQuery 14 │     284.98 / 314.15 ±16.60 / 330.66 ms │             319.46 / 328.91 ±8.76 / 344.68 ms │     no change │
│ QQuery 15 │     276.84 / 298.01 ±18.55 / 320.53 ms │             312.31 / 317.90 ±5.32 / 327.14 ms │  1.07x slower │
│ QQuery 16 │  1238.76 / 1273.35 ±33.77 / 1337.52 ms │         1256.69 / 1342.81 ±52.12 / 1416.22 ms │  1.05x slower │
│ QQuery 17 │     898.65 / 921.54 ±16.68 / 941.53 ms │           940.45 / 998.66 ±39.90 / 1060.94 ms │  1.08x slower │
│ QQuery 18 │ 2455.27 / 2599.91 ±116.87 / 2750.73 ms │         2464.09 / 2541.95 ±69.68 / 2672.30 ms │     no change │
│ QQuery 19 │         29.77 / 30.61 ±0.77 / 31.58 ms │                27.66 / 28.22 ±0.55 / 29.20 ms │ +1.08x faster │
│ QQuery 20 │      517.52 / 523.78 ±4.51 / 531.10 ms │             510.92 / 524.84 ±8.84 / 534.47 ms │     no change │
│ QQuery 21 │      518.26 / 530.80 ±8.21 / 540.00 ms │            509.53 / 525.11 ±11.95 / 545.07 ms │     no change │
│ QQuery 22 │     977.48 / 987.68 ±8.84 / 1002.03 ms │           977.16 / 987.63 ±10.29 / 1006.44 ms │     no change │
│ QQuery 23 │  3000.63 / 3046.06 ±37.37 / 3113.11 ms │         2958.54 / 3039.22 ±88.25 / 3204.36 ms │     no change │
│ QQuery 24 │         40.79 / 44.59 ±5.79 / 56.13 ms │               41.28 / 48.60 ±13.44 / 75.48 ms │  1.09x slower │
│ QQuery 25 │      109.83 / 114.20 ±6.64 / 127.43 ms │             108.64 / 109.31 ±0.56 / 110.33 ms │     no change │
│ QQuery 26 │         41.31 / 46.52 ±7.65 / 61.73 ms │                41.13 / 47.43 ±9.70 / 66.74 ms │     no change │
│ QQuery 27 │      508.08 / 516.37 ±5.30 / 523.45 ms │             508.58 / 513.40 ±3.44 / 518.91 ms │     no change │
│ QQuery 28 │  2907.16 / 2942.96 ±35.07 / 3008.63 ms │         2980.89 / 3037.02 ±49.41 / 3099.25 ms │     no change │
│ QQuery 29 │        41.41 / 52.36 ±20.82 / 93.99 ms │              41.08 / 57.44 ±31.57 / 120.56 ms │  1.10x slower │
│ QQuery 30 │      302.75 / 307.19 ±4.85 / 315.84 ms │             297.41 / 310.09 ±8.62 / 322.16 ms │     no change │
│ QQuery 31 │      273.33 / 283.38 ±6.78 / 293.89 ms │             280.18 / 290.74 ±9.38 / 303.48 ms │     no change │
│ QQuery 32 │ 3282.50 / 3445.59 ±145.25 / 3620.75 ms │         3248.08 / 3336.39 ±61.54 / 3412.53 ms │     no change │
│ QQuery 33 │  2590.55 / 2688.67 ±62.72 / 2769.88 ms │         2520.60 / 2616.21 ±99.98 / 2766.88 ms │     no change │
│ QQuery 34 │  2692.59 / 2721.40 ±37.26 / 2794.25 ms │         2580.73 / 2649.07 ±72.19 / 2784.12 ms │     no change │
│ QQuery 35 │      277.90 / 282.94 ±5.46 / 292.69 ms │            300.20 / 333.61 ±20.64 / 354.46 ms │  1.18x slower │
│ QQuery 36 │       67.22 / 84.13 ±27.76 / 139.47 ms │                70.81 / 73.49 ±2.53 / 77.55 ms │ +1.14x faster │
│ QQuery 37 │         35.86 / 37.09 ±1.37 / 39.54 ms │                38.17 / 38.51 ±0.27 / 38.87 ms │     no change │
│ QQuery 38 │         40.22 / 41.81 ±1.50 / 44.11 ms │              42.64 / 57.36 ±24.04 / 105.23 ms │  1.37x slower │
│ QQuery 39 │     130.00 / 140.95 ±10.12 / 152.99 ms │             159.50 / 165.39 ±5.39 / 173.60 ms │  1.17x slower │
│ QQuery 40 │         13.96 / 14.43 ±0.31 / 14.82 ms │                16.06 / 17.49 ±1.13 / 19.06 ms │  1.21x slower │
│ QQuery 41 │         13.59 / 15.05 ±1.86 / 18.67 ms │                15.23 / 15.47 ±0.17 / 15.75 ms │     no change │
│ QQuery 42 │         13.39 / 13.56 ±0.24 / 14.03 ms │                14.62 / 14.81 ±0.14 / 15.04 ms │  1.09x slower │
└───────────┴────────────────────────────────────────┴───────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                            ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                            │ 27130.58ms │
│ Total Time (claude_single-distinct-to-groupby-allow-count)   │ 27179.87ms │
│ Average Time (HEAD)                                          │   630.94ms │
│ Average Time (claude_single-distinct-to-groupby-allow-count) │   632.09ms │
│ Queries Faster                                               │          4 │
│ Queries Slower                                               │         10 │
│ Queries with No Change                                       │         29 │
│ Queries with Failure                                         │          0 │
└──────────────────────────────────────────────────────────────┴────────────┘

Memory Pool Peaks

Peak MemoryPool reservation per query — what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with DATAFUSION_RUNTIME_MEMORY_LIMIT set.

Base: da89c7c (merge-base) | Changed: claude/single-distinct-to-groupby-allow-count

clickbench_partitioned

Query	Base	Changed	Change
Query 0	0 B	0 B	0.0%
Query 1	104 B	104 B	+0.0%
Query 2	936 B	936 B	+0.0%
Query 3	312 B	312 B	+0.0%
Query 4	757.5 MiB	772.8 MiB	+2.0%
Query 5	1.1 GiB	1.2 GiB	+6.7%
Query 6	0 B	0 B	0.0%
Query 7	60.2 MiB	50.3 MiB	-16.4%
Query 8	846.7 MiB	899.0 MiB	+6.2%
Query 9	677.7 MiB	593.5 MiB	-12.4%
Query 10	104.3 MiB	112.8 MiB	+8.1%
Query 11	110.0 MiB	114.7 MiB	+4.2%
Query 12	1.3 GiB	1.3 GiB	-3.8%
Query 13	1.0 GiB	1.0 GiB	+0.5%
Query 14	1.2 GiB	1.3 GiB	+1.8%
Query 15	1.2 GiB	1.2 GiB	+0.9%
Query 16	1.8 GiB	1.8 GiB	-1.0%
Query 17	1.7 GiB	2.1 GiB	+26.7%
Query 18	1.9 GiB	1.8 GiB	-8.0%
Query 19	0 B	0 B	0.0%
Query 20	104 B	104 B	+0.0%
Query 21	3.3 MiB	3.3 MiB	-1.2%
Query 22	3.0 MiB	2.6 MiB	-13.8%
Query 23	29.3 MiB	25.2 MiB	-13.9%
Query 24	60.8 MiB	61.6 MiB	+1.2%
Query 25	172.1 MiB	177.3 MiB	+3.0%
Query 26	63.8 MiB	60.8 MiB	-4.7%
Query 27	2.4 MiB	2.4 MiB	+0.0%
Query 28	1.5 GiB	1.5 GiB	-4.8%
Query 29	624 B	624 B	+0.0%
Query 30	731.0 MiB	662.2 MiB	-9.4%
Query 31	1.5 GiB	1.5 GiB	+0.0%
Query 32	928.7 MiB	926.4 MiB	-0.3%
Query 33	2.1 GiB	2.1 GiB	+1.9%
Query 34	2.1 GiB	2.0 GiB	-4.4%
Query 35	596.1 MiB	595.9 MiB	-0.0%
Query 36	122.5 MiB	113.9 MiB	-7.0%
Query 37	6.9 MiB	6.9 MiB	+0.0%
Query 38	5.2 MiB	5.6 MiB	+8.5%
Query 39	297.8 MiB	297.8 MiB	+0.0%
Query 40	1.7 MiB	2.0 MiB	+17.2%
Query 41	3.1 MiB	3.1 MiB	-0.0%
Query 42	1.7 MiB	2.1 MiB	+25.1%

Pool accounting vs. process RSS

Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.

Benchmark	Side	Max pool peak	Peak RSS	Gap	RSS / pool
clickbench_partitioned	base (`da89c7c (merge-base)`)	2.1 GiB	9.3 GiB	7.2 GiB	4.4×
clickbench_partitioned	changed (`claude/single-distinct-to-groupby-allow-count`)	2.1 GiB	8.6 GiB	6.5 GiB	4.1×

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	140.0s
Peak memory	9.3 GiB
Avg memory	5.6 GiB
CPU user	1383.9s
CPU sys	128.0s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	140.0s
Peak memory	8.6 GiB
Avg memory	5.1 GiB
CPU user	1379.7s
CPU sys	133.6s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-09-02T19:20:59Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing claude/single-distinct-to-groupby-allow-count (d510dd4) to da89c7c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned
env:
  DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ claude_single-distinct-to-groupby-allow-count ┃    Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0  │    1.25 ms │                                       1.26 ms │ no change │
│ QQuery 1  │   12.04 ms │                                      12.31 ms │ no change │
│ QQuery 2  │   37.24 ms │                                      36.99 ms │ no change │
│ QQuery 3  │   32.08 ms │                                      31.95 ms │ no change │
│ QQuery 4  │  238.55 ms │                                     234.46 ms │ no change │
│ QQuery 5  │  283.29 ms │                                     283.96 ms │ no change │
│ QQuery 6  │    1.29 ms │                                       1.33 ms │ no change │
│ QQuery 7  │   13.56 ms │                                      13.59 ms │ no change │
│ QQuery 8  │  359.70 ms │                                     359.76 ms │ no change │
│ QQuery 9  │  484.05 ms │                                     496.72 ms │ no change │
│ QQuery 10 │   72.00 ms │                                      73.83 ms │ no change │
│ QQuery 11 │   83.33 ms │                                      84.50 ms │ no change │
│ QQuery 12 │  280.11 ms │                                     285.04 ms │ no change │
│ QQuery 13 │ 1018.57 ms │                                    1021.09 ms │ no change │
│ QQuery 14 │  296.70 ms │                                     296.85 ms │ no change │
│ QQuery 15 │  286.15 ms │                                     291.21 ms │ no change │
│ QQuery 16 │ 1238.77 ms │                                    1238.03 ms │ no change │
│ QQuery 17 │  958.92 ms │                                     999.14 ms │ no change │
│ QQuery 18 │ 2558.25 ms │                                    2624.19 ms │ no change │
│ QQuery 19 │   29.59 ms │                                      29.33 ms │ no change │
│ QQuery 20 │  517.42 ms │                                     529.51 ms │ no change │
│ QQuery 21 │  516.30 ms │                                     527.07 ms │ no change │
│ QQuery 22 │ 1002.16 ms │                                    1014.28 ms │ no change │
│ QQuery 23 │ 3089.00 ms │                                    3127.31 ms │ no change │
│ QQuery 24 │   42.94 ms │                                      41.81 ms │ no change │
│ QQuery 25 │  113.80 ms │                                     113.77 ms │ no change │
│ QQuery 26 │   43.30 ms │                                      42.36 ms │ no change │
│ QQuery 27 │  518.63 ms │                                     524.14 ms │ no change │
│ QQuery 28 │ 2978.05 ms │                                    2986.05 ms │ no change │
│ QQuery 29 │   42.11 ms │                                      42.46 ms │ no change │
│ QQuery 30 │  327.55 ms │                                     321.58 ms │ no change │
│ QQuery 31 │  291.20 ms │                                     293.10 ms │ no change │
│ QQuery 32 │ 3488.76 ms │                                    3437.55 ms │ no change │
│ QQuery 33 │ 2680.67 ms │                                    2632.40 ms │ no change │
│ QQuery 34 │ 2798.53 ms │                                    2759.08 ms │ no change │
│ QQuery 35 │  310.34 ms │                                     305.11 ms │ no change │
│ QQuery 36 │   67.79 ms │                                      68.55 ms │ no change │
│ QQuery 37 │   37.27 ms │                                      36.66 ms │ no change │
│ QQuery 38 │   41.42 ms │                                      41.20 ms │ no change │
│ QQuery 39 │  136.99 ms │                                     137.55 ms │ no change │
│ QQuery 40 │   15.21 ms │                                      15.10 ms │ no change │
│ QQuery 41 │   14.91 ms │                                      14.67 ms │ no change │
│ QQuery 42 │   14.52 ms │                                      14.10 ms │ no change │
└───────────┴────────────┴───────────────────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                            ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                            │ 27374.33ms │
│ Total Time (claude_single-distinct-to-groupby-allow-count)   │ 27440.98ms │
│ Average Time (HEAD)                                          │   636.61ms │
│ Average Time (claude_single-distinct-to-groupby-allow-count) │   638.16ms │
│ Queries Faster                                               │          0 │
│ Queries Slower                                               │          0 │
│ Queries with No Change                                       │         43 │
│ Queries with Failure                                         │          0 │
└──────────────────────────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                   HEAD ┃ claude_single-distinct-to-groupby-allow-count ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │           1.25 / 4.13 ±5.61 / 15.36 ms │                  1.26 / 4.21 ±5.70 / 15.60 ms │     no change │
│ QQuery 1  │         12.04 / 12.53 ±0.27 / 12.81 ms │                12.31 / 12.58 ±0.22 / 12.93 ms │     no change │
│ QQuery 2  │         37.24 / 37.77 ±0.36 / 38.15 ms │                36.99 / 37.28 ±0.33 / 37.91 ms │     no change │
│ QQuery 3  │         32.08 / 33.20 ±0.78 / 34.13 ms │                31.95 / 32.11 ±0.13 / 32.28 ms │     no change │
│ QQuery 4  │      238.55 / 241.74 ±2.31 / 245.36 ms │             234.46 / 241.16 ±4.40 / 247.32 ms │     no change │
│ QQuery 5  │      283.29 / 287.49 ±3.02 / 291.53 ms │             283.96 / 290.74 ±4.13 / 296.74 ms │     no change │
│ QQuery 6  │            1.29 / 1.43 ±0.22 / 1.87 ms │                   1.33 / 1.49 ±0.24 / 1.96 ms │     no change │
│ QQuery 7  │         13.56 / 14.48 ±1.15 / 16.75 ms │                13.59 / 15.23 ±2.48 / 20.17 ms │  1.05x slower │
│ QQuery 8  │      359.70 / 364.02 ±2.84 / 366.95 ms │             359.76 / 364.52 ±2.98 / 369.14 ms │     no change │
│ QQuery 9  │      484.05 / 495.25 ±7.45 / 504.76 ms │             496.72 / 504.66 ±8.58 / 520.61 ms │     no change │
│ QQuery 10 │         72.00 / 72.65 ±0.49 / 73.38 ms │                73.83 / 75.06 ±0.97 / 76.47 ms │     no change │
│ QQuery 11 │         83.33 / 84.08 ±0.90 / 85.80 ms │                84.50 / 87.00 ±2.99 / 92.69 ms │     no change │
│ QQuery 12 │      280.11 / 286.46 ±4.35 / 292.04 ms │            285.04 / 300.48 ±12.85 / 320.77 ms │     no change │
│ QQuery 13 │  1018.57 / 1028.17 ±13.05 / 1054.04 ms │          1021.09 / 1028.06 ±3.86 / 1032.88 ms │     no change │
│ QQuery 14 │      296.70 / 302.18 ±4.52 / 307.42 ms │            296.85 / 309.11 ±18.89 / 346.57 ms │     no change │
│ QQuery 15 │     286.15 / 296.29 ±14.52 / 324.69 ms │             291.21 / 298.52 ±9.86 / 317.77 ms │     no change │
│ QQuery 16 │  1238.77 / 1291.13 ±34.65 / 1338.84 ms │         1238.03 / 1326.62 ±74.17 / 1458.76 ms │     no change │
│ QQuery 17 │    958.92 / 981.52 ±24.26 / 1026.10 ms │          999.14 / 1011.04 ±19.32 / 1049.47 ms │     no change │
│ QQuery 18 │  2558.25 / 2607.84 ±28.90 / 2637.26 ms │         2624.19 / 2684.03 ±36.03 / 2735.05 ms │     no change │
│ QQuery 19 │         29.59 / 34.34 ±7.12 / 48.36 ms │                29.33 / 29.71 ±0.44 / 30.52 ms │ +1.16x faster │
│ QQuery 20 │     517.42 / 534.52 ±10.93 / 550.84 ms │             529.51 / 534.55 ±4.05 / 540.57 ms │     no change │
│ QQuery 21 │      516.30 / 526.22 ±7.42 / 536.04 ms │             527.07 / 534.47 ±5.75 / 544.70 ms │     no change │
│ QQuery 22 │   1002.16 / 1004.25 ±2.64 / 1009.41 ms │          1014.28 / 1022.86 ±6.46 / 1032.03 ms │     no change │
│ QQuery 23 │  3089.00 / 3168.46 ±48.27 / 3237.70 ms │         3127.31 / 3152.36 ±15.67 / 3172.62 ms │     no change │
│ QQuery 24 │         42.94 / 43.79 ±0.86 / 45.43 ms │                41.81 / 46.31 ±6.05 / 57.74 ms │  1.06x slower │
│ QQuery 25 │      113.80 / 115.70 ±1.56 / 118.50 ms │             113.77 / 115.43 ±1.50 / 117.97 ms │     no change │
│ QQuery 26 │         43.30 / 47.76 ±4.32 / 54.98 ms │                42.36 / 44.15 ±1.81 / 47.37 ms │ +1.08x faster │
│ QQuery 27 │      518.63 / 527.15 ±5.77 / 534.51 ms │             524.14 / 531.48 ±7.46 / 543.78 ms │     no change │
│ QQuery 28 │  2978.05 / 3085.15 ±76.47 / 3198.31 ms │         2986.05 / 3002.24 ±14.17 / 3023.59 ms │     no change │
│ QQuery 29 │         42.11 / 43.05 ±0.84 / 44.28 ms │                42.46 / 42.93 ±0.40 / 43.59 ms │     no change │
│ QQuery 30 │      327.55 / 334.22 ±4.62 / 341.41 ms │            321.58 / 335.40 ±11.36 / 351.21 ms │     no change │
│ QQuery 31 │     291.20 / 305.89 ±10.06 / 321.25 ms │             293.10 / 300.07 ±4.92 / 307.82 ms │     no change │
│ QQuery 32 │  3488.76 / 3514.56 ±16.08 / 3536.17 ms │         3437.55 / 3484.28 ±33.12 / 3524.84 ms │     no change │
│ QQuery 33 │ 2680.67 / 2854.69 ±114.70 / 3012.87 ms │         2632.40 / 2720.12 ±76.93 / 2839.08 ms │     no change │
│ QQuery 34 │  2798.53 / 2899.67 ±58.26 / 2964.30 ms │         2759.08 / 2795.77 ±25.02 / 2828.75 ms │     no change │
│ QQuery 35 │      310.34 / 316.23 ±5.72 / 325.13 ms │             305.11 / 321.58 ±9.58 / 332.81 ms │     no change │
│ QQuery 36 │         67.79 / 73.23 ±6.64 / 85.89 ms │                68.55 / 73.16 ±3.75 / 78.74 ms │     no change │
│ QQuery 37 │         37.27 / 42.84 ±8.99 / 60.50 ms │               36.66 / 49.52 ±24.98 / 99.48 ms │  1.16x slower │
│ QQuery 38 │         41.42 / 43.62 ±2.67 / 47.65 ms │                41.20 / 41.59 ±0.23 / 41.84 ms │     no change │
│ QQuery 39 │     136.99 / 150.38 ±11.56 / 170.92 ms │            137.55 / 155.87 ±12.92 / 168.88 ms │     no change │
│ QQuery 40 │         15.21 / 15.81 ±0.61 / 16.88 ms │                15.10 / 15.52 ±0.22 / 15.68 ms │     no change │
│ QQuery 41 │         14.91 / 15.22 ±0.29 / 15.62 ms │                14.67 / 14.86 ±0.17 / 15.18 ms │     no change │
│ QQuery 42 │         14.52 / 14.57 ±0.05 / 14.63 ms │                14.10 / 14.42 ±0.19 / 14.65 ms │     no change │
└───────────┴────────────────────────────────────────┴───────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                            ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                            │ 28153.72ms │
│ Total Time (claude_single-distinct-to-groupby-allow-count)   │ 28002.54ms │
│ Average Time (HEAD)                                          │   654.74ms │
│ Average Time (claude_single-distinct-to-groupby-allow-count) │   651.22ms │
│ Queries Faster                                               │          2 │
│ Queries Slower                                               │          3 │
│ Queries with No Change                                       │         38 │
│ Queries with Failure                                         │          0 │
└──────────────────────────────────────────────────────────────┴────────────┘

Memory Pool Peaks

Peak MemoryPool reservation per query — what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with DATAFUSION_RUNTIME_MEMORY_LIMIT set.

Base: da89c7c (merge-base) | Changed: claude/single-distinct-to-groupby-allow-count

clickbench_partitioned

Query	Base	Changed	Change
Query 0	0 B	0 B	0.0%
Query 1	104 B	104 B	+0.0%
Query 2	936 B	936 B	+0.0%
Query 3	312 B	312 B	+0.0%
Query 4	777.8 MiB	770.4 MiB	-1.0%
Query 5	1.2 GiB	1.2 GiB	+2.3%
Query 6	0 B	0 B	0.0%
Query 7	40.1 MiB	60.1 MiB	+49.8%
Query 8	849.5 MiB	849.1 MiB	-0.0%
Query 9	593.7 MiB	593.9 MiB	+0.0%
Query 10	114.7 MiB	111.8 MiB	-2.5%
Query 11	107.8 MiB	112.1 MiB	+4.0%
Query 12	1.3 GiB	1.3 GiB	+5.2%
Query 13	1.0 GiB	1.0 GiB	-0.7%
Query 14	1.3 GiB	1.3 GiB	+1.5%
Query 15	1.2 GiB	1.2 GiB	-1.6%
Query 16	1.8 GiB	2.1 GiB	+16.7%
Query 17	1.9 GiB	2.0 GiB	+6.9%
Query 18	1.9 GiB	1.8 GiB	-5.8%
Query 19	0 B	0 B	0.0%
Query 20	104 B	104 B	+0.0%
Query 21	3.3 MiB	3.6 MiB	+9.3%
Query 22	4.8 MiB	3.0 MiB	-37.3%
Query 23	26.8 MiB	31.9 MiB	+19.0%
Query 24	59.3 MiB	59.1 MiB	-0.3%
Query 25	176.1 MiB	184.0 MiB	+4.5%
Query 26	64.8 MiB	60.9 MiB	-5.9%
Query 27	2.2 MiB	2.4 MiB	+10.0%
Query 28	1.5 GiB	1.5 GiB	-1.2%
Query 29	624 B	624 B	+0.0%
Query 30	716.0 MiB	697.8 MiB	-2.5%
Query 31	1.5 GiB	1.5 GiB	+1.6%
Query 32	970.2 MiB	928.1 MiB	-4.3%
Query 33	2.1 GiB	2.1 GiB	+1.2%
Query 34	2.1 GiB	2.1 GiB	+1.3%
Query 35	602.6 MiB	608.0 MiB	+0.9%
Query 36	123.2 MiB	118.1 MiB	-4.2%
Query 37	6.3 MiB	7.5 MiB	+20.0%
Query 38	5.2 MiB	4.7 MiB	-10.6%
Query 39	298.1 MiB	298.1 MiB	+0.0%
Query 40	2.0 MiB	2.0 MiB	-0.1%
Query 41	3.1 MiB	3.1 MiB	+0.0%
Query 42	1.7 MiB	2.1 MiB	+24.5%

Pool accounting vs. process RSS

Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.

Benchmark	Side	Max pool peak	Peak RSS	Gap	RSS / pool
clickbench_partitioned	base (`da89c7c (merge-base)`)	2.1 GiB	8.4 GiB	6.3 GiB	4.0×
clickbench_partitioned	changed (`claude/single-distinct-to-groupby-allow-count`)	2.1 GiB	8.4 GiB	6.3 GiB	4.0×

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	145.0s
Peak memory	8.4 GiB
Avg memory	5.1 GiB
CPU user	1430.0s
CPU sys	136.8s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	145.0s
Peak memory	8.4 GiB
Avg memory	5.1 GiB
CPU user	1420.1s
CPU sys	139.9s
Peak spill	0 B

File an issue against this benchmark runner

`cargo doc` runs with `-D warnings`, and a doc link from the public `SingleDistinctToGroupBy` to the private `rewrite_pays_for_count` is an error. Say the same thing in prose instead.

`AggregateUDFImpl::groups_accumulator_supported_for_types` returned `bool` with a `false` default, and `rewrite_pays_for_count` read `false` as "the rewrite pays". Every aggregate that did not override the method was therefore waved through the gate, which is the permissive answer rather than the safe one. Return `Option<bool>` instead, defaulting to `None`, and rewrite only on `Some(false)`. `None` says the aggregate does not answer the question, and silence is not evidence that the rewrite pays. Measured on unmodified upstream, over 4,000,000 rows in 2,000 groups, `SELECT g, count(*), sum(DISTINCT int_col) FROM t GROUP BY g` reached the gated path and regressed 3.15x: 71.0 MiB unrewritten against 223.4 MiB rewritten. `min(DISTINCT x)` reached it too, and regresses much further, because `min(DISTINCT x)` is `min(x)` and the unrewritten plan keeps one scalar per group. `Count` is the only implementor and its answers are unchanged, so no plan that the gate already allowed changes shape.

The two grouped `COUNT(DISTINCT <string>)` memory limit tests only reach the per group accumulators while `single_distinct_aggregation_to_group_by` declines to rewrite the query. They leant on `count(*)` for that, which the rule rejects only because `count` is missing from the `sum`/`min`/`max` allow list. #24859 proposes adding `count` to that list, which would rewrite the query, remove the accumulators, and leave both tests passing at any memory limit while still looking like they test something. Aggregate `avg(payload)` over a new `Int64` column instead. `avg` cannot be added to that list: the rule re-aggregates its own partial results over the deduplicated inner group by, and averaging per group averages of different sizes gives the wrong answer. That is why ClickBench Q9 keeps its distinct aggregate under #24859. Verified from the physical plan with #24859 cherry-picked on top of this branch: the `avg` query still plans as `aggr=[count(DISTINCT t.value), avg(t.payload)]`, while the `count(*)` query becomes `aggr=[count(alias1), sum(alias2)]` over an inner `GROUP BY group_key, value`, and drops from needing ~1.9 MB to ~0.9 MB. Re-swept both thresholds against the base commit. `Utf8` needs ~35.5 MB before and ~1.9 MB after; `Utf8View` needs ~123 MB before and ~2.5 MB after, so the 8 MB and 16 MB limits keep at least 4x margin on each side and are unchanged.

adriangb and others added 2 commits September 1, 2026 12:53

github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate labels Sep 1, 2026

This was referenced Sep 1, 2026

fix: stop pre-allocating and undercounting a hash table per COUNT(DISTINCT) group #24857

Open

fix: stop pre-allocating and undercounting a hash table per COUNT(DISTINCT) group pydantic/datafusion#70

Closed

github-actions Bot added logical-expr Logical plan and expressions functions Changes to functions implementation labels Sep 2, 2026

adriangb mentioned this pull request Sep 2, 2026

Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy pydantic/datafusion#72

Closed

adriangb added 2 commits September 2, 2026 14:23

Do not link a private item from public documentation

ecd4e98

`cargo doc` runs with `-D warnings`, and a doc link from the public `SingleDistinctToGroupBy` to the private `rewrite_pays_for_count` is an error. Say the same thing in prose instead.

Navigation Menu

Uh oh!

Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy - #24859

Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy#24859
adriangb wants to merge 5 commits into
mainfrom
claude/single-distinct-to-groupby-allow-count

adriangb commented Sep 1, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Sep 1, 2026 •

edited

Loading

Uh oh!

adriangb commented Sep 1, 2026

Uh oh!

adriangbot commented Sep 1, 2026

Uh oh!

adriangbot commented Sep 1, 2026

Uh oh!

adriangb commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

adriangb commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

Reproduce it

What this PR changes for that query

Why the rewrite helps here

The measurements

Where the rewrite stops paying

The gate is a proxy

What changes are included in this PR?

The gate

What is the testing strategy for this PR?

Benchmarks

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

adriangb commented Sep 1, 2026

Uh oh!

adriangbot commented Sep 1, 2026

Uh oh!

adriangbot commented Sep 1, 2026

Uh oh!

adriangb commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

adriangbot commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adriangb commented Sep 1, 2026 •

edited

Loading

codecov-commenter commented Sep 1, 2026 •

edited

Loading