bench: add a ClickBench extended query for a grouped COUNT(DISTINCT) over a string

[adriangb]

Which issue does this PR close?

No existing issue. This is benchmark coverage carved out of #24859 so that it can land first: a benchmark query added in the same PR that needs it cannot appear in an A/B run, because the bot compares against the merge base and the merge base does not have the query.

Rationale for this change

COUNT(DISTINCT) has a specialized GroupsAccumulator for the integer types and for no other type. Every other type falls back to GroupsAccumulatorAdapter, which holds one boxed Accumulator, and therefore one hash table, for each group. The cost of that fallback is per group, so the group cardinality is what decides how much it costs.

Nothing in either suite measures that:

• Extended Q2 is the only query that puts a COUNT(DISTINCT) on a non-integer column at all. It groups by BrowserCountry, so it exercises the adapter at a low group cardinality, which is where the adapter is cheapest.
• Standard Q8, Q10, Q11 and Q13 hold a distinct aggregate that stands alone, so SingleDistinctToGroupBy already rewrites them and they never reach the adapter.
• Standard Q9 carries an AVG, which that rule has never accepted.
• Standard Q22 is the one query that pairs a lone distinct aggregate with a non-distinct count, and it counts distinct UserID, an Int64, which has a specialized accumulator.

So a lone COUNT(DISTINCT <string>) grouped by a high cardinality key, next to a non-distinct COUNT(*), is uncovered. That is an ordinary analytics shape, it is the shape #24857 changed the memory profile of, and it is the shape #24859 proposes to rewrite. Neither of those could show its effect on any benchmark in this repository.

What changes are included in this PR?

The query:

SELECT "SearchPhrase", COUNT(*) AS c, COUNT(DISTINCT "MobilePhoneModel") AS models
FROM hits
WHERE "SearchPhrase" <> ''
GROUP BY "SearchPhrase"
ORDER BY c DESC
LIMIT 10;

The extended ClickBench queries live in three places, and a query added to only one of them is invisible to the others. This PR adds it to all three:

• benchmarks/queries/clickbench/extended/q14.sql, which feeds dfbench clickbench --queries-path. Queries are discovered from the directory, so the runner needs no change.
• benchmarks/sql_benchmarks/clickbench_extended/benchmarks/q14.benchmark, which feeds benchmark_runner clickbench_extended. Structurally identical to q13.benchmark apart from the query. Suite files are also discovered from the directory; benchmark_runner clickbench_extended --list now reports 15 queries.
• datafusion/sqllogictest/test_files/clickbench_extended.slt, which runs the extended queries against the committed ten row clickbench_hits_10.parquet fixture.

That last file had only ever mirrored q0 through q6. q7 through q13 were added to the queries directory without a matching sqllogictest entry, and q14 would have widened that gap, so this backfills q7 through q14 together. That is why the diff is larger than one query.

Having to add one query in three places is itself the problem, and the copies have already drifted: extended q6 carries a cast in its sql_benchmarks copy that the other two do not have. I filed #25031 for that; it is out of scope here.

There is one pre-existing staleness this PR does not fix. datafusion/core/benches/sql_planner.rs builds its ClickBench planning set from a hardcoded (0..=7) over the extended directory, so extended Q8 through Q13 were already outside it before this PR and Q14 joins them. That is a separate cleanup.

What is the testing strategy for this PR?

The sqllogictest entries are the test. clickbench_extended.slt runs every extended query against the committed ten row fixture and asserts its output, so q7 through q14 are now executed on every CI run rather than only by whoever runs the benchmark suite by hand. cargo test -p datafusion-sqllogictest --test sqllogictests -- clickbench passes, 2 of 2 files.

cargo test -p datafusion-benchmarks passes, 192 tests, and benchmark_runner clickbench_extended --list reports the suite at 15 queries, confirming the new .benchmark file parses and is discovered.

I do not have a hits.parquet to hand, so I have not run the query against the full dataset. The plan shape, which is the whole value of the query, was checked separately: on an equivalent local table with the same filter, grouping, COUNT(*) and string COUNT(DISTINCT), the aggregate keeps aggr=[[count(Int64(1)), count(DISTINCT ...)]], which is the GroupsAccumulatorAdapter path this query exists to measure, and it is not rewritten away.

What I have not established is the effect size on the real dataset, since that depends on how many distinct SearchPhrase values survive the filter. If a reviewer with the data runs it, that number is worth having on this PR.

ci/scripts/doc_prettier_check.sh passes on the README change.

Are there any user-facing changes?

No. This adds a benchmark query and documentation only.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.85%. Comparing base (e1ca94f) to head (d386092).
⚠️ Report is 10 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25026      +/-   ##
==========================================
+ Coverage   81.69%   81.85%   +0.15%     
==========================================
  Files        1127     1127              
  Lines      415471   418811    +3340     
  Branches   415471   418811    +3340     
==========================================
+ Hits       339424   342815    +3391     
+ Misses      56108    56027      -81     
- Partials    19939    19969      +30     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kumarUjjawal]

Thank you @adriangb

I was thinking since we are also using the clickbench_extended should we add this to there as well?

[jayzhan211]

Thanks @adriangb 👍🏻

[adriangb]

I was thinking since we are also using the clickbench_extended should we add this to there as well?

Good call. I noticed multiple queries were missing, so I added them all. Since this increased the scope I'll let you re-approve before merging.

[kumarUjjawal]

Since this increased the scope I'll let you re-approve before merging.

Thanks, Ping me when you add them.

[adriangb]

Sorry forgot to push! The change is now in 23a2181

[kumarUjjawal]

We can update q14 here https://github.com/apache/datafusion/tree/23a2181e0d00909f5223f0553dc5ca9fd3c100ef/benchmarks/sql_benchmarks/clickbench_extended/benchmarks