DataFusion may run low on memory while processing a partial GROUP BY. When this happens, it sends out the groups collected so far and continues processing.
Today, EXPLAIN ANALYZE does not show when this happens. This makes it difficult to understand why a partial aggregate produced more rows than expected.
Something like an early_emit_count metric that shows how many times the aggregate had to flush because of memory pressure would be helpful
DataFusion may run low on memory while processing a partial GROUP BY. When this happens, it sends out the groups collected so far and continues processing.
Today, EXPLAIN ANALYZE does not show when this happens. This makes it difficult to understand why a partial aggregate produced more rows than expected.
Something like an
early_emit_countmetric that shows how many times the aggregate had to flush because of memory pressure would be helpful