[CALCITE-7028] Implement SQL pipe syntax by gianm · Pull Request #5349 · apache/calcite

gianm · 2026-10-21T21:37:35Z

This patch adds FROM and |> WHERE from SQL pipe syntax. The syntax is implemented in the core parser, and is available only in certain conformances.

The approach is to keep the operators in the parse tree through validation, ultimately converting them to relational operators in SqlToRelConverter. This approach enables us to adhere to pipe syntax rules, especially the rules around table alias visibility and sort ordering retention. It also enables us to write pipe-syntax-aware error messages.

To simplify validation and conversion, each pipe operator is associated with a surrogate SELECT that acts as its stand-in. The pipe operator itself is used for pipe-specific validations, and retaining the correct scopes and collations.

gianm · 2026-10-21T22:26:06Z

Some notes about behavior and implementation.

In general this patch tries to adhere to GoogleSQL behavior. One thing that was hard to align with GoogleSQL, and remains in this patch, is treatment of aliases in subqueries. In GoogleSQL, SELECT e.ename FROM (FROM emp AS e) works fine, but in this implementation, Calcite will reject it. It wasn't obvious how to cleanly get aliases to escape a subquery. Maybe it can be done in a follow-up.

I want to call attention to one design decision: the "surrogate SELECT" concept. It's kind of a hybrid of two possible designs: rewriting the parse tree, and validating & converting pipe operators directly. Rewriting the parse tree has issues with achieving the proper behavior around retention of aliases and sort-order. Validating and converting pipe operators directly could work, but I was worried about the extra code it would end up needing. The surrogate SELECT approach gets to leverage a lot of pre-existing code.

mihaibudiu · 2026-10-22T16:49:23Z

In general we have trouble getting reviews for large PRs. I hope someone will find the time.

gianm · 2026-10-22T17:57:38Z

In general we have trouble getting reviews for large PRs. I hope someone will find the time.

It could be made smaller by restricting it to FROM and WHERE only, as the original Jira suggested, rather than also including SELECT, ORDER BY, and LIMIT. I chose to include those three because I felt this set provides better motivation for some of the design decisions, since it includes an operator that projects and an operator that sorts.

Anyway, please let me know if cutting back the scope to just FROM and WHERE would be better.

mihaibudiu · 2026-10-22T18:30:06Z

in general two smaller PRs is easier than a big one for reviewers.
Can you estimate how big it would be after removing these pieces?

gianm · 2026-10-22T20:54:59Z

in general two smaller PRs is easier than a big one for reviewers.

I just cut it back to only FROM and WHERE. It's about 2200 lines now, vs 3600 originally. I can do the other operators in a follow-up, which would be about 1400 lines.

mihaibudiu · 2026-10-22T21:18:02Z

I like pipe syntax, I will try to review this, but it may take a while.
Unless someone else volunteers...

mihaibudiu

This is generally fine, I think I only have style comments.
I think I understand the general structure of this PR, but it's complicated. The proof is in the tests.

mihaibudiu · 2026-10-25T17:59:05Z

+/**
+ * Reads optional pipe operators and applies them to the query {@code e}.
+ *
+ * <p>If {@code with}, {@code e} is a WITH, or an ORDER BY of a WITH, that is


can this be {@code with e}?.
How can it not be a WITH if it's a with?
The period should probably be not here.
Who is "they" who can "see"?

Reworded it to try to be more clear:

/** * Parses zero or more pipe operators that follow the query {@code e}. * * <p>The flag {@code with} signifies whether {@code e} is a WITH that is not * in parentheses, or an ORDER BY of such a WITH. If so, the operators are * applied to the body of the WITH, so that they can reference its CTEs. */

mihaibudiu · 2026-10-25T18:00:54Z

+}
+
+/**
+ * Parses a pipe WHERE operator, and applies it to the query {@code e}.


what does it mean to "apply it to e"?
I don't see e being modified.
Maybe it applies the filter to the result produced by e.

It just means it returns a call whose input is e. Reworded it to be clearer:

/** * Parses a pipe WHERE operator, and returns a SqlPipeWhere whose input * is the query {@code e}. */

mihaibudiu · 2026-10-25T18:01:29Z

+                throw SqlUtil.newContextException(getPos(),
+                    RESOURCE.pipeSyntaxNotAllowed());
+            }
+            throw SqlUtil.newContextException(getPos(),


if this will be done later, maybe you can add a comment.

Ah, this is intentional, the behavior is taken from GoogleSQL. It requires parentheses in such situations. I added a comment.

mihaibudiu · 2026-10-25T18:10:28Z

    RelDataType[] argTypes = new RelDataType[2];
    argTypes[0] = type1;
-    final boolean isSubQuery = node2 instanceof SqlSelect;
+    // Not isA(SqlKind.QUERY): coercing a WITH as a sub-query gives it a type


I don't understand this comment.

I just removed the comment. It was trying to say that we need to use PIPE_QUERY here rather than QUERY. I don't think it was adding much, the line of code stands on its own.

mihaibudiu · 2026-10-25T18:11:09Z

      return rowTypeCoercion(validator.getOverScope(query), body, columnIndex, targetType);
+    case FROM_QUERY:
+    case PIPE_OPERATOR:
+      // A query in pipe syntax is validated and converted as its surrogate


can you define "surrogate"?
Is this a word used in the definition of the pipe syntax?

It was defined in SqlPipeOperator (which is now renamed to SqlPipeCall) although I moved the definition to SqlPipeQuery where the surrogateSelect() method is defined.

mihaibudiu · 2026-10-27T00:29:17Z


+    case PIPE_INPUT:
+      // The input's order matters only if the operator is at the top of the
+      // statement. Then convert the input as top, so that its sort is kept.


what does "convert the input as top" mean?

It means to set the top flag during conversion, which causes the converter to retain ORDER BY. I reworded the comment to try to be clearer:

// If the "bb.top" flag is set, it means we are converting a pipe // operator call at the top of the statement; therefore, we need to // also convert the input with the top flag set. (This causes it to // retain its ordering, which would otherwise be dropped for a // subquery.)

mihaibudiu · 2026-10-27T00:29:57Z

+      final boolean keepOrder = bb.top;
+      SqlNode pipeInput = ((SqlPipeInput) from).getInput();
+      while (pipeInput instanceof SqlWith) {
+        // convertWith would convert the body as the top of the statement.


I also don't know what the top of a statement is

I removed this comment, I don't think it adds much.

mihaibudiu · 2026-10-27T00:33:04Z

    final List<CorrelationId> correlNames = new ArrayList<>();
    // Mapping from (correlId, originalFieldIndex) to projectedFieldIndex for aggregation
    final Map<Pair<CorrelationId, Integer>, Integer> fieldMapping = new HashMap<>();
+    int correlFieldCount = Integer.MAX_VALUE;


This is a bit ugly. Why not just use null?

changed it to a @Nullable Integer.

mihaibudiu · 2026-10-27T00:35:49Z

-    // to update correlation variable row type and field indices
+    // The rel of a pipe operator's input may also have trailing columns for
+    // sort keys (see convertNonAggregateSelectList), which the correlation
+    // variable lacks. Give it the rel's row type.


what is "it"?
What does it mean to give something to a type?

"it" is the correlation variable, and giving it the row type means changing the type of the variable to the rel's row type. I think the code flow here could be improved, so I restructured it a bit to split the logic for aggregation and for pipe operators. They were previously back-to-back. The comment is associated with pipe operators only now, and is reworded to try to be clearer.

mihaibudiu · 2026-10-27T00:37:39Z

  }

+  /**
+   * Converts a query written in pipe syntax, by converting its surrogate


maybe this is true, but there's nothing about surrogates in this body.
In general I would not document the implementation in the javadoc, only in the body.

I moved the comment to the body, and it's now clearer because of the rename of PipeNamespace#getSelect -> PipeNamespace#getSurrogateSelect. (which is now referred to in the body.)

gianm · 2026-10-28T22:49:27Z

@mihaibudiu Thank you for reviewing, I have pushed various updates and replied to your comments.

mihaibudiu · 2026-10-28T23:35:52Z

CI is unhappy. I plan to re-review this after you fix it.

gianm · 2026-10-29T03:48:47Z

CI is unhappy. I plan to re-review this after you fix it.

@mihaibudiu There was a style problem, I just fixed it.

mihaibudiu · 2026-10-30T22:58:18Z

Please, you amend your commit on a 2.3KLOC PR? How am I supposed to find out what you changed? Now I have to read everything again? Maybe you can dig your history and make the changes as a separate commit.

gianm · 2026-10-31T14:55:54Z

Please, you amend your commit on a 2.3KLOC PR? How am I supposed to find out what you changed? Now I have to read everything again? Maybe you can dig your history and make the changes as a separate commit.

I updated the branch to be the commit you originally reviewed + the follow up as a separate commit. I also merged in main, which was needed to fix one of the tests. This approach (follow-up commits, merging main when needed, no force pushes) is more what I am accustomed to doing in other repos.

And, sorry, I had got confused about how things work around here. Originally I tried doing a second commit and it ran afoul of the LintTest. I looked around at what people were doing in other recent PRs, and generally saw force pushing happening, so I figured that it was standard here to avoid having multiple commits on a branch. Looking closer I now see that the LintTest just wants follow up commit messages to be formatted a certain way. Blame trying to get CI working at the end of the day.

sonarqubecloud · 2026-10-31T15:14:54Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
91.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

mihaibudiu

I only left one comment, but I think this is good.
If no one else objects I think we should merge this (this will require the commits to be squashed, but let's wait a bit in case someone else is reviewing).

mihaibudiu · 2026-11-02T23:48:25Z

From my pov you can squash the commits.

gianm · 2026-11-02T23:48:45Z

I only left one comment

I just replied to that comment: #5349 (comment) and pushed a small follow up commit: 2ac63b5. No main code changes, just comments and a test.

This patch adds FROM and |> WHERE from SQL pipe syntax. The syntax is implemented in the core parser, and is available only in certain conformances. The approach is to keep the operators in the parse tree through validation, ultimately converting them to relational operators in SqlToRelConverter. This approach enables us to adhere to pipe syntax rules, especially the rules around table alias visibility and sort ordering retention. It also enables us to write pipe-syntax-aware error messages. To simplify validation and conversion, each pipe operator is associated with a surrogate SELECT that acts as its stand-in. The pipe operator itself is used for pipe-specific validations, and retaining the correct scopes and collations.

gianm · 2026-11-03T20:59:49Z

I have squashed the commits.

mihaibudiu · 2026-11-03T21:39:57Z

I will merge this, but I won't mark [CALCITE-7028] as closed yet, since I think you are planning a follow-up.

gianm · 2026-11-03T21:45:45Z

I will merge this, but I won't mark [CALCITE-7028] as closed yet, since I think you are planning a follow-up.

Thank you. I am hoping to do follow ups for all of the common pipe operators. I could do them all in CALCITE-7028 or I could raise a new Jira for each follow up. If you have a preference please let me know.

Fwiw, the list I had in mind for "common pipe operators" is: SELECT (done in this PR), WHERE (done in this PR), ORDER BY, LIMIT, AGGREGATE, UNION, INTERSECT, EXCEPT, EXTEND, SET, DROP, JOIN, RENAME, AS, and DISTINCT. To avoid huge PRs it would probably need to be ~8 PRs total.

mihaibudiu · 2026-11-03T21:51:54Z

Smaller PRs are easier to review. You can bundle these fixes in a few smaller PRs - it depends really how big they would turn out.

gianm force-pushed the 7028-pipe-syntax branch from cd0c885 to 17f4df7 Compare October 22, 2026 05:23

gianm force-pushed the 7028-pipe-syntax branch from 17f4df7 to c15d68a Compare October 22, 2026 20:53

gianm force-pushed the 7028-pipe-syntax branch from c15d68a to 5c32bb7 Compare October 24, 2026 07:00

mihaibudiu reviewed Oct 27, 2026

View reviewed changes

gianm force-pushed the 7028-pipe-syntax branch from 5c32bb7 to 9d41e13 Compare October 28, 2026 22:45

gianm force-pushed the 7028-pipe-syntax branch from 9d41e13 to 6645402 Compare October 29, 2026 03:48

gianm force-pushed the 7028-pipe-syntax branch from 6645402 to 5f42088 Compare October 31, 2026 14:30

mihaibudiu approved these changes Nov 1, 2026

View reviewed changes

mihaibudiu added the LGTM-will-merge-soon Overall PR looks OK. Only minor things left. label Nov 1, 2026

gianm force-pushed the 7028-pipe-syntax branch from 2ac63b5 to 3e9cfde Compare November 3, 2026 20:59

mihaibudiu merged commit a7f3290 into apache:main Nov 3, 2026
13 checks passed

gianm deleted the 7028-pipe-syntax branch November 3, 2026 21:42

Conversation

gianm commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gianm commented Oct 21, 2026

Uh oh!

mihaibudiu commented Oct 22, 2026

Uh oh!

gianm commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mihaibudiu commented Oct 22, 2026

Uh oh!

gianm commented Oct 22, 2026

Uh oh!

mihaibudiu commented Oct 22, 2026

Uh oh!

mihaibudiu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gianm commented Oct 28, 2026

Uh oh!

mihaibudiu commented Oct 28, 2026

Uh oh!

gianm commented Oct 29, 2026

Uh oh!

mihaibudiu commented Oct 30, 2026

Uh oh!

gianm commented Oct 31, 2026

Uh oh!

sonarqubecloud Bot commented Oct 31, 2026

Quality Gate passed

Uh oh!

mihaibudiu left a comment

Choose a reason for hiding this comment

Uh oh!

mihaibudiu commented Nov 2, 2026

Uh oh!

gianm commented Nov 2, 2026

Uh oh!

gianm commented Oct 21, 2026 •

edited

Loading

gianm commented Oct 22, 2026 •

edited

Loading