test(spark): characterize SparkPredicateToVortexExpression conversion - #9452
Open
jackylee-ch wants to merge 1 commit into
Open
test(spark): characterize SparkPredicateToVortexExpression conversion#9452jackylee-ch wants to merge 1 commit into
jackylee-ch wants to merge 1 commit into
Conversation
`isPushable` documents an invariant: when it returns true, `convert` must succeed, otherwise a predicate Spark is allowed to drop is silently skipped by the reader and the scan returns rows the query excluded. The two sides are maintained by separate code, but only `isPushable` was tested. Assert both sides over every accepted shape: the comparison operators with the column on either side, column-to-column, null checks, `IN`, the string matches with LIKE meta-characters in the needle, bare boolean columns, the constants, nested `AND`/`OR`/`NOT`, and all 13 literal types. Rejections are covered on both sides too, including the empty named reference whose guard lives only in `columnOf`. Signed-off-by: jackylee <qcsd2011@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
isPushabledocuments an invariant: when it returns true,convertmust succeed. OtherwiseVortexScanBuilder.pushPredicateslets Spark drop a predicate thatVortexPartitionReadersilently skips, and the scan returns rows the query excluded. OnlyisPushablewas tested;converthad no coverage.What changes are included in this PR?
Test only. Asserts both sides of the invariant over every accepted shape: the comparison operators with the column on either side, column-to-column, null checks,
IN, the string matches with%,_and\in the needle, bare boolean columns, the constants, nestedAND/OR/NOT, and all 13 literal types. Rejections too: wrong arity, unknown predicate name, empty named reference, and a decimal that does not fit its scale.Verified non-vacuous: five mutations — dropping
BOOLEAN_EXPRESSIONfromconvert, the null decimal literal, the commuted comparison,columnOf's zero-part guard, anddefault -> false— each fail a targeted test.27 pass on
:vortex-spark_2.13:testand on:vortex-spark_2.12:test; the full suites andspotlessCheckare clean apart from the S3Mock test, which needs Docker.What APIs are changed? Are there any user-facing changes?
None.
AI assistance
Prepared with agentic AI assistance. I read the translator end to end and confirmed the mutations above fail.