[spark][flink] Fix vector-field column conversion in catalogs - #9014
Open
jackylee-ch wants to merge 1 commit into
Open
[spark][flink] Fix vector-field column conversion in catalogs#9014jackylee-ch wants to merge 1 commit into
jackylee-ch wants to merge 1 commit into
Conversation
### Purpose `SparkCatalog.toInitialSchema` converts a column listed in the `vector-field` option into a Paimon `VECTOR`. Three problems on that path: 1. The array check reports `"The type of blob field must be array"` — the message was copied from the BLOB branch above it and names the wrong option. 2. `checkArgument` is called with a template containing `%s` but no varargs, so `Preconditions` leaves the placeholders unsubstituted and the user is told to set the literal key `field.%s.vector-dim`. `LogicalTypeConversion.toVectorType` on the Flink side has the same defect. 3. `Integer.parseInt(properties.get(dimKey))` is unguarded, so a non-integer dimension surfaces as a raw `NumberFormatException: For input string: "abc"` from inside the catalog. Flink already guards this; Spark did not. `DataTypes.VECTOR(length, element)` also forces `isNullable = true`, so a `NOT NULL` array column declared in Spark silently lost its nullability. The BLOB branch in the same method and the `__VECTOR_FIELD` comment-directive path in `ColumnDirectiveUtils.convertType` both preserve it. ### Changes - Extract the vector conversion into `SparkCatalog#toVectorType`, mirroring the existing `toBlobType`: name the option correctly, pass the `checkArgument` arguments, reject blank/non-integer dimensions with the option key in the message, and build `VectorType` with the declared nullability. - Align the two `checkArgument` messages in Flink's `toVectorType` and reuse `StringUtils.isNullOrWhitespaceOnly` so a whitespace-only dimension is rejected the same way in both engines. No storage format or option semantics change; only validation messages and the nullability carried into the schema. ### Tests - Added `Paimon DDL: create table with vector-field` and `Paimon DDL: create table with invalid vector-field` to `DDLTestBase`, covering the round-trip (length, element type, `NOT NULL`) and all four rejection paths. These run for every Spark version via the per-module `DDLTest` subclasses. - `mvn -pl paimon-spark/paimon-spark-3.5 -am -Pfast-build,spark3 -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.DDLTest -Dtest=none test` → 35 succeeded, 0 failed (33 on master before this change). - `mvn -pl paimon-flink/paimon-flink-common -Pfast-build -Dtest=LogicalTypeConversionTest test` → 6 succeeded, 0 failed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
SparkCatalog.toInitialSchemaconverts a column listed invector-fieldinto a PaimonVECTOR. Problems on that path:The type of blob field must be array— copied from the BLOB branch above it, naming the wrong option.checkArgumentis given a template with%sbut no varargs, so the user is told to set the literal keyfield.%s.vector-dim. Flink'sLogicalTypeConversion.toVectorTypehas the same defect.Integer.parseInt(properties.get(dimKey))is unguarded, so a non-integer dimension surfaces as a rawNumberFormatException. Flink already guards this.DataTypes.VECTOR(length, element)forcesisNullable = true, soNOT NULLwas silently dropped. The BLOB branch in the same method andColumnDirectiveUtils.convertTypeboth preserve it.Extracts the conversion into
SparkCatalog#toVectorType, mirroring the existingtoBlobType. No storage format or option semantics change.Tests
DDLTestBase,LogicalTypeConversionTest.