Skip to content

fix: preserve concat binary string bytes - #5267

Open
manuzhang wants to merge 1 commit into
apache:mainfrom
manuzhang:codex/config-concat-binary-as-string
Open

fix: preserve concat binary string bytes#5267
manuzhang wants to merge 1 commit into
apache:mainfrom
manuzhang:codex/config-concat-binary-as-string

Conversation

@manuzhang

Copy link
Copy Markdown
Member

Which issue does this PR close?

Relates to #4180.

Rationale for this change

spark.sql.function.concatBinaryAsString=true makes Spark insert BinaryType-to-StringType casts before concat. Spark preserves malformed bytes in those casts, while Comet's native Arrow string path normalizes them to UTF-8 replacement bytes. For example, Spark keeps FFFE41, but Comet produced EFBFBDEFBFBD41.

What changes are included in this PR?

  • Route concat trees containing BinaryType-to-StringType casts through Spark's generated code inside the Comet pipeline.
  • Add a two-value SQL config matrix covering result bytes, malformed UTF-8, empty values, and nulls.
  • Add a Scala assertion for result types and direct proof that both config values use the JVM codegen dispatcher without operator fallback.
  • Record the supported behavior in the Spark configuration contributor guide.

How are these changes tested?

  • make core
  • Focused CometSqlFileTestSuite concat_binary_as_string on Spark 3.4.3, 3.5.8, 4.0.2, and 4.1.2
  • Focused CometCodegenSuite concatBinaryAsString on the same versions
  • ./mvnw spotless:check -Pspark-4.1 -Pjdk17 -DskipTests -Dscalastyle.skip=true
  • Dynamic documentation generation for all four Spark profiles

Co-authored-by: Codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant