Repository navigation
Conversation
A join_conditions entry can set null_safe: true to compare its pair with IS NOT DISTINCT FROM instead of =, so a key part that is NULL on both sides matches. Consumers that need NULL-safe composite keys currently COALESCE both sides to a sentinel string. IS NOT DISTINCT FROM is ANSI (no Spark-only <=>) and Spark plans it as an equi-join key. The (a = b OR (a IS NULL AND b IS NULL)) expansion is not used: with no other equi key Spark plans it as a nested-loop join. OutgoingRelationship now validates condition values: null_safe must be a boolean (a quoted 'false' would otherwise read as on), everything else a string. umf.schema.json regenerated. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bead:
tablespec-c0c75177.A composite
join_conditionsentry can now setnull_safe: true. The pair then renders asIS NOT DISTINCT FROMinstead of=, so a key part that is NULL on both sides matches.renders
... AND base.PayorClaimNumber IS NOT DISTINCT FROM target.PayorClaimNumber.Why. Composite join keys always render
=, andNULL = NULLis never true. A downstream pipeline joins on a 4-part key (ServiceLineID+ claim number + NPI + CPT) where some parts are NULL on both sides. Today it works around that by COALESCE-ing both sides to a sentinel string:COALESCE(CAST(x AS STRING), '<null>'). That's verbose, it hides the key columns behind expressions, and a real'<null>'value would falsely match.Why
IS NOT DISTINCT FROM. Checked on Spark 4 (local, broadcast disabled, nullable keys), as a FULL OUTER JOIN condition:IS NOT DISTINCT FROMcoalesce+isnull)COALESCE(...) = COALESCE(...)(a = b OR (a IS NULL AND b IS NULL))So it plans like the sentinel without the collision risk, and it's ANSI SQL, which fits the generator's existing choice to avoid Spark-only
<=>. The OR expansion that thealternative_joinsjoin-back uses was deliberately not reused here.Change
models/umf.py:join_conditionsvalues arestr | bool. A new validator requiresnull_safeto be a boolean, so a quoted'false'can't silently turn it on, and keeps every other key a string. The field description documents the option.schemas/sql_generator.py: a composite condition withnull_saferendersIS NOT DISTINCT FROM. Every other condition is unchanged.schemas/relationship_resolver.py: theJoinInfo.join_conditionstype matches the model. Dicts were already passed through as-is.schemas/umf.schema.json: regenerated.Not in scope: a null-safe primary key pair (
source_column/target_column),join_via/lookup_joinkeys, and changing thealternative_joinsexpansion.Testing
TestCompositeJoinKeysAndFullOuter:null_safe: truerendersIS NOT DISTINCT FROMon that pair only.null_safe: falsekeeps=.null_safeor a non-string column value raisesValidationError.tests/unit/test_custom_gx_parity.pyare a local Spark/Delta jar RPC startup failure and error identically onupstream/main(12 passed, 12 errors).🤖 Generated with Claude Code