Skip to content

feat(sql-plan): null-safe composite join_conditions [tablespec-c0c75177] - #59

Open
dmautz1 wants to merge 1 commit into
DocumentDrivenDX:mainfrom
dmautz1:feat/null-safe-join-conditions
Open

dmautz1 wants to merge 1 commit into
DocumentDrivenDX:mainfrom
dmautz1:feat/null-safe-join-conditions

Conversation

@dmautz1

@dmautz1 dmautz1 commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Bead: tablespec-c0c75177.

A composite join_conditions entry can now set null_safe: true. The pair then renders as IS NOT DISTINCT FROM instead of =, so a key part that is NULL on both sides matches.

join_conditions:
- source_column: PayorClaimNumber
  target_column: PayorClaimNumber
  null_safe: true

renders ... AND base.PayorClaimNumber IS NOT DISTINCT FROM target.PayorClaimNumber.

Why. Composite join keys always render =, and NULL = NULL is never true. A downstream pipeline joins on a 4-part key (ServiceLineID + claim number + NPI + CPT) where some parts are NULL on both sides. Today it works around that by COALESCE-ing both sides to a sentinel string: COALESCE(CAST(x AS STRING), '<null>'). That's verbose, it hides the key columns behind expressions, and a real '<null>' value would falsely match.

Why IS NOT DISTINCT FROM. Checked on Spark 4 (local, broadcast disabled, nullable keys), as a FULL OUTER JOIN condition:

Form With another equi key As the only key
IS NOT DISTINCT FROM sort-merge join, null-safe pair is a join key (coalesce + isnull) sort-merge join
sentinel COALESCE(...) = COALESCE(...) sort-merge join sort-merge join
(a = b OR (a IS NULL AND b IS NULL)) sort-merge join on the other key + residual filter BroadcastNestedLoopJoin

So it plans like the sentinel without the collision risk, and it's ANSI SQL, which fits the generator's existing choice to avoid Spark-only <=>. The OR expansion that the alternative_joins join-back uses was deliberately not reused here.

Change

  • models/umf.py: join_conditions values are str | bool. A new validator requires null_safe to be a boolean, so a quoted 'false' can't silently turn it on, and keeps every other key a string. The field description documents the option.
  • schemas/sql_generator.py: a composite condition with null_safe renders IS NOT DISTINCT FROM. Every other condition is unchanged.
  • schemas/relationship_resolver.py: the JoinInfo.join_conditions type matches the model. Dicts were already passed through as-is.
  • schemas/umf.schema.json: regenerated.
  • FEAT-019: documents the option.

Not in scope: a null-safe primary key pair (source_column/target_column), join_via/lookup_join keys, and changing the alternative_joins expansion.

Testing

  • New tests in TestCompositeJoinKeysAndFullOuter:
    • null_safe: true renders IS NOT DISTINCT FROM on that pair only.
    • null_safe: false keeps =.
    • A string null_safe or a non-string column value raises ValidationError.
  • CI fast suite locally: 4044 passed, 25 skipped. The 12 errors in tests/unit/test_custom_gx_parity.py are a local Spark/Delta jar RPC startup failure and error identically on upstream/main (12 passed, 12 errors).
  • ruff 0.13.3 check and format are clean.

🤖 Generated with Claude Code

A join_conditions entry can set null_safe: true to compare its pair with
IS NOT DISTINCT FROM instead of =, so a key part that is NULL on both
sides matches. Consumers that need NULL-safe composite keys currently
COALESCE both sides to a sentinel string.

IS NOT DISTINCT FROM is ANSI (no Spark-only <=>) and Spark plans it as an
equi-join key. The (a = b OR (a IS NULL AND b IS NULL)) expansion is not
used: with no other equi key Spark plans it as a nested-loop join.

OutgoingRelationship now validates condition values: null_safe must be a
boolean (a quoted 'false' would otherwise read as on), everything else a
string. umf.schema.json regenerated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant