[iwanttobepowerful]

CALCITE-7882

1. Symptom

With expand=false, a JOIN ... ON condition containing multiple sub-queries can fail with “Unable to implement”:

SELECT e.ename
FROM emp e JOIN dept d
  ON e.deptno = d.deptno
  AND e.sal < (SELECT max(s.sal) FROM emp s)
  AND EXISTS (SELECT 1 FROM emp s WHERE s.mgr = e.empno)

The scalar sub-query is uncorrelated, while the EXISTS references the current join scope. Both remain in the ON condition as RexSubQuery, and the Join declares the correlation variable used by the EXISTS, for example variablesSet=[$cor0].

2. Root cause: rewriting one sub-query loses the scope of the remaining ones

SubQueryRemoveRule.matchJoin processes one selected sub-query per invocation and relies on repeated rule application. Equal occurrences of the selected sub-query are replaced together.

In the original implementation:

1. The first selected sub-query is the uncorrelated scalar query. It has no operands or local correlation dependencies, so the rule takes the right-input branch.

2. That branch rebuilds the Join using the two-argument RelBuilder.join overload, which supplies an empty variablesSet.

3. The rebuilt Join therefore stops declaring $cor0, although its ON condition still contains the correlated EXISTS referencing $cor0.EMPNO.

4. On the next invocation, the rule intersects the variables used by the EXISTS with the rebuilt Join’s now-empty variablesSet. The result is empty, so the sub-query is incorrectly treated as uncorrelated.

5. The rewrite creates a plain Join rather than the required Correlate. The correlation reference survives into the executable plan without a binding, causing implementation to fail.

The underlying problem is therefore loss of correlation-variable scope across incremental sub-query rewrites, rather than an incorrectly computed scalar expression.

3. Preserve the scope and remap correlation references

(a) Keep declaring variables needed by the remaining sub-queries

The rule computes remainingVariables by intersecting:

• the variables declared by the original Join; and
• the variables used by the other sub-queries still present in the ON condition.

Both single-side branches rebuild the Join through pushJoin.

When remainingVariables is non-empty, pushJoin explicitly creates a LogicalJoin declaring those variables. It does not use:

• RelBuilder.join with a non-empty variable set, because that may create a Correlate, which cannot represent an arbitrary ON condition referencing both inputs;
• Join.copy, because that would retain the original, potentially stale variable set.

When no variables need to remain defined, pushJoin can use the ordinary builder overload.

For the cross-product branch, the remaining variables are instead declared on the replacement Filter.

(b) Update both the correlation row type and field indexes

A correlation variable declared by the Join references the combined input row. Rewriting a sub-query may append auxiliary fields, changing that row’s shape.

remapCorrelationVariables uses a Mapping and RelOptUtil.RexCorrelVariableMapShuttle to update the correlation row type and field-access indexes, including references inside nested sub-queries.

Branch	Auxiliary fields appended	Remapping of remaining variables
Cross-product + Filter	After both original inputs	Retype only
Left-input rewrite	Between the original left and right fields	Retype and shift right-side indexes
Right-input rewrite	After the original right fields	Retype only

The selected sub-query also needs its correlation references adapted to the input on which it is rewritten:

• a left-side rewrite changes the correlation row type to the left input;
• a right-side rewrite changes it to the right input and subtracts nFieldsLeft from the relevant indexes.

Indexes shifted outside the target row type are left unmapped, so invalid references fail rather than silently resolving to the wrong field.

The correlation row type is constructed from both inputs, not simply from the Join’s output type. This distinction matters for SEMI and ANTI joins, whose ON conditions can reference the right input even though their output does not contain it.

(c) Separate correlation IDs shared by the selected and remaining sub-queries

Preserving remainingVariables is not sufficient when multiple correlated sub-queries share an ID.

For example, two EXISTS sub-queries may both reference $cor0. Rewriting one of them moves its binding to an auxiliary Correlate, while the other still needs the original join scope. Reusing $cor0 for both would associate the same ID with different binding contexts and row types.

renameSharedVariables assigns fresh IDs only to variables shared with the remaining sub-queries. Non-shared variables and outer-scope correlations are left unchanged.

The uncorrelated scalar query in the original example does not share an ID with the correlated EXISTS; it demonstrates scope preservation, while shared-ID handling addresses an additional case.

4. Use one complete-expression renaming implementation

Correlation renaming is now shared between:

• the selected sub-query in matchJoin; and
• the predicate moved to a domain join in rewriteOnDomain.

renameSharedVariables accepts a RexNode and returns a Pair<RexNode, Set<CorrelationId>> containing the rewritten expression and its updated variable set.

It delegates to the Rex-expression overload of DeduplicateCorrelateVariables.go, which reuses the existing traversal and covers:

• ordinary expression operands;
• sub-queries nested inside another sub-query’s operands;
• each sub-query’s relational expression;
• further sub-queries nested within those relational expressions.

This removes the separate renaming implementations and prevents traversal-depth inconsistencies.

In particular, renaming only subQuery.rel is insufficient when an IN operand contains a correlated scalar sub-query. If that operand retains the old ID while the domain declares only the new one, its correlation reference becomes undefined. The complete-expression traversal handles both locations consistently.

5. Rewrite operand-nested sub-queries innermost first

The current implementation uses findInnermostSubQuery: it visits a sub-query’s operands before selecting the enclosing sub-query.

This ensures that an inner sub-query becomes an input-field reference before the enclosing query is rewritten. Otherwise, rewriting the outer query can move the inner query into an auxiliary Join condition and leave it outside the current expression-rewrite loop.

This selection order is used by the ordinary Project, Filter and Join paths, and by applyEnableMarkJoin.

“Innermost” here refers to Rex operand nesting. The finder does not traverse the relational expression stored in RexSubQuery.rel.

LogicVisitor.visitSubQuery also visits operands using TRUE_FALSE_UNKNOWN, because their values are compared with sub-query rows. FALSE and UNKNOWN must not be discarded merely because the enclosing expression appears in a filtering context.

6. Handle dependencies spanning both join inputs

A selected sub-query may depend on both inputs through its operands, correlation references, or both.

For an INNER join, the rule can rewrite the join as a cross-product followed by a Filter, preserving the remaining correlation scope on that Filter.

For supported non-inner join types, rewriteOnDomain instead moves the relevant predicate to an INNER join of duplicate-free input domains and joins the original inputs back to the matching bindings:

• domain keys use null-safe equality;
• the original inputs retain their multiplicities;
• shared IDs between the domain predicate and the remaining condition are separated using the same renaming helper;
• FULL joins evaluate the entire ON condition on the domain and use a marker to distinguish unmatched domain rows from genuine NULL-key bindings;
• the transformation is skipped when the duplicated inputs or moved predicate contain non-deterministic calls.

This domain transformation does not itself remove the sub-queries. Subsequent rule invocations remove them from the domain’s INNER join condition.

These changes preserve correlation scope, binding identity and field geometry throughout incremental rewrites. They do not establish that every join-type-specific row-preservation case is solved; the separately discussed single-side ANTI/LEFT_MARK Logic selection remains outside this fix.

[mihaibudiu]

I see this is still a draft. Should I wait with the review?

[iwanttobepowerful]

I see this is still a draft. Should I wait with the review?

I'd like more time to consider the most appropriate implementation approach.

[mihaibudiu]

The high-level plan makes sense, but this won't be easy to review.
I hope that unsupported constructs will provide a clear error message.

[mihaibudiu]

Please describe how these results were validated in a comment.

[zzwqqq]

Should we also consider non-repeatable Sample inputs? Reading them twice can produce different samples and incorrect results.

[iwanttobepowerful]

Thx. I'll look into it later.

[zzwqqq]

Should we use the configured JoinFactory here to support custom join types?

[zzwqqq]

Should we use the configured JoinFactory here as well?

[iwanttobepowerful]

I'll look into it later.

[mihaibudiu]

While looking at [CALCITE-7894] I came up with this test which seems to be fixed by this PR:

select (select count(*) from (values (0, 1), (1, 2), (2, 3)) as l(k, a)
        where exists (select 1 from (values (0), (1), (2)) as s(v) where s.v = l.k)) as c1,
       (select count(*) from (values (0, 1), (1, 2), (5, 3)) as r(k, b)
        where exists (select 1 from (values (0), (1), (2)) as s(v) where s.v = r.k)) as c2;

[iwanttobepowerful]

While looking at [CALCITE-7894] I came up with this test which seems to be fixed by this PR:

select (select count(*) from (values (0, 1), (1, 2), (2, 3)) as l(k, a)
        where exists (select 1 from (values (0), (1), (2)) as s(v) where s.v = l.k)) as c1,
       (select count(*) from (values (0, 1), (1, 2), (5, 3)) as r(k, b)
        where exists (select 1 from (values (0), (1), (2)) as s(v) where s.v = r.k)) as c2;

This case was already working before; it is not affected by this change.

[iwanttobepowerful]

The high-level plan makes sense, but this won't be easy to review. I hope that unsupported constructs will provide a clear error message.

Sorry, I've bundled several fixes into one commit. I'll split them into multiple smaller PRs later so they're easier to review.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
82.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[mihaibudiu]

checkstyle is unhappy