Preserve empty-input values when pulling up aggregate subqueries (bugfix to main)

[Alena0704]

What does this PR do?

Fix incorrect results in Cloudberry's PostgreSQL planner (optimizer = off, or an ORCA fallback) when a correlated scalar aggregate subquery in WHERE is pulled up into a join.

Previously, convert_EXPR_to_join() could use an inner join and discard outer rows with no matching inner input, even when the comparison against the aggregate's empty-input value should pass.

For example:

CREATE TABLE t_out (a int) DISTRIBUTED BY (a);
CREATE TABLE t_in (a int) DISTRIBUTED BY (a);
INSERT INTO t_out VALUES (1);
INSERT INTO t_in VALUES (2);

SET optimizer = off;

SELECT *
FROM t_out
WHERE a > (
    SELECT count(*)
    FROM t_in
    WHERE t_in.a = t_out.a
);

Before the fix, this query returns no rows.

postgres=# set optimizer=0;
SET
postgres=# select * from t_out where a > (select count(*) from t_in where t_in.a = t_out.a); --incorrect result
 a
---
(0 rows)

postgres=# set optimizer=1;
SET
postgres=# select * from t_out where a > (select count(*) from t_in where t_in.a = t_out.a);
 a
---
 1
(1 row)

Before the fix - the query returns no rows:

Gather Motion 3:1  (slice1; segments: 3)
  ->  Hash Join
        Hash Cond: (t_out.a = "Expr_SUBQUERY".csq_c0)
        Join Filter: (t_out.a > "Expr_SUBQUERY".csq_c1)
        ->  Seq Scan on t_out
        ->  Hash
              ->  Subquery Scan on "Expr_SUBQUERY"
                    ->  HashAggregate
                          Group Key: t_in.a
                          Filter: (t_in.a > count(*))
                          ->  Seq Scan on t_in
Optimizer: Postgres query optimizer

With the fix, it returns:

postgres=# set optimizer=0;
SET
postgres=# select * from t_out where a > (select count(*) from t_in where t_in.a = t_out.a);
 a
---
 1
(1 row)

After the fix - the query returns one row with a = 1:

Gather Motion 3:1  (slice1; segments: 3)
  ->  Hash Left Join
        Hash Cond: (t_out.a = "Expr_SUBQUERY".csq_c0)
        Filter: (t_out.a > CASE WHEN "Expr_SUBQUERY".csq_c1 THEN "Expr_SUBQUERY".csq_c2 ELSE '0'::bigint END)
        ->  Seq Scan on t_out
        ->  Hash
              ->  Subquery Scan on "Expr_SUBQUERY"
                    ->  HashAggregate
                          Group Key: t_in.a
                          ->  Seq Scan on t_in
Optimizer: Postgres query optimizer

There is no matching inner row, so the subquery returns 0 and the comparison is 1 > 0.

Preserve aggregate values on empty input

Keep an inner join only when the comparison against the empty-input value folds to FALSE or NULL. Otherwise, use a left join, add a match flag, and evaluate the comparison above the join using:

CASE WHEN matched
     THEN actual_value
     ELSE value_on_empty_input
END

The match flag distinguishes a missing group from an existing group whose expression is NULL. The comparison must remain above the left join: placing it in the join condition could make a matched row that fails the comparison appear missing and pass using the empty-input value.

Derive empty-input constants from pg_aggregate, including initial transition values and foldable final functions. For supported cases that require executor evaluation, construct empty-input aggregate subqueries that become InitPlans. This handles non-NULL defaults beyond count(), including regr_count(), hypothetical-set aggregates, and aggregates with a non-NULL initial state.

Recognize NULL results of the built-in numeric and int8 AVG/SUM final functions on a NULL internal state, allowing strict comparisons to retain inner joins without an empty-input InitPlan. Also recognize the empty-input value 1 for built-in hypothetical rank() with constant direct arguments.

Preserve expression evaluation behavior

Keep only grouping keys, the match flag, and aggregate values in the grouped subquery. Evaluate expressions around aggregates in the matched CASE branch above the join, so an unused inner group cannot cause an error such as division by zero. Mark the inputs with the left join's nulling relid.

Use InitPlan parameters where substituting empty-input constants into compound expressions could cause errors during planning, such as in 10 / count(*) or 10 / coalesce(sum(x), 0). Evaluate the surrounding expression at execution time when its branch is needed. An empty outer table or a query where every outer row has a match must not fail merely because the hypothetical empty-input expression would raise an error.

Do not fold user-defined final functions or invoke them speculatively through empty-input InitPlans. Retain correlated execution unless the aggregate's final function is safe on empty input.

Keep unsupported window functions, correlated targetlists, and clauses whose variables are unavailable at the join attachment point on their existing SubPlan path. Allow nested subqueries inside aggregate arguments and FILTER, keeping them at their original query level in the grouped subquery. Retain correlated execution for SubLinks in the surrounding expression.

Regression coverage

Add PostgreSQL planner regression coverage for empty and matching outer inputs, multiple aggregates, NULL and non-NULL defaults, unsafe final functions, and parallel execution. Check that comparisons rejecting empty input still use inner joins, and cover the empty-input optimization for hypothetical rank().

Refreshed affected PostgreSQL planner plans in subselect_gp, eagerfree, and the PAX answer files.

Type of Change

• Bug fix (non-breaking change)
• New feature (non-breaking change)
• Breaking change (fix or feature with breaking changes)
• Documentation update

Test Plan

• Added/updated SQL regression tests and expected output.
• Compiler syntax checks passed for the updated PostgreSQL planner code.
• git diff --check passed.
• Run regression tests on the final PostgreSQL-only branch.
• Passed make installcheck.
• Passed make -C src/test installcheck-cbdb-parallel.

The full regression suites have not been run on this final PostgreSQL-only version.

Impact

Queries whose aggregate comparison can pass on empty input now preserve unmatched outer rows. Comparisons proven to reject empty input retain inner joins. Unsupported or unsafe cases retain correlated execution.

Some queries require a left join and an empty-input InitPlan to preserve their semantics. Recognized constant empty-input values avoid unnecessary InitPlans. No new dependencies are introduced.

Additional Context

Updated from open-gpdb/cloudberry commit ad62f729c20ee00447bd926cfafd65310ecf066e (PR #89), originally adapted from open-gpdb/gpdb PR #397 and GreengageDB/greengage PR #546.

Co-Authored-By: excaliiibur [email protected]

Checklist

• Followed contribution guide
• Added/updated documentation
• Reviewed code for security implications
• This PR contains AI-assisted code generation
• Requested review from cloudberry committers

[leborchuk]

LGTM

[Alena0704]

Updated commit - fixed comment explanation.

[Alena0704]

Found the bug, reproduction:

SET optimizer = off;

CREATE TABLE test_outer (a int) DISTRIBUTED BY (a);
CREATE TABLE test_inner (a int) DISTRIBUTED BY (a);

INSERT INTO test_outer VALUES (1), (2);
INSERT INTO test_inner VALUES (1);

SELECT o.a
FROM test_outer o
WHERE 0 = (
    SELECT regr_count(i.a, i.a)
    FROM test_inner i
    WHERE i.a = o.a
);

-- Expected: one row, a = 2.
-- Actual with only the original COUNT fix: no rows.

DROP TABLE test_outer, test_inner;
RESET optimizer;

The original code in the pr assumed that every aggregate except count() returns NULL on empty input. This is incorrect for aggregates such as regr_count(), which returns 0, and user-defined aggregates with a non-NULL initial value.
For example, when there are no matching inner rows, 0 = (SELECT regr_count(...) ...) is true. The planner instead treated it as 0 = NULL and chose an INNER JOIN, incorrectly dropping the outer row.
The fix derives the empty-input value from the aggregate’s catalog definition where possible, or computes it through an InitPlan otherwise.

test_outer contains rows 1 and 2, while test_inner contains only 1.
For o.a = 1, the subquery finds one matching row, so regr_count() returns 1. The condition 0 = 1 is false.
For a = 2, there are no matching inner rows, so regr_count() returns 0 and the condition 0 = 0 is true, and the query must return 2. The original fix incorrectly assumed that every aggregate except count() returns NULL on empty input. It therefore chose an INNER JOIN and dropped this row.
Now patch derives the correct empty-input value from the aggregate’s catalog definition and preserves the row.

[Alena0704]

Since my previous comment, I’ve updated the patch to preserve when expressions around aggregates are evaluated, retain decorrelation through HAVING in ORCA, and avoid unnecessary empty-input calculations for several built-in aggregates.

The examples below use actual EXPLAIN (COSTS OFF) output from the current code on a three-primary-segment cluster. Queries were executed with both optimizer=off and optimizer=on; plans shown as GPORCA were checked for actual ORCA execution. Result rows are sorted for presentation where the query does not specify an order.

Common SQL setup

Common setup:

SET optimizer_enable_master_only_queries = on;
SET optimizer_segments = 3;
SET optimizer_nestloop_factor = 1.0;

CREATE TABLE demo_outer (a int) DISTRIBUTED BY (a);
CREATE TABLE demo_inner (a int, b int) DISTRIBUTED BY (a);
INSERT INTO demo_inner VALUES (1, 10), (1, 20), (2, NULL);
ANALYZE demo_inner;

1. Expressions around aggregates are evaluated in the appropriate CASE branch above the join.

The reported division-by-zero error also involved evaluating the expression for an unused inner group. The grouped inner side now computes individual aggregate values and a match flag. The surrounding expression is evaluated after joining to the outer row.

SQL, PG/ORCA plans and results

SELECT o.a
FROM demo_outer o
WHERE o.a < (
    SELECT 100 / coalesce(sum(i.b), 0)
    FROM demo_inner i
    WHERE i.a = o.a
);

The following are four independent tests of the same SELECT above. Each setup replaces the rows in demo_outer. The inner table stays unchanged in every test:

demo_inner
a | b
--+-----
1 | 10
1 | 20
2 | NULL

Run the SELECT above after each setup. The results below were verified with both optimizer=off and optimizer=on.

Test 1 — no outer rows.

TRUNCATE demo_outer;

Result of the SELECT:

a
-
(0 rows)

There is no outer row that needs the expression, so the query must finish without a division-by-zero error.

Test 2 — one outer row, with a=1.

TRUNCATE demo_outer;
INSERT INTO demo_outer VALUES (1);

Result of the SELECT:

a
-
1
(1 row)

For this outer row, the matching inner values are 10 and 20. Their sum is 30, integer division gives 100 / 30 = 3, and 1 < 3 is true. The inner row (2, NULL) is not used by this outer row and must not cause an error.

Test 3 — two outer rows, with a=1 and a=2.

TRUNCATE demo_outer;
INSERT INTO demo_outer VALUES (1), (2);

Result of the SELECT:

ERROR: division by zero

The outer row with a=2 now needs the inner row (2, NULL). Its sum is NULL, so coalesce(sum(i.b), 0) is zero. The division by zero is required by the query, and the whole SELECT fails.

Test 4 — two outer rows, with a=1 and a=3.

TRUNCATE demo_outer;
INSERT INTO demo_outer VALUES (1), (3);

Result of the SELECT:

ERROR: division by zero

There is no inner row with a=3. The SUM over that empty input is NULL, and coalesce again supplies zero. The whole SELECT must fail when evaluating this outer row.

Tests 1 and 2 reproduce the cases that previously raised an unwanted error in the comment. Tests 3 and 4 check that the patch still reports division by zero when the original query requires it.

The plans below are for Test 2. Restore its outer-table contents first:

TRUNCATE demo_outer;
INSERT INTO demo_outer VALUES (1);
ANALYZE demo_outer;

Current ORCA plan (SET optimizer=on):

Gather Motion 3:1  (slice1; segments: 3)
  ->  Result
        Filter: (o.a < CASE WHEN (true) THEN (100 / COALESCE((sum(i.b)), '0'::bigint)) ELSE (100 / COALESCE(NULL::bigint, '0'::bigint)) END)
        ->  Hash Left Join
              Hash Cond: (o.a = i.a)
              ->  Seq Scan on demo_outer o
              ->  Hash
                    ->  GroupAggregate
                          Group Key: i.a
                          ->  Sort
                                Sort Key: i.a
                                ->  Seq Scan on demo_inner i
Optimizer: GPORCA

Current PG plan (SET optimizer=off):

Gather Motion 3:1  (slice1; segments: 3)
  InitPlan 1 (returns $0)  (slice2)
    ->  Aggregate
          ->  Result
                One-Time Filter: false
  ->  Hash Left Join
        Hash Cond: (o.a = "Expr_SUBQUERY".csq_c0)
        Filter: (o.a < CASE WHEN "Expr_SUBQUERY".csq_c1 THEN (100 / COALESCE("Expr_SUBQUERY".csq_c2, '0'::bigint)) ELSE (100 / COALESCE($0, '0'::bigint)) END)
        ->  Seq Scan on demo_outer o
        ->  Hash
              ->  Subquery Scan on "Expr_SUBQUERY"
                    ->  HashAggregate
                          Group Key: i.a
                          ->  Seq Scan on demo_inner i
Optimizer: Postgres query optimizer

The division is in the join filter’s CASE, rather than in the grouped inner projection. In the PG plan, $0 is the aggregate’s empty-input value; the InitPlan does not compute the surrounding division. In ORCA’s display, (true) is the inner match flag and becomes NULL for an unmatched row.

2. The existing correlated execution path was adapted to the new expression placement.

SQL, ORCA plan and result

With demo_outer still containing only (1):

SELECT o.a
FROM demo_outer o
WHERE o.a + 4 IN (
    SELECT count(*) + 100 / coalesce(sum(i.b), 0)
    FROM demo_inner i
    WHERE i.a <= o.a
);

Current ORCA plan:

Result
  Filter: (SubPlan 1)
  ->  Gather Motion 3:1  (slice1; segments: 3)
        ->  Seq Scan on demo_outer o
  SubPlan 1
    ->  Aggregate
          ->  Result
                Filter: (i.a <= o.a)
                ->  Materialize
                      ->  Gather Motion 3:1  (slice2; segments: 3)
                            ->  Seq Scan on demo_inner i
Optimizer: GPORCA

Result with both optimizer settings:

a
-
1
(1 row)

This query still uses a SubPlan: this change does not add decorrelation for <=. Internally, the correlated Apply representation now carries all aggregate values and the match flag needed by the expression moved above it, instead of assuming one output column. This is a supporting adaptation for the expression-placement change.

Additional data for examples 3–5

For the remaining examples, add the all-NULL and missing groups on the outer side:

INSERT INTO demo_outer VALUES (2), (3);
ANALYZE demo_outer;

3. ORCA preserves decorrelation through HAVING while evaluating the target expression only for accepted rows.

HAVING: SQL, ORCA plan and results

SELECT o.a
FROM demo_outer o
WHERE 0 < (
    SELECT 100 / count(*)
    FROM demo_inner i
    WHERE i.a = o.a
    HAVING count(*) > 0
);

Current ORCA plan:

Gather Motion 3:1  (slice1; segments: 3)
  ->  Result
        Filter: (0 < CASE WHEN CASE WHEN (true) THEN ((count(*)) > 0) ELSE ('0'::bigint > 0) END THEN CASE WHEN (true) THEN (100 / (count(*))) ELSE (100 / '0'::bigint) END ELSE NULL::bigint END)
        ->  Hash Left Join
              Hash Cond: (o.a = i.a)
              ->  Seq Scan on demo_outer o
              ->  Hash
                    ->  GroupAggregate
                          Group Key: i.a
                          ->  Sort
                                Sort Key: i.a
                                ->  Seq Scan on demo_inner i
Optimizer: GPORCA

Result with both optimizer settings:

a
-
1
2
(2 rows)

The CASE first checks HAVING against the matched or empty-input aggregate value. For a=3, count(*) is zero and HAVING rejects the row, so 100 / 0 is never evaluated. The scalar values can also be inspected directly:

SELECT o.a,
       (SELECT 100 / count(*)
        FROM demo_inner i
        WHERE i.a = o.a
        HAVING count(*) > 0) AS quotient
FROM demo_outer o
ORDER BY o.a;

a | quotient
--+---------
1 |       50
2 |      100
3 |     NULL
(3 rows)

The PG planner continues to use a SubPlan for this HAVING query; the new decorrelation support described here is in ORCA.

4. Strict comparisons with numeric and bigint AVG/SUM retain INNER joins.

AVG: SQL, PG/ORCA plans and result

SELECT o.a
FROM demo_outer o
WHERE 40 > (
    SELECT avg(i.b::numeric)
    FROM demo_inner i
    WHERE i.a = o.a
);

Current ORCA plan:

Gather Motion 3:1  (slice1; segments: 3)
  ->  Hash Join
        Hash Cond: (o.a = i.a)
        ->  Seq Scan on demo_outer o
        ->  Hash
              ->  Result
                    Filter: ('40'::numeric > (avg((i.b)::numeric)))
                    ->  GroupAggregate
                          Group Key: i.a
                          ->  Sort
                                Sort Key: i.a
                                ->  Seq Scan on demo_inner i
Optimizer: GPORCA

Current PG plan:

Gather Motion 3:1  (slice1; segments: 3)
  ->  Hash Join
        Hash Cond: (o.a = "Expr_SUBQUERY".csq_c0)
        ->  Seq Scan on demo_outer o
        ->  Hash
              ->  Subquery Scan on "Expr_SUBQUERY"
                    ->  HashAggregate
                          Group Key: i.a
                          Filter: ('40'::numeric > avg((i.b)::numeric))
                          ->  Seq Scan on demo_inner i
Optimizer: Postgres query optimizer

Result with both optimizer settings:

a
-
1
(1 row)

Both plans use an ordinary Hash Join, without an empty-input InitPlan or an extra empty aggregate branch. The missing group produces NULL for AVG, so it cannot satisfy 40 > AVG(...). The same empty-input recognition was added for SUM and for bigint inputs; the regression tests cover all four combinations. Expressions such as coalesce(avg(...), 0) still require their own empty-input handling.

5. A separate commit replaces the empty-input value of hypothetical rank with 1 when its direct arguments are constants.

rank: SQL, PG/ORCA plans and result

SELECT o.a
FROM demo_outer o
WHERE o.a > (
    SELECT rank(5) WITHIN GROUP (ORDER BY i.b)
    FROM demo_inner i
    WHERE i.a = o.a
);

Before this optimization, the PG regression plan calculated the empty-input value through an InitPlan, with ELSE $0 in the comparison. ORCA used an additional aggregate over an empty relation and a join to its result. The current plans use ELSE '1'::bigint.

Current PG plan:

Gather Motion 3:1  (slice1; segments: 3)
  ->  Hash Left Join
        Hash Cond: (o.a = "Expr_SUBQUERY".csq_c0)
        Filter: (o.a > CASE WHEN "Expr_SUBQUERY".csq_c1 THEN "Expr_SUBQUERY".csq_c2 ELSE '1'::bigint END)
        ->  Seq Scan on demo_outer o
        ->  Hash
              ->  Subquery Scan on "Expr_SUBQUERY"
                    ->  GroupAggregate
                          Group Key: i.a
                          ->  Sort
                                Sort Key: i.a
                                ->  Seq Scan on demo_inner i
Optimizer: Postgres query optimizer

Current ORCA plan:

Gather Motion 3:1  (slice1; segments: 3)
  ->  Result
        Filter: (o.a > CASE WHEN (true) THEN (rank(5) WITHIN GROUP (ORDER BY i.b)) ELSE '1'::bigint END)
        ->  Hash Left Join
              Hash Cond: (o.a = i.a)
              ->  Seq Scan on demo_outer o
              ->  Hash
                    ->  GroupAggregate
                          Group Key: i.a
                          ->  Sort
                                Sort Key: i.a
                                ->  Seq Scan on demo_inner i
Optimizer: GPORCA

Result with both optimizer settings:

a
-
2
3
(2 rows)

For a=3, there are no matching inner rows, and hypothetical rank(5) is 1, so 3 > 1 passes. For a=2, the matching group contains one NULL sort value and the rank is also 1. The optimization retains evaluation of nonconstant direct arguments and the existing protections for expressions around aggregates.

Local subselect_gp regressions pass with both optimizer settings, with parallel execution disabled and enabled. The examples above were also checked in a separate focused run. Expected plan files were refreshed, including the affected PAX expectations; PAX storage itself was not run locally.

All three include Backpatch-through: REL_2_STABLE.

[Alena0704]

I'm investigating why regression tests failed again and I'm going to move code with Orca fix in separate pr. I'll let you know when this pr will be ready for review

[Alena0704]

Moved bug fix for orca in #2127. It is ready for review