Skip to content

test(compliance): add initial layer 2 query compliance test framework - #632

Open
christianeu-db wants to merge 10 commits into
apache:mainfrom
christianeu-db:relational-compliance-a1
Open

christianeu-db wants to merge 10 commits into
apache:mainfrom
christianeu-db:relational-compliance-a1

Conversation

@christianeu-db

@christianeu-db christianeu-db commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The goal of this PR is to merge one small example and use it to bootstrap the query compliance tests, starting with the Relational Query Interface (#354).

The test cases in this PR are deliberately few. The draft #628 has 29 cases for discussion. This PR takes only one of them, A1 (grand total), so that the review can focus on the structure. Further cases can be added in later pull requests, each with one case or a small group of cases.

What this PR adds

Path Contents
compliance/query/README.md The case format, the semantics of the expected results, the rules to compare results, and the steps to add a case.
compliance/query/scott/scott.sql The SCOTT schema and its data
compliance/query/scott/scott_model.ossie.yaml An Ossie model over SCOTT with dataset-scoped metrics (#591). It is a real model file, and it passes validation/validate.py.
compliance/query/cases/a01_grand_total/ Case A1: query.ossie.sql, expected.sql, and reference.duckdb.sql.
compliance/README.md Two index lines for portability/ and query/.

Each case is a directory with three files. The part of each SQL file name before .sql gives its dialect; a name without one is standard SQL:

  • query.ossie.sql: the Ossie query, with AGG(). The header (ID, behavior under test, spec sections) is in SQL comments, so a harness can submit the file as it is.
  • expected.sql: the expected result, as one standard SQL query over literals (SELECT * FROM (VALUES ...) AS expected (...)).
  • reference.duckdb.sql: a DuckDB query over the base tables. It is non-normative. It helps to derive expected.sql, and a command in the README checks that their results are equal.

An implementation passes a case if it accepts query.ossie.sql against the SCOTT model and returns a result that is equal to the result of expected.sql. One case goes in each directory, so two contributors who add different cases do not change the same file.

Case A1:

-- query.ossie.sql
SELECT AGG(sum_sal) AS sum_sal,
       AGG(emp_count) AS emp_count
FROM scott_model.emp;

-- expected.sql
SELECT *
FROM (VALUES (29025.00, 14)) AS expected (sum_sal, emp_count);

Changes from #628

@justint-db offered to make the "host dialect" wording in the spec clearer. This PR does not change the spec. The README states the rule for the tests only.

Out of scope

Verification

  • With the DuckDB 1.5.6 CLI (not yet in the repo), each command in the README runs as written. For A1, reference.duckdb.sql and expected.sql both give 29025.00 | 14.
  • uv run validation/validate.py compliance/query/scott/scott_model.ossie.yaml passes on current main.

Related Issues

Related: #354, #591, #628, #501, #502.

Checklist

Specification

  • Spec changes are included in core-spec/ and follow the existing structure (N/A: no spec change)
  • Spec changes have been discussed on the mailing list or in a linked issue (N/A)
  • Breaking changes to the spec are clearly called out in the summary (N/A: additive, new files only)

Ontology

  • Ontology changes in ontology/ are consistent with spec changes (N/A)
  • New or modified terms are defined and documented (N/A)

Converters

Validation

  • Validation rules in validation/ are updated if the spec changed (N/A)
  • New validation cases are covered by tests (N/A; the new model passes validate.py)

Documentation

  • docs/ is updated to reflect any user-facing changes (N/A)
  • New features or behaviors are documented with examples where appropriate (compliance/query/README.md)
  • CONTRIBUTING.md is updated if the contribution process changed (N/A)

Examples

  • examples/ are added or updated for any new spec constructs (N/A)

Tests

  • All existing tests pass (pytest / CI green) (no code change; waiting for CI)
  • New functionality is covered by tests (this PR adds tests)

Compliance

  • ASF license headers are present on all new source files
  • No third-party dependencies are added without PMC/IPMC approval (DuckDB is necessary only to run the cases; it is not a project dependency)
  • If third-party source code is vendored/copied (not just declared as a dependency), NOTICE and/or LICENSE have been updated per ASF policy (N/A: no code is copied; the SCOTT data values are the same as in the Apache Calcite tests)

AI disclosure

Per the ASF Generative Tooling Guidance, this contribution was prepared with AI assistance. I have reviewed and verified every change.

Add compliance/query/ with the shared SCOTT data, an Ossie model over it,
and one case (A1, grand total) for the Relational Query Interface. The
README defines the case format, the semantics of the expected results
(standard SQL), the rules to compare results, and the steps to add a case.

Co-authored-by: Isaac <[email protected]>
@christianeu-db christianeu-db changed the title test(compliance): add query compliance test layout with case A1 test(compliance): add initial layer 2 query compliance test framework Oct 23, 2026
@christianeu-db
christianeu-db marked this pull request as ready for review October 23, 2026 18:17
@justint-db

Copy link
Copy Markdown
Contributor

I think it would be useful to split the Ossie query and the expected results out into their own files. The implementations could check that end-to-end they can accept an Ossie query and output the expected results.

christianeu-db and others added 8 commits October 23, 2026 20:34
…erence files

Each case is now a directory with query.sql (the Ossie query), expected.csv
(the expected result), and reference.sql (the DuckDB query that produces
expected.csv). An implementation can submit query.sql and compare its result
to expected.csv, end to end.

Co-authored-by: Isaac <[email protected]>
Rename query.sql to query.ossie.sql and reference.sql to
reference.duckdb.sql, so that the file name shows the dialect.

Co-authored-by: Isaac <[email protected]>
…olerance

Make emp.sal and emp.comm DOUBLE PRECISION. For AVG over an exact numeric,
the SQL standard leaves the scale of the result implementation-defined, so it
cannot be compared precisely. Replace the absolute 0.01 tolerance with a
relative tolerance of 1e-9 for approximate numerics; exact numerics must be
equal.

Co-authored-by: Isaac <[email protected]>
Replace expected.csv with expected.sql, a standard SQL query over literals.
Define the comparison: columns by position, bag semantics, NULL-aware
equality, and row order only for a unique ORDER BY. All expected values are
exact, with no tolerance, so the money columns are DECIMAL(7, 2) again and
the model and case A1 have no AVG.

Co-authored-by: Isaac <[email protected]>
…rics

Allow approximate numeric results, compared with a relative tolerance of
1e-9 and an absolute tolerance of 1e-9 near zero, and state that this rule
is provisional. Replace the EXCEPT ALL check with a command that shows the
reference result and the expected result.

Co-authored-by: Isaac <[email protected]>
State why the tolerance does not cover AVG over an exact numeric, and that a
case can use AVG over a cast to DOUBLE PRECISION.

Co-authored-by: Isaac <[email protected]>
@christianeu-db

Copy link
Copy Markdown
Contributor Author

I think it would be useful to split the Ossie query and the expected results out into their own files. The implementations could check that end-to-end they can accept an Ossie query and output the expected results.

Thanks for the feedback! Broke each case into 3 parts: ossie.sql, duck.sql and expected (in .sql)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants