Repository navigation
test(compliance): add initial layer 2 query compliance test framework - #632
Open
christianeu-db wants to merge 10 commits into
Open
christianeu-db wants to merge 10 commits into
christianeu-db wants to merge 10 commits into
Conversation
Add compliance/query/ with the shared SCOTT data, an Ossie model over it, and one case (A1, grand total) for the Relational Query Interface. The README defines the case format, the semantics of the expected results (standard SQL), the rules to compare results, and the steps to add a case. Co-authored-by: Isaac <[email protected]>
christianeu-db
marked this pull request as ready for review
October 23, 2026 18:17
Co-authored-by: Isaac <[email protected]>
Contributor
|
I think it would be useful to split the Ossie query and the expected results out into their own files. The implementations could check that end-to-end they can accept an Ossie query and output the expected results. |
…erence files Each case is now a directory with query.sql (the Ossie query), expected.csv (the expected result), and reference.sql (the DuckDB query that produces expected.csv). An implementation can submit query.sql and compare its result to expected.csv, end to end. Co-authored-by: Isaac <[email protected]>
Rename query.sql to query.ossie.sql and reference.sql to reference.duckdb.sql, so that the file name shows the dialect. Co-authored-by: Isaac <[email protected]>
…olerance Make emp.sal and emp.comm DOUBLE PRECISION. For AVG over an exact numeric, the SQL standard leaves the scale of the result implementation-defined, so it cannot be compared precisely. Replace the absolute 0.01 tolerance with a relative tolerance of 1e-9 for approximate numerics; exact numerics must be equal. Co-authored-by: Isaac <[email protected]>
Replace expected.csv with expected.sql, a standard SQL query over literals. Define the comparison: columns by position, bag semantics, NULL-aware equality, and row order only for a unique ORDER BY. All expected values are exact, with no tolerance, so the money columns are DECIMAL(7, 2) again and the model and case A1 have no AVG. Co-authored-by: Isaac <[email protected]>
Co-authored-by: Isaac <[email protected]>
…rics Allow approximate numeric results, compared with a relative tolerance of 1e-9 and an absolute tolerance of 1e-9 near zero, and state that this rule is provisional. Replace the EXCEPT ALL check with a command that shows the reference result and the expected result. Co-authored-by: Isaac <[email protected]>
State why the tolerance does not cover AVG over an exact numeric, and that a case can use AVG over a cast to DOUBLE PRECISION. Co-authored-by: Isaac <[email protected]>
Co-authored-by: Isaac <[email protected]>
Contributor
Author
Thanks for the feedback! Broke each case into 3 parts: ossie.sql, duck.sql and expected (in .sql) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The goal of this PR is to merge one small example and use it to bootstrap the query compliance tests, starting with the Relational Query Interface (#354).
The test cases in this PR are deliberately few. The draft #628 has 29 cases for discussion. This PR takes only one of them, A1 (grand total), so that the review can focus on the structure. Further cases can be added in later pull requests, each with one case or a small group of cases.
What this PR adds
compliance/query/README.mdcompliance/query/scott/scott.sqlcompliance/query/scott/scott_model.ossie.yamlvalidation/validate.py.compliance/query/cases/a01_grand_total/query.ossie.sql,expected.sql, andreference.duckdb.sql.compliance/README.mdportability/andquery/.Each case is a directory with three files. The part of each SQL file name before
.sqlgives its dialect; a name without one is standard SQL:query.ossie.sql: the Ossie query, withAGG(). The header (ID, behavior under test, spec sections) is in SQL comments, so a harness can submit the file as it is.expected.sql: the expected result, as one standard SQL query over literals (SELECT * FROM (VALUES ...) AS expected (...)).reference.duckdb.sql: a DuckDB query over the base tables. It is non-normative. It helps to deriveexpected.sql, and a command in the README checks that their results are equal.An implementation passes a case if it accepts
query.ossie.sqlagainst the SCOTT model and returns a result that is equal to the result ofexpected.sql. One case goes in each directory, so two contributors who add different cases do not change the same file.Case A1:
Changes from #628
validate.pycan check it.bonustable is removed. No case uses it.@justint-db offered to make the "host dialect" wording in the spec clearer. This PR does not change the spec. The README states the rule for the tests only.
Out of scope
.ost, a file format for idempotent test scripts #501) and a view expander (View expander #502). When those are available, each case can become an.osttest.Verification
reference.duckdb.sqlandexpected.sqlboth give29025.00 | 14.uv run validation/validate.py compliance/query/scott/scott_model.ossie.yamlpasses on currentmain.Related Issues
Related: #354, #591, #628, #501, #502.
Checklist
Specification
core-spec/and follow the existing structure (N/A: no spec change)Ontology
ontology/are consistent with spec changes (N/A)Converters
Validation
validation/are updated if the spec changed (N/A)validate.py)Documentation
docs/is updated to reflect any user-facing changes (N/A)compliance/query/README.md)CONTRIBUTING.mdis updated if the contribution process changed (N/A)Examples
examples/are added or updated for any new spec constructs (N/A)Tests
pytest/ CI green) (no code change; waiting for CI)Compliance
NOTICEand/orLICENSEhave been updated per ASF policy (N/A: no code is copied; the SCOTT data values are the same as in the Apache Calcite tests)AI disclosure
Per the ASF Generative Tooling Guidance, this contribution was prepared with AI assistance. I have reviewed and verified every change.