From 764b77a6a9d390367f06adec33177bc2b4d73d45 Mon Sep 17 00:00:00 2001 From: "Mr. Black 1995" Date: Fri, 16 Oct 2026 11:01:50 +0200 Subject: [PATCH 1/3] test(compliance): add portability measurement fixtures M0-M3 The fixture half of #576 (measurements for model portability across converters): a small portable ladder where each rung adds one capability on top of the one below, so a converter round-trip difference localizes to a single construct. M0 baseline (one dataset, one field, one SUM metric, three variants), M1 a scalar calculated-column field, M2 the core aggregate set, M3 a filtered aggregate in the portable SUM(CASE WHEN ...) form. Every fixture is authored in a portable dialect (ANSI_SQL / OSSIE_SQL_2026) so it runs through every converter, and the qualified dataset.field form is carried as primary with the bare form as a sibling variant per #486 and #343. manifest.yaml lists the fixtures, the element each rung exercises, and the per-element outcome vocabulary a grader assigns. Scoring is left to the interoperability harness in #348; conformance stays a floor (all fixtures pass validation/validate.py on current main). Refs #576 Co-authored-by: Isaac --- compliance/portability/README.md | 65 +++++++++++++ .../m0/m0_ansi_qualified.ossie.yaml | 41 ++++++++ .../m0/m0_ossiesql_qualified.ossie.yaml | 44 +++++++++ .../portability/m0/m0_sum_bare.ossie.yaml | 46 +++++++++ .../portability/m1/m1_scalar_field.ossie.yaml | 49 ++++++++++ .../m2/m2_aggregates_bare.ossie.yaml | 81 ++++++++++++++++ .../m2/m2_aggregates_qualified.ossie.yaml | 82 ++++++++++++++++ .../m3/m3_filtered_agg_bare.ossie.yaml | 46 +++++++++ .../m3/m3_filtered_agg_qualified.ossie.yaml | 46 +++++++++ compliance/portability/manifest.yaml | 95 +++++++++++++++++++ 10 files changed, 595 insertions(+) create mode 100644 compliance/portability/README.md create mode 100644 compliance/portability/m0/m0_ansi_qualified.ossie.yaml create mode 100644 compliance/portability/m0/m0_ossiesql_qualified.ossie.yaml create mode 100644 compliance/portability/m0/m0_sum_bare.ossie.yaml create mode 100644 compliance/portability/m1/m1_scalar_field.ossie.yaml create mode 100644 compliance/portability/m2/m2_aggregates_bare.ossie.yaml create mode 100644 compliance/portability/m2/m2_aggregates_qualified.ossie.yaml create mode 100644 compliance/portability/m3/m3_filtered_agg_bare.ossie.yaml create mode 100644 compliance/portability/m3/m3_filtered_agg_qualified.ossie.yaml create mode 100644 compliance/portability/manifest.yaml diff --git a/compliance/portability/README.md b/compliance/portability/README.md new file mode 100644 index 000000000..5315cdc11 --- /dev/null +++ b/compliance/portability/README.md @@ -0,0 +1,65 @@ + + +# Portability measurement ladder (M0-M3) + +A small set of portable fixtures for measuring how faithfully a model survives a +round-trip through each converter. This is the fixture half of issue #576 +(measurements for model portability across converters). The scoring half is a +grader that diffs each returned document against the input element by element; +that lives with the interoperability harness in #348, not here. + +## What is here + +Each rung adds exactly one capability on top of the one below it, so a +difference in a converter's output localizes to a single construct: + +| Rung | Capability added | Construct | +|------|------------------|-----------| +| M0 | baseline | one dataset, one field, one `SUM` metric | +| M1 | a scalar field, no aggregation | `amount * 1.1` calculated column | +| M2 | the core aggregate set | `SUM`, `COUNT`, `COUNT(DISTINCT)`, `AVG`, `MIN`, `MAX` | +| M3 | a filtered aggregate | `SUM(CASE WHEN ... THEN x ELSE 0 END)` | + +`manifest.yaml` lists the fixtures, the element each rung exercises, and the +per-element outcome vocabulary a grader assigns (PORTABLE, SURVIVED-vendor-only, +ADDED, RELABELED, DROPPED-diagnosed, DROPPED-silent, REJECTED). + +## Design rules + +- **Portable dialect only.** Every fixture is authored in `ANSI_SQL` or + `OSSIE_SQL_2026`. A vendor-tagged input would only exercise one converter, + which defeats a cross-converter measurement. +- **Qualified reference form is primary.** Where a metric references a dataset + field, the qualified `dataset.field` form is the primary fixture and the bare + form is a sibling variant, pending the name-resolution decisions in #486 and + #343. +- **Conformance is a floor, not the measurement.** Every fixture passes + `validation/validate.py` on current `main`. Fidelity is what the grader + measures on top of that. + +## Scope and placement + +Rungs M4-M8 (relationships and cross-dataset metrics, metric-to-metric +composition, date and time scalars, window and offset functions) are held +pending #354, #591 and #587, and are not in this set. + +Location is a starting point. If these should sit with the interoperability +harness from #348 or elsewhere once that lands, they can move; the fixtures are +self-contained and the manifest paths are relative. diff --git a/compliance/portability/m0/m0_ansi_qualified.ossie.yaml b/compliance/portability/m0/m0_ansi_qualified.ossie.yaml new file mode 100644 index 000000000..64b06bb36 --- /dev/null +++ b/compliance/portability/m0/m0_ansi_qualified.ossie.yaml @@ -0,0 +1,41 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M0 ladder rung -- one dataset, one field, one SUM metric. +# Variant: ANSI_SQL tag, QUALIFIED reference form (dataset.field). PRIMARY fixture. +# The qualified form is carried as primary at every rung per the open #486 +# (qualified dataset.field only) decision; the bare form is a sibling variant +# (see m0_sum_bare.ossie.yaml) pending #343. +version: "0.2.0.dev0" +name: m0_sales +description: M0 baseline -- single dataset, single field, single SUM metric. +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal +metrics: + - name: total_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(sales.amount) diff --git a/compliance/portability/m0/m0_ossiesql_qualified.ossie.yaml b/compliance/portability/m0/m0_ossiesql_qualified.ossie.yaml new file mode 100644 index 000000000..ea812222d --- /dev/null +++ b/compliance/portability/m0/m0_ossiesql_qualified.ossie.yaml @@ -0,0 +1,44 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# yaml-language-server: $schema=../../../../core-spec/ossie-schema.json +# +# M0 ladder rung -- one dataset, one field, one SUM metric. +# Variant: OSSIE_SQL_2026 tag, QUALIFIED reference form (dataset.field). +# This is the variant that RELABELED on kayemkim's M0 run (dbt, wisdom, +# honeydew, orionbelt returned it tagged ANSI_SQL with the text unchanged; +# only cube kept the OSSIE_SQL_2026 tag). "tag preserved" is tracked as a +# sub-metric under PORTABLE so that 1-of-N signal survives. +version: "0.2.0.dev0" +name: m0_sales +description: M0 baseline -- single dataset, single field, single SUM metric. +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: OSSIE_SQL_2026 + expression: amount + datatype: Decimal +metrics: + - name: total_amount + expression: + dialects: + - dialect: OSSIE_SQL_2026 + expression: SUM(sales.amount) diff --git a/compliance/portability/m0/m0_sum_bare.ossie.yaml b/compliance/portability/m0/m0_sum_bare.ossie.yaml new file mode 100644 index 000000000..f51f99f1f --- /dev/null +++ b/compliance/portability/m0/m0_sum_bare.ossie.yaml @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# yaml-language-server: $schema=../../../../core-spec/ossie-schema.json +# +# M0 ladder rung -- one dataset, one field, one SUM metric. +# Variant: BARE reference form -- SUM(amount), a bare column name rather than +# dataset.field. ANSI_SQL tag. This is the "SUM(amount)" variant from the runs. +# Kept as a sibling variant to the qualified primary, not baked into the rung, +# because #343 (bare name means a source column inside a dataset-scoped metric) +# and #486 (qualified dataset.field only) are still settling which form is the +# model-level reference. sigma drops this as a cross-dataset metric on a +# one-dataset model. +version: "0.2.0.dev0" +name: m0_sales +description: M0 baseline -- single dataset, single field, single SUM metric. +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal +metrics: + - name: total_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(amount) diff --git a/compliance/portability/m1/m1_scalar_field.ossie.yaml b/compliance/portability/m1/m1_scalar_field.ossie.yaml new file mode 100644 index 000000000..fb75cd69a --- /dev/null +++ b/compliance/portability/m1/m1_scalar_field.ossie.yaml @@ -0,0 +1,49 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M1 ladder rung: baseline plus one scalar field expression (calculated +# column), no aggregation. Element under test: the amount_with_tax field. +# Reference-form note: a field expression resolves inside its own dataset, so +# it references the sibling column bare (amount). The qualified/bare axis that +# #486 and #343 settle applies to metric references, not to intra-dataset field +# expressions, so this rung has a single fixture. +version: "0.2.0.dev0" +name: m1_sales +description: "M1: single dataset, one scalar calculated-column field expression." +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal + - name: amount_with_tax + expression: + dialects: + - dialect: ANSI_SQL + expression: amount * 1.1 + datatype: Decimal +metrics: + - name: total_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(sales.amount) + datatype: Decimal diff --git a/compliance/portability/m2/m2_aggregates_bare.ossie.yaml b/compliance/portability/m2/m2_aggregates_bare.ossie.yaml new file mode 100644 index 000000000..bf5ca36c3 --- /dev/null +++ b/compliance/portability/m2/m2_aggregates_bare.ossie.yaml @@ -0,0 +1,81 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M2 ladder rung: the core aggregate set as metrics. Variant: BARE reference +# form (column name only), kept as a sibling variant pending #343. +version: "0.2.0.dev0" +name: m2_sales +description: "M2: core aggregate set, bare reference form." +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal + - name: order_id + expression: + dialects: + - dialect: ANSI_SQL + expression: order_id + datatype: Integer + - name: customer_id + expression: + dialects: + - dialect: ANSI_SQL + expression: customer_id + datatype: Integer +metrics: + - name: sum_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(amount) + datatype: Decimal + - name: count_orders + expression: + dialects: + - dialect: ANSI_SQL + expression: COUNT(order_id) + datatype: Integer + - name: distinct_customers + expression: + dialects: + - dialect: ANSI_SQL + expression: COUNT(DISTINCT customer_id) + datatype: Integer + - name: avg_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: AVG(amount) + datatype: Decimal + - name: min_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: MIN(amount) + datatype: Decimal + - name: max_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: MAX(amount) + datatype: Decimal diff --git a/compliance/portability/m2/m2_aggregates_qualified.ossie.yaml b/compliance/portability/m2/m2_aggregates_qualified.ossie.yaml new file mode 100644 index 000000000..8ab5b6252 --- /dev/null +++ b/compliance/portability/m2/m2_aggregates_qualified.ossie.yaml @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M2 ladder rung: the core aggregate set as metrics (SUM, COUNT, +# COUNT DISTINCT, AVG, MIN, MAX). Variant: QUALIFIED reference form +# (dataset.field), carried as primary per #486. +version: "0.2.0.dev0" +name: m2_sales +description: "M2: core aggregate set, qualified reference form." +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal + - name: order_id + expression: + dialects: + - dialect: ANSI_SQL + expression: order_id + datatype: Integer + - name: customer_id + expression: + dialects: + - dialect: ANSI_SQL + expression: customer_id + datatype: Integer +metrics: + - name: sum_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(sales.amount) + datatype: Decimal + - name: count_orders + expression: + dialects: + - dialect: ANSI_SQL + expression: COUNT(sales.order_id) + datatype: Integer + - name: distinct_customers + expression: + dialects: + - dialect: ANSI_SQL + expression: COUNT(DISTINCT sales.customer_id) + datatype: Integer + - name: avg_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: AVG(sales.amount) + datatype: Decimal + - name: min_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: MIN(sales.amount) + datatype: Decimal + - name: max_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: MAX(sales.amount) + datatype: Decimal diff --git a/compliance/portability/m3/m3_filtered_agg_bare.ossie.yaml b/compliance/portability/m3/m3_filtered_agg_bare.ossie.yaml new file mode 100644 index 000000000..244483381 --- /dev/null +++ b/compliance/portability/m3/m3_filtered_agg_bare.ossie.yaml @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M3 ladder rung: a filtered aggregate in the spec's portable form +# SUM(CASE WHEN ... THEN x ELSE 0 END). Variant: BARE reference form, kept as a +# sibling variant pending #343. +version: "0.2.0.dev0" +name: m3_sales +description: "M3: conditional aggregation, bare reference form." +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal + - name: status + expression: + dialects: + - dialect: ANSI_SQL + expression: status + datatype: String +metrics: + - name: completed_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(CASE WHEN status = 'completed' THEN amount ELSE 0 END) + datatype: Decimal diff --git a/compliance/portability/m3/m3_filtered_agg_qualified.ossie.yaml b/compliance/portability/m3/m3_filtered_agg_qualified.ossie.yaml new file mode 100644 index 000000000..0edaa573b --- /dev/null +++ b/compliance/portability/m3/m3_filtered_agg_qualified.ossie.yaml @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M3 ladder rung: a filtered aggregate in the spec's portable form +# SUM(CASE WHEN ... THEN x ELSE 0 END). Variant: QUALIFIED reference form, +# carried as primary per #486. +version: "0.2.0.dev0" +name: m3_sales +description: "M3: conditional aggregation, qualified reference form." +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal + - name: status + expression: + dialects: + - dialect: ANSI_SQL + expression: status + datatype: String +metrics: + - name: completed_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(CASE WHEN sales.status = 'completed' THEN sales.amount ELSE 0 END) + datatype: Decimal diff --git a/compliance/portability/manifest.yaml b/compliance/portability/manifest.yaml new file mode 100644 index 000000000..53fece897 --- /dev/null +++ b/compliance/portability/manifest.yaml @@ -0,0 +1,95 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Portability measurement ladder, rungs M0-M3 (strawman for #576). +# +# One capability is added per rung so that a difference in a converter's +# round-trip localizes to a single construct. Every rung is authored in a +# PORTABLE dialect (ANSI_SQL / OSSIE_SQL_2026) so it is vendor-neutral and runs +# through every converter; a vendor-tagged input would only exercise one +# converter, which is not what a cross-converter measurement wants. +# +# This manifest lists the fixtures and the single element each rung exercises. +# It does NOT encode per-converter expected outputs: those are measured by a +# grader that diffs the returned document against the input element by element +# into the outcome vocabulary below. Conformance stays a floor (validate.py +# passing); scoring stays a harness job (see #348). +# +# Reference form: the qualified dataset.field form is carried as the primary +# fixture wherever a metric references a dataset field, with the bare form as a +# sibling variant, pending #486 and #343. M1's element is an intra-dataset field +# expression, so it carries a single fixture. +# +# All fixtures validate.py-pass on apache/ossie main (9531429) with sqlglot +# 29.0.1. Rungs M4-M8 (relationships, metric composition, date scalars, window +# and offset functions) are held pending #354, #591 and #587; they are not in +# this set. + +outcomes: + # per-element, assigned by the grader after diffing input vs returned output + - PORTABLE # returns tagged ANSI_SQL or OSSIE_SQL_2026 + - TAG_PRESERVED # sub-flag under PORTABLE: the OSSIE_SQL_2026 tag survived + - SURVIVED_VENDOR_ONLY # returns in a vendor dialect only + - ADDED # an element appears that was not in the input + - RELABELED # same text, different tag than authored + - DROPPED_DIAGNOSED # removed with a warning or notice + - DROPPED_SILENT # removed with no diagnostic + - REJECTED # the converter refuses the whole document + +rungs: + - id: M0 + description: one dataset, one field, one SUM metric + element_under_test: + kind: metric + name: total_amount + construct: single-column aggregate (SUM) + reference_form_primary: qualified + variants: + - {id: M0.ansi.qualified, file: m0/m0_ansi_qualified.ossie.yaml, tag: ANSI_SQL, reference_form: qualified, role: primary} + - {id: M0.ossiesql.qualified, file: m0/m0_ossiesql_qualified.ossie.yaml, tag: OSSIE_SQL_2026, reference_form: qualified, role: tag-sensitivity} + - {id: M0.sum.bare, file: m0/m0_sum_bare.ossie.yaml, tag: ANSI_SQL, reference_form: bare, role: reference-form-variant} + + - id: M1 + description: baseline plus one scalar calculated-column field expression, no aggregation + element_under_test: + kind: field + name: amount_with_tax + construct: scalar field expression (amount * 1.1) + reference_form_primary: n/a # intra-dataset field expression references its sibling column bare + variants: + - {id: M1.scalar_field, file: m1/m1_scalar_field.ossie.yaml, tag: ANSI_SQL, reference_form: intra-dataset, role: primary} + + - id: M2 + description: the core aggregate set (SUM, COUNT, COUNT DISTINCT, AVG, MIN, MAX) + element_under_test: + kind: metrics + construct: core aggregate set + reference_form_primary: qualified + variants: + - {id: M2.aggregates.qualified, file: m2/m2_aggregates_qualified.ossie.yaml, tag: ANSI_SQL, reference_form: qualified, role: primary} + - {id: M2.aggregates.bare, file: m2/m2_aggregates_bare.ossie.yaml, tag: ANSI_SQL, reference_form: bare, role: reference-form-variant} + + - id: M3 + description: a filtered aggregate in the portable form SUM(CASE WHEN ... THEN x ELSE 0 END) + element_under_test: + kind: metric + name: completed_amount + construct: conditional aggregation + reference_form_primary: qualified + variants: + - {id: M3.filtered_agg.qualified, file: m3/m3_filtered_agg_qualified.ossie.yaml, tag: ANSI_SQL, reference_form: qualified, role: primary} + - {id: M3.filtered_agg.bare, file: m3/m3_filtered_agg_bare.ossie.yaml, tag: ANSI_SQL, reference_form: bare, role: reference-form-variant} From ea88c1ecc34620a2c0f7462a3a8e7fa89c6a8626 Mon Sep 17 00:00:00 2001 From: "Mr. Black 1995" Date: Mon, 19 Oct 2026 09:19:38 +0200 Subject: [PATCH 2/3] test(compliance): address #596 review (datatype consistency, grader, new outcomes) From @kayemkim's nine-converter run on #596: - Give the three M0 metrics datatype: Decimal so the metric datatype is constant from M0 and each rung adds exactly one construct (M1 was changing two things and pulling in thoughtspot's datatype warning for an unrelated reason). - Add REWRITTEN (portable tag kept, expression changed) and PLACEHOLDER (a valid stand-in such as BLANK() replaces the content, with a diagnostic) to the outcome vocabulary, the two cases the real run exposed that the eight did not name. - Add the grader the measurement needs under grader/: stdlib diff of input vs returned document into the manifest outcomes, with sqlglot (already a converter and validate.py dependency) used when available to tell a cosmetic REWRITTEN (requalify, unquote, reorder) from one that needs review (e.g. the dbt SUM(SUM(CASE ...)) that #437 fixes). Pairs a renamed element by expression so the orionbelt field-renamed-to-its-expression case scores as one changed element, not a drop plus an add. REJECTED is recorded at document scope (thoughtspot). 13 tests, one per outcome plus an identity check grading the real fixtures all-PORTABLE. Refs #576 Co-authored-by: Isaac --- compliance/portability/README.md | 19 +- compliance/portability/grader/.gitignore | 2 + compliance/portability/grader/README.md | 66 +++ compliance/portability/grader/grade.py | 388 ++++++++++++++++++ compliance/portability/grader/test_grade.py | 159 +++++++ .../m0/m0_ansi_qualified.ossie.yaml | 1 + .../m0/m0_ossiesql_qualified.ossie.yaml | 1 + .../portability/m0/m0_sum_bare.ossie.yaml | 1 + compliance/portability/manifest.yaml | 14 +- 9 files changed, 642 insertions(+), 9 deletions(-) create mode 100644 compliance/portability/grader/.gitignore create mode 100644 compliance/portability/grader/README.md create mode 100644 compliance/portability/grader/grade.py create mode 100644 compliance/portability/grader/test_grade.py diff --git a/compliance/portability/README.md b/compliance/portability/README.md index 5315cdc11..f090f271a 100644 --- a/compliance/portability/README.md +++ b/compliance/portability/README.md @@ -19,11 +19,13 @@ # Portability measurement ladder (M0-M3) -A small set of portable fixtures for measuring how faithfully a model survives a -round-trip through each converter. This is the fixture half of issue #576 -(measurements for model portability across converters). The scoring half is a -grader that diffs each returned document against the input element by element; -that lives with the interoperability harness in #348, not here. +A small set of portable fixtures, and a grader, for measuring how faithfully a +model survives a round-trip through each converter. This is issue #576 +(measurements for model portability across converters). The fixtures are the +inputs; `grader/` scores each returned document against its input, element by +element, into the outcome vocabulary in `manifest.yaml`. A thin runner that drives +the converters (modelled on `converters/cube/tools/interop_matrix.py`) turns the +whole thing into one command; the grader is written to sit behind it. ## What is here @@ -38,8 +40,11 @@ difference in a converter's output localizes to a single construct: | M3 | a filtered aggregate | `SUM(CASE WHEN ... THEN x ELSE 0 END)` | `manifest.yaml` lists the fixtures, the element each rung exercises, and the -per-element outcome vocabulary a grader assigns (PORTABLE, SURVIVED-vendor-only, -ADDED, RELABELED, DROPPED-diagnosed, DROPPED-silent, REJECTED). +per-element outcome vocabulary the grader assigns (PORTABLE / TAG_PRESERVED, +REWRITTEN, SURVIVED-vendor-only, PLACEHOLDER, ADDED, RELABELED, +DROPPED-diagnosed, DROPPED-silent, REJECTED). `grader/grade.py` implements it +and `grader/test_grade.py` has one test per outcome plus an identity check that +grades the real fixtures all-PORTABLE. See `grader/README.md`. ## Design rules diff --git a/compliance/portability/grader/.gitignore b/compliance/portability/grader/.gitignore new file mode 100644 index 000000000..7a60b85e1 --- /dev/null +++ b/compliance/portability/grader/.gitignore @@ -0,0 +1,2 @@ +__pycache__/ +*.pyc diff --git a/compliance/portability/grader/README.md b/compliance/portability/grader/README.md new file mode 100644 index 000000000..b0f97fab8 --- /dev/null +++ b/compliance/portability/grader/README.md @@ -0,0 +1,66 @@ + + +# Portability fidelity grader + +`grade.py` is the scoring half of issue #576. Conformance (`validate.py` passing) +is a floor and says nothing about whether a model survived a converter round-trip; +this grader is the fidelity half. + +It does not drive the converters. It takes the input document and the document a +converter returned, and diffs them element by element into the outcome vocabulary +in `../manifest.yaml`. A thin runner (modelled on +`converters/cube/tools/interop_matrix.py`) drives each converter and feeds the +grader, so the measurement table becomes one command. + +## Usage + +``` +python grade.py --input model.ossie.yaml --returned returned.ossie.yaml \ + --converter dbt [--diagnostics warnings.txt] [--json] +``` + +## Outcomes + +PORTABLE (with a TAG_PRESERVED sub-flag), REWRITTEN, SURVIVED_VENDOR_ONLY, +PLACEHOLDER, ADDED, RELABELED, DROPPED_DIAGNOSED, DROPPED_SILENT, REJECTED. + +## Design + +- **Standard library for the diff.** `sqlglot` (already used by the converters + and `validate.py`) is used when available to compare normalized expression + trees, so a REWRITTEN that is only a requalify, an unquote or a reorder is + marked cosmetic, while one it cannot prove equivalent is flagged for review. + Without `sqlglot` the grader still runs and marks every rewrite as needing + review. +- **Textual by default, semantic where it can be.** PORTABLE means the converter + returned the same expression it was given (modulo cosmetic whitespace). True + semantic equivalence of SQL is undecidable in general, so the grader auto-clears + only the changes a normalized tree proves equal and leaves the rest for a human. +- **Identity beyond name.** A dropped input element and an added output element + that carry an equivalent expression are paired as one element whose identity + (name) was not preserved, rather than scored as a drop plus an unrelated add. +- **Diagnosed vs silent, and PLACEHOLDER, need the converter's warnings.** Pass + them with `--diagnostics`. + +## Tests + +`test_grade.py` has one test per outcome plus an identity check that runs the real +M0-M3 fixtures through a no-op converter and asserts every element grades +PORTABLE. Run with `PYTHONPATH=. pytest test_grade.py`. diff --git a/compliance/portability/grader/grade.py b/compliance/portability/grader/grade.py new file mode 100644 index 000000000..b8944dbac --- /dev/null +++ b/compliance/portability/grader/grade.py @@ -0,0 +1,388 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Fidelity grader for the portability measurement ladder (issue #576). + +The grader answers one question per element: when a model is driven through a +converter and back, what happened to each metric and field expression it started +with? Conformance (``validate.py`` passing) is a floor and says nothing about +fidelity; this grader is the fidelity half. + +It does NOT drive the converters: that is the thin runner's job (modelled on +``converters/cube/tools/interop_matrix.py``). It takes the input document and the +document a converter returned, and diffs them element by element into the outcome +vocabulary in ``../manifest.yaml``: + +* PORTABLE returned under a portable tag, expression unchanged +* TAG_PRESERVED sub-flag of PORTABLE: an OSSIE_SQL_2026 input kept its tag +* REWRITTEN same portable tag, expression text changed. A change the + normalized comparison proves is only spelling (requalify, + unquote, reorder) is marked benign; anything it cannot + prove equivalent is flagged for review +* SURVIVED_VENDOR_ONLY returned in a vendor dialect only, as a real translation +* PLACEHOLDER replaced by a valid stand-in that is not the content + (e.g. ``BLANK()``), with a diagnostic: a diagnosed loss +* RELABELED same text, a different tag than authored +* DROPPED_DIAGNOSED gone, and a diagnostic names it +* DROPPED_SILENT gone, no diagnostic +* REJECTED the converter refused the whole document (document scope) +* ADDED an element in the output that was not in the input + +The diff is standard library. ``sqlglot`` (already used by the converters and by +``validate.py``) is used when available to compare normalized expression trees so +a cosmetic rewrite is told from a possible change in meaning; without it the +grader still runs and simply marks every rewrite as needing review. + +The diagnosed-versus-silent and PLACEHOLDER distinctions need the converter's own +warnings; pass them with ``--diagnostics``. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from dataclasses import dataclass, field + +PORTABLE_DIALECTS = frozenset({"ANSI_SQL", "OSSIE_SQL_2026"}) + +# Normalized expressions a converter emits as a stand-in when it cannot translate +# the authored one. Compared after _normalize + upper-casing. +PLACEHOLDER_TOKENS = frozenset({"BLANK()", "BLANK", "NULL"}) + +PORTABLE = "PORTABLE" +TAG_PRESERVED = "TAG_PRESERVED" +REWRITTEN = "REWRITTEN" +SURVIVED_VENDOR_ONLY = "SURVIVED_VENDOR_ONLY" +PLACEHOLDER = "PLACEHOLDER" +RELABELED = "RELABELED" +DROPPED_DIAGNOSED = "DROPPED_DIAGNOSED" +DROPPED_SILENT = "DROPPED_SILENT" +REJECTED = "REJECTED" +ADDED = "ADDED" + + +def _load(path): + """Load an Ossie document from a YAML or JSON file.""" + with open(path, "r", encoding="utf-8") as handle: + body = handle.read() + if path.endswith(".json"): + return json.loads(body) + try: + import yaml + except ImportError as exc: # pragma: no cover - environment dependent + raise SystemExit("PyYAML is needed to read a YAML document; pass JSON instead") from exc + return yaml.safe_load(body) + + +def _normalize(expression): + """Collapse insignificant whitespace, including around parentheses and commas, + so a reformatted expression still matches textually. Textual, not semantic.""" + if expression is None: + return "" + collapsed = re.sub(r"\s+", " ", str(expression)).strip() + return re.sub(r"\s*([(),])\s*", r"\1", collapsed) + + +def _equivalent_tree(left, right): + """Return True if two SQL expressions are equal after stripping table + qualifiers and quoting, False if they differ, or None if it cannot be decided + (sqlglot missing, or either side does not parse). Used only to tell a cosmetic + REWRITTEN from one that needs review.""" + try: + import sqlglot + from sqlglot import exp + except ImportError: # pragma: no cover - environment dependent + return None + + def canonical(text): + try: + tree = sqlglot.parse_one(text) + except Exception: + return None + for column in tree.find_all(exp.Column): + # Drop the table qualifier so sales.amount and amount compare equal. + column.set("table", None) + column.set("db", None) + column.set("catalog", None) + for identifier in tree.find_all(exp.Identifier): + # Unquote so "amount" and amount compare equal (sigma, orionbelt quote). + identifier.set("quoted", False) + # normalize_identifiers lower-cases identifiers consistently. + try: + from sqlglot.optimizer.normalize_identifiers import normalize_identifiers + + tree = normalize_identifiers(tree) + except Exception: + pass + return tree.sql(normalize=True) + + left_sql = canonical(left) + right_sql = canonical(right) + if left_sql is None or right_sql is None: + return None + return left_sql == right_sql + + +def _dialect_map(expression_obj): + result = {} + if not isinstance(expression_obj, dict): + return result + for entry in expression_obj.get("dialects") or []: + if isinstance(entry, dict) and entry.get("dialect") is not None: + result[entry["dialect"]] = _normalize(entry.get("expression")) + return result + + +@dataclass +class Element: + kind: str # "metric" or "field" + name: str + dialects: dict + + +def _collect(document): + """Collect metrics and field expressions, keyed by (kind, name).""" + elements = {} + if not isinstance(document, dict): + return elements + for metric in document.get("metrics") or []: + if isinstance(metric, dict) and metric.get("name"): + key = ("metric", metric["name"]) + elements[key] = Element("metric", metric["name"], _dialect_map(metric.get("expression"))) + for dataset in document.get("datasets") or []: + if not isinstance(dataset, dict): + continue + dataset_name = dataset.get("name") or "" + for field_def in dataset.get("fields") or []: + if isinstance(field_def, dict) and field_def.get("name"): + name = f"{dataset_name}.{field_def['name']}" + elements[("field", name)] = Element("field", name, _dialect_map(field_def.get("expression"))) + return elements + + +@dataclass +class Finding: + kind: str + name: str + outcome: str + tag_preserved: bool = False + detail: str = "" + + +@dataclass +class Report: + converter: str = "" + rejected: bool = False + findings: list = field(default_factory=list) + + def counts(self): + tally = {} + for finding in self.findings: + tally[finding.outcome] = tally.get(finding.outcome, 0) + 1 + return tally + + +def _authored_tag(dialects): + for dialect in dialects: + if dialect in PORTABLE_DIALECTS: + return dialect + return next(iter(dialects), None) + + +def _diagnosed(name, diagnostics): + return name in diagnostics or name.split(".")[-1] in diagnostics + + +def _classify_present(input_element, output_element): + """Classify an input element that still has a counterpart in the output.""" + in_dialects = input_element.dialects + out_dialects = output_element.dialects + authored_tag = _authored_tag(in_dialects) + authored_text = in_dialects.get(authored_tag, "") + + # PORTABLE: authored portable tag returned with the same text under that tag. + if authored_tag in PORTABLE_DIALECTS and out_dialects.get(authored_tag) == authored_text: + return Finding( + input_element.kind, + input_element.name, + PORTABLE, + tag_preserved=authored_tag == "OSSIE_SQL_2026", + detail=f"returned as {authored_tag}", + ) + + # RELABELED: the authored expression came back verbatim, under a different tag. + for dialect, text in out_dialects.items(): + if text == authored_text and dialect != authored_tag: + return Finding( + input_element.kind, input_element.name, RELABELED, + detail=f"authored {authored_tag}, returned {dialect}", + ) + + portable_out = {d: t for d, t in out_dialects.items() if d in PORTABLE_DIALECTS} + + # REWRITTEN: a portable tag is present but the expression text changed. + if portable_out: + returned_text = portable_out.get(authored_tag) or next(iter(portable_out.values())) + equivalent = _equivalent_tree(authored_text, returned_text) + if equivalent is True: + note = "cosmetic (normalized trees equal)" + elif equivalent is False: + note = "review: normalized trees differ" + else: + note = "review: could not compare trees" + return Finding( + input_element.kind, input_element.name, REWRITTEN, + detail=f"{note}; returned {returned_text!r}", + ) + + # No portable tag: either a placeholder stand-in or a real vendor-only return. + if out_dialects: + sole_text = next(iter(out_dialects.values())).upper() + if sole_text in PLACEHOLDER_TOKENS: + return Finding( + input_element.kind, input_element.name, PLACEHOLDER, + detail=f"stand-in {next(iter(out_dialects.values()))!r} replaced the content", + ) + return Finding( + input_element.kind, input_element.name, SURVIVED_VENDOR_ONLY, + detail=f"returned as {','.join(sorted(out_dialects))}", + ) + return Finding(input_element.kind, input_element.name, SURVIVED_VENDOR_ONLY, detail="no expression returned") + + +def _repair_by_expression(dropped, added): + """Pair a DROPPED input element with an ADDED output element that carries an + equivalent expression, so a converter that did not preserve an element's name + (orionbelt writes a field's expression as its name) scores as one element whose + identity was not preserved, not as a drop plus an unrelated addition. + + Returns a list of (input_element, output_element) pairs and mutates nothing. + """ + pairs = [] + used = set() + for in_el in dropped: + in_text = next(iter(in_el.dialects.values()), "") + for out_el in added: + if id(out_el) in used: + continue + out_text = next(iter(out_el.dialects.values()), "") + if in_text and (_normalize(out_el.name) == in_text or out_text == in_text + or _equivalent_tree(in_text, out_text) is True): + pairs.append((in_el, out_el)) + used.add(id(out_el)) + break + return pairs + + +def grade(input_document, output_document, diagnostics_text="", converter=""): + """Grade a returned document against its input, element by element.""" + report = Report(converter=converter) + input_elements = _collect(input_document) + output_elements = _collect(output_document) + + # A converter that returns nothing refused the whole document (document scope). + if not output_elements: + report.rejected = True + for element in input_elements.values(): + report.findings.append(Finding(element.kind, element.name, REJECTED)) + return report + + diagnostics = diagnostics_text or "" + + present = {} + dropped = [] + for key, element in input_elements.items(): + if key in output_elements: + present[key] = _classify_present(element, output_elements[key]) + else: + dropped.append(element) + + added_elements = [el for key, el in output_elements.items() if key not in input_elements] + + # Recover identity: an unnamed-preserving rename shows up as drop + add. + repaired = _repair_by_expression(dropped, added_elements) + repaired_in = {id(a): b for a, b in repaired} + repaired_out = {id(b) for _, b in repaired} + + report.findings.extend(present.values()) + + for element in dropped: + if id(element) in repaired_in: + report.findings.append(Finding( + element.kind, element.name, REWRITTEN, + detail="identity not preserved: returned under a different name, expression equivalent", + )) + continue + report.findings.append(Finding( + element.kind, element.name, + DROPPED_DIAGNOSED if _diagnosed(element.name, diagnostics) else DROPPED_SILENT, + )) + + for element in added_elements: + if id(element) in repaired_out: + continue # accounted for by its repaired partner + report.findings.append(Finding(element.kind, element.name, ADDED, detail="not in input")) + + return report + + +def _format(report): + lines = [f"converter: {report.converter}" if report.converter else "grade report"] + for finding in report.findings: + flag = " (tag preserved)" if finding.tag_preserved else "" + detail = f" -- {finding.detail}" if finding.detail else "" + lines.append(f" [{finding.outcome}{flag}] {finding.kind} {finding.name}{detail}") + tally = report.counts() + lines.append("summary: " + ", ".join(f"{name}={count}" for name, count in sorted(tally.items()))) + return "\n".join(lines) + + +def main(argv=None): + parser = argparse.ArgumentParser(description="Grade a converter round-trip for #576.") + parser.add_argument("--input", required=True, help="the input .ossie.yaml/.json document") + parser.add_argument("--returned", required=True, help="the document the converter returned") + parser.add_argument("--diagnostics", help="a text file of the converter's warnings/notices") + parser.add_argument("--converter", default="", help="converter name, for the report header") + parser.add_argument("--json", action="store_true", help="emit JSON instead of text") + args = parser.parse_args(argv) + + diagnostics_text = "" + if args.diagnostics: + with open(args.diagnostics, "r", encoding="utf-8") as handle: + diagnostics_text = handle.read() + + report = grade( + _load(args.input), _load(args.returned), + diagnostics_text=diagnostics_text, converter=args.converter, + ) + + if args.json: + print(json.dumps({ + "converter": report.converter, + "rejected": report.rejected, + "findings": [vars(f) for f in report.findings], + "summary": report.counts(), + }, indent=2)) + else: + print(_format(report)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/compliance/portability/grader/test_grade.py b/compliance/portability/grader/test_grade.py new file mode 100644 index 000000000..1b35bb84f --- /dev/null +++ b/compliance/portability/grader/test_grade.py @@ -0,0 +1,159 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Outcome tests for the #576 fidelity grader. + +Each test builds a tiny input document and a synthetic returned document that +exercises exactly one outcome. The last test runs the real posted M0-M3 fixtures +through a no-op converter: returned unchanged, every element must grade PORTABLE. +""" + +import glob +import os + +import grade + + +def _metric(name, dialect, expression): + return {"name": name, "expression": {"dialects": [{"dialect": dialect, "expression": expression}]}} + + +def _field(dataset, name, dialect, expression): + return { + "name": dataset, + "fields": [{"name": name, "expression": {"dialects": [{"dialect": dialect, "expression": expression}]}}], + } + + +def _doc(metrics=None, datasets=None): + document = {"version": "0.2.0.dev0", "name": "t"} + if metrics is not None: + document["metrics"] = metrics + if datasets is not None: + document["datasets"] = datasets + return document + + +def _by_name(report): + return {f.name: f for f in report.findings} + + +def _only(report): + assert len(report.findings) == 1, [f.outcome for f in report.findings] + return report.findings[0] + + +def test_portable_ansi(): + doc = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + assert _only(grade.grade(doc, doc)).outcome == grade.PORTABLE + + +def test_portable_ossiesql_sets_tag_preserved(): + doc = _doc(metrics=[_metric("total", "OSSIE_SQL_2026", "SUM(sales.amount)")]) + finding = _only(grade.grade(doc, doc)) + assert finding.outcome == grade.PORTABLE + assert finding.tag_preserved is True + + +def test_relabeled_when_tag_changes_but_text_is_kept(): + src = _doc(metrics=[_metric("total", "OSSIE_SQL_2026", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + assert _only(grade.grade(src, returned)).outcome == grade.RELABELED + + +def test_rewritten_cosmetic_requalify_is_marked_benign(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(amount)")]) + returned = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + finding = _only(grade.grade(src, returned)) + assert finding.outcome == grade.REWRITTEN + # When sqlglot is present the requalify is recognised as cosmetic. + if grade._equivalent_tree("SUM(amount)", "SUM(sales.amount)") is True: + assert "cosmetic" in finding.detail + + +def test_rewritten_structural_change_is_flagged_for_review(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(SUM(sales.amount))")]) + finding = _only(grade.grade(src, returned)) + assert finding.outcome == grade.REWRITTEN + if grade._equivalent_tree("SUM(sales.amount)", "SUM(SUM(sales.amount))") is False: + assert "review" in finding.detail + + +def test_survived_vendor_only_real_translation(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("total", "DAX", "SUM('Sales'[Amount])")]) + assert _only(grade.grade(src, returned)).outcome == grade.SURVIVED_VENDOR_ONLY + + +def test_placeholder_blank_stand_in(): + src = _doc(datasets=[_field("sales", "amount_with_tax", "ANSI_SQL", "amount * 1.1")]) + returned = _doc(datasets=[_field("sales", "amount_with_tax", "DAX", "BLANK()")]) + assert _only(grade.grade(src, returned)).outcome == grade.PLACEHOLDER + + +def test_dropped_silent_and_added(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("other", "ANSI_SQL", "SUM(sales.cost)")]) + outcomes = {name: f.outcome for name, f in _by_name(grade.grade(src, returned)).items()} + assert outcomes["total"] == grade.DROPPED_SILENT + assert outcomes["other"] == grade.ADDED + + +def test_dropped_diagnosed_when_warning_names_it(): + src = _doc(metrics=[_metric("margin", "ANSI_SQL", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("keep", "ANSI_SQL", "SUM(sales.cost)")]) + report = grade.grade(src, returned, diagnostics_text="measure 'margin' excluded, preserved in stash") + assert _by_name(report)["margin"].outcome == grade.DROPPED_DIAGNOSED + + +def test_rejected_when_output_has_no_elements(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + report = grade.grade(src, {"version": "0.2.0.dev0", "name": "t"}) + assert report.rejected is True + assert _only(report).outcome == grade.REJECTED + + +def test_identity_not_preserved_is_repaired_not_dropped_plus_added(): + # orionbelt writes a field's expression as its name on the way back. + src = _doc(datasets=[_field("sales", "amount_with_tax", "ANSI_SQL", "amount * 1.1")]) + returned = _doc(datasets=[_field("sales", "amount * 1.1", "ANSI_SQL", "amount * 1.1")]) + report = grade.grade(src, returned) + outcomes = [f.outcome for f in report.findings] + assert outcomes == [grade.REWRITTEN], outcomes + assert "identity not preserved" in report.findings[0].detail + + +def test_whitespace_is_not_a_difference(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM( sales.amount )")]) + assert _only(grade.grade(src, returned)).outcome == grade.PORTABLE + + +def test_identity_on_real_m0_m3_fixtures_is_all_portable(): + here = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + files = sorted( + path + for rung in ("m0", "m1", "m2", "m3") + for path in glob.glob(os.path.join(here, rung, "*.ossie.yaml")) + ) + assert files, "no M0-M3 fixtures found" + for path in files: + document = grade._load(path) + report = grade.grade(document, document) + bad = [(f.name, f.outcome) for f in report.findings if f.outcome != grade.PORTABLE] + assert not bad, f"{os.path.basename(path)} graded non-PORTABLE: {bad}" diff --git a/compliance/portability/m0/m0_ansi_qualified.ossie.yaml b/compliance/portability/m0/m0_ansi_qualified.ossie.yaml index 64b06bb36..b1df704c2 100644 --- a/compliance/portability/m0/m0_ansi_qualified.ossie.yaml +++ b/compliance/portability/m0/m0_ansi_qualified.ossie.yaml @@ -39,3 +39,4 @@ metrics: dialects: - dialect: ANSI_SQL expression: SUM(sales.amount) + datatype: Decimal diff --git a/compliance/portability/m0/m0_ossiesql_qualified.ossie.yaml b/compliance/portability/m0/m0_ossiesql_qualified.ossie.yaml index ea812222d..04244cac6 100644 --- a/compliance/portability/m0/m0_ossiesql_qualified.ossie.yaml +++ b/compliance/portability/m0/m0_ossiesql_qualified.ossie.yaml @@ -42,3 +42,4 @@ metrics: dialects: - dialect: OSSIE_SQL_2026 expression: SUM(sales.amount) + datatype: Decimal diff --git a/compliance/portability/m0/m0_sum_bare.ossie.yaml b/compliance/portability/m0/m0_sum_bare.ossie.yaml index f51f99f1f..aac63ba18 100644 --- a/compliance/portability/m0/m0_sum_bare.ossie.yaml +++ b/compliance/portability/m0/m0_sum_bare.ossie.yaml @@ -44,3 +44,4 @@ metrics: dialects: - dialect: ANSI_SQL expression: SUM(amount) + datatype: Decimal diff --git a/compliance/portability/manifest.yaml b/compliance/portability/manifest.yaml index 53fece897..0382f779f 100644 --- a/compliance/portability/manifest.yaml +++ b/compliance/portability/manifest.yaml @@ -41,14 +41,24 @@ outcomes: # per-element, assigned by the grader after diffing input vs returned output - - PORTABLE # returns tagged ANSI_SQL or OSSIE_SQL_2026 + - PORTABLE # returns tagged ANSI_SQL or OSSIE_SQL_2026, expression unchanged - TAG_PRESERVED # sub-flag under PORTABLE: the OSSIE_SQL_2026 tag survived + - REWRITTEN # same portable tag, expression text changed. The grader must + # compare normalized expressions (e.g. sqlglot trees after + # qualification and unquoting) so a benign requalify/quote/reorder + # (cube, dbt, omni, sigma, orionbelt, honeydew on this run) is + # separated from a change in meaning (dbt SUM(SUM(CASE...)), #437) - SURVIVED_VENDOR_ONLY # returns in a vendor dialect only + - PLACEHOLDER # replaced by a valid stand-in that is not the authored content + # (e.g. microsoft emits BLANK() with the original stashed in an + # annotation the import does not read back), with a diagnostic. A + # diagnosed loss of content, distinct from SURVIVED_VENDOR_ONLY - ADDED # an element appears that was not in the input - RELABELED # same text, different tag than authored - DROPPED_DIAGNOSED # removed with a warning or notice - DROPPED_SILENT # removed with no diagnostic - - REJECTED # the converter refuses the whole document + - REJECTED # the converter refuses the whole document (document-level, e.g. + # thoughtspot exits non-zero when any one element will not translate) rungs: - id: M0 From db0b14da66ced91252eee504e51af21bbdbf64f4 Mon Sep 17 00:00:00 2001 From: "Mr. Black 1995" Date: Wed, 21 Oct 2026 13:25:42 +0200 Subject: [PATCH 3/3] docs(compliance): point the runner reference at apache/ossie-converters The converters moved to apache/ossie-converters (#523), so the interop matrix the thin runner is modelled on now lives at cube/tools/interop_matrix.py there. Co-authored-by: Isaac --- compliance/portability/README.md | 4 ++-- compliance/portability/grader/README.md | 4 ++-- compliance/portability/grader/grade.py | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/compliance/portability/README.md b/compliance/portability/README.md index f090f271a..9039231ac 100644 --- a/compliance/portability/README.md +++ b/compliance/portability/README.md @@ -24,8 +24,8 @@ model survives a round-trip through each converter. This is issue #576 (measurements for model portability across converters). The fixtures are the inputs; `grader/` scores each returned document against its input, element by element, into the outcome vocabulary in `manifest.yaml`. A thin runner that drives -the converters (modelled on `converters/cube/tools/interop_matrix.py`) turns the -whole thing into one command; the grader is written to sit behind it. +the converters (modelled on `cube/tools/interop_matrix.py` in apache/ossie-converters) +turns the whole thing into one command; the grader is written to sit behind it. ## What is here diff --git a/compliance/portability/grader/README.md b/compliance/portability/grader/README.md index b0f97fab8..0f699de97 100644 --- a/compliance/portability/grader/README.md +++ b/compliance/portability/grader/README.md @@ -25,8 +25,8 @@ this grader is the fidelity half. It does not drive the converters. It takes the input document and the document a converter returned, and diffs them element by element into the outcome vocabulary -in `../manifest.yaml`. A thin runner (modelled on -`converters/cube/tools/interop_matrix.py`) drives each converter and feeds the +in `../manifest.yaml`. A thin runner (modelled on `cube/tools/interop_matrix.py` in +apache/ossie-converters) drives each converter and feeds the grader, so the measurement table becomes one command. ## Usage diff --git a/compliance/portability/grader/grade.py b/compliance/portability/grader/grade.py index b8944dbac..7d8cd34a0 100644 --- a/compliance/portability/grader/grade.py +++ b/compliance/portability/grader/grade.py @@ -23,7 +23,7 @@ fidelity; this grader is the fidelity half. It does NOT drive the converters: that is the thin runner's job (modelled on -``converters/cube/tools/interop_matrix.py``). It takes the input document and the +``cube/tools/interop_matrix.py`` in apache/ossie-converters). It takes the input document and the document a converter returned, and diffs them element by element into the outcome vocabulary in ``../manifest.yaml``: