fix(converters): emit schema-conformant OSI documents (drop non-spec root dialects/vendors) by andreybavt · Pull Request #148 · apache/ossie

andreybavt · 2026-06-08T09:52:42Z

Problem

Output from the dbt→OSI converter fails OSI's own validator (validation/validate.py)
against the published core schema:

[Schema] (root): Additional properties are not allowed ('dialects' was unexpected)

Because the CLI writes via to_osi_yaml(), every dbt→OSI conversion produces a
non-conformant document.

Root cause

OSIDocument (python/src/osi/models.py) declared optional root-level
dialects/vendors fields that aren't in core-spec/osi-schema.json, whose root is
additionalProperties: false (only version + semantic_model). The dbt converter
set dialects=[self._dialect], so dialects was emitted at the document root.

Fix

Remove the non-spec dialects/vendors fields from OSIDocument.
Stop the dbt converter from populating a document-root dialect.

Per-expression dialect tagging (OSIExpression.dialects) is unchanged and remains the
schema-valid home for dialects, so no information is lost. Dialect selection still
flows end-to-end; the two affected converter tests now assert it on the per-expression path.

Regression guard

Adds converters/dbt/tests/test_schema_conformance.py, which converts a representative
manifest and validates the emitted document (YAML and JSON, for ANSI_SQL and
SNOWFLAKE) against core-spec/osi-schema.json, reusing validation/validate.py.
CI now fails if a converter emits a non-conformant document root again.

Testing

validate.py on converter output: fails before ('dialects' was unexpected),
passes after.
Full dbt converter test suite green.

Out of scope

Whether OSI should support a document-level (default) dialect is the open discussion in
#52 (one dialect per document) and #16 (default dialect at dataset level). This PR takes
no position and makes no schema change - it only aligns the reference model and
converter with the schema as published today.

Drop the non-spec root `dialects`/`vendors` fields from OSIDocument and stop the dbt converter emitting a root dialect, so dbt->OSI output validates against core-spec/osi-schema.json. Dialects remain per-expression (no information lost). Add a regression test that schema-validates converter output.

andreybavt · 2026-06-08T14:59:52Z

@khush-bhatia , let me know if this first PR follows the community guidelines. Happy to take bigger scope after this first one lands

khush-bhatia

Thanks for the fix.

QMalcolm

Looks good to me! Thank you for putting this together 🙂

khush-bhatia · 2026-06-11T21:28:28Z

Thanks @andreybavt, We will merge this PR soon, we are waiting on some process to complete.

andreybavt · 2026-06-11T22:00:55Z

Thanks @khush-bhatia and @QMalcolm ! Will continue with the other PRs then

…a-conformant OSI Addresses review feedback on this PR. The converter only emitted ANSI_SQL on the OBML -> OSI path, so a third-party OSI model (Snowflake/Databricks authored, or a non-SQL dialect) could lose metrics on import with only a warning, contradicting the README's lossless roundtrip claim. - Dialect catching: read ANSI_SQL, then SNOWFLAKE, then DATABRICKS expressions (SQL engines whose aggregations are ANSI-compatible); MDX/TABLEAU/MAQL are not parsed as SQL. - No silent loss: metrics with no OBML representation are preserved verbatim under an OSI-vendor model-level customExtension (obml_unconverted_metrics) and re-emitted on OBML -> OSI, with a loud LOSSY: warning. OSI -> OBML -> OSI stays lossless. - Schema conformance (aligns with PR apache#148): stop emitting root-level dialects/vendors. The published OSI core schema's root is additionalProperties:false (version + semantic_model only). Dialects live per-expression in expression.dialects[] and vendors per-entity in custom_extensions, the schema-valid homes, so no information is lost. Vendored schema updated to match published; a guard test asserts the root arrays are now rejected. - Name-collision safety: a preserved metric whose name a converted metric now owns is skipped on export, so the output has no duplicate metric names. - Idempotency: convert() resets per-run accumulators on both directions. - Docs: scope the README fidelity claim; refresh stale v0.1.1 references and the primary_key entry in the mapping analysis. - Tests: foreign-OSI fixtures covering both former drop paths, schema conformance, name collision, and idempotency.

…a-conformant OSI Addresses review feedback on this PR. The converter only emitted ANSI_SQL on the OBML -> OSI path, so a third-party OSI model (Snowflake/Databricks authored, or a non-SQL dialect) could lose metrics on import with only a warning, contradicting the README's lossless roundtrip claim. - Dialect catching: read ANSI_SQL, then SNOWFLAKE, then DATABRICKS expressions (SQL engines whose aggregations are ANSI-compatible); MDX/TABLEAU/MAQL are not parsed as SQL. - Identifier resolution: parsed dataset.column references are resolved against the OSI dataset/field map case-insensitively and with SQL quoting stripped, so Snowflake forms like SUM(ORDERS.AMOUNT) or SUM("Orders"."amount") map to the real Orders.amount. Unmappable references preserve the metric verbatim instead of emitting a dangling reference. - No silent loss: metrics with no OBML representation are preserved verbatim under an OSI-vendor model-level customExtension (obml_unconverted_metrics) and re-emitted on OBML -> OSI, with a loud LOSSY: warning. OSI -> OBML -> OSI stays lossless. - Schema conformance (aligns with PR apache#148): stop emitting root-level dialects/vendors. The published OSI core schema's root is additionalProperties:false (version + semantic_model only). Dialects live per-expression in expression.dialects[] and vendors per-entity in custom_extensions, the schema-valid homes, so no information is lost. Vendored schema updated to match published; a guard test asserts the root arrays are now rejected. - Name-collision safety: a preserved metric whose name a converted metric now owns is skipped on export, so the output has no duplicate metric names. - Idempotency: convert() resets per-run accumulators on both directions. - Docs: scope the README fidelity claim; refresh stale v0.1.1 references and the primary_key entry in the mapping analysis. - Tests: foreign-OSI fixtures covering both former drop paths, identifier resolution, schema conformance, name collision, and idempotency.

khush-bhatia · 2026-07-13T22:19:04Z

@andreybavt Because of the rename, there are some merge conflicts, could you please resolve the conflicts.

kayemkim · 2026-09-14T03:51:17Z

Since this was opened, #297 added root-level dialects/vendors to the schema, so the premise here (that the schema rejects them) no longer holds on main. I have written the details up in #393 rather than here, since it also touches #306, the Sigma converter and the orionbelt tests.

christianeu-db · 2026-10-23T15:34:27Z

Closing due to changes in the core spec + move of converters to ossie-converters. @andreybavt if you're interested, there are a number of new issues to solve in ossie and the ossie-converters repo

khush-bhatia requested a review from QMalcolm June 8, 2026 18:23

khush-bhatia approved these changes Jun 8, 2026

View reviewed changes

QMalcolm approved these changes Jun 11, 2026

View reviewed changes

ralfbecher mentioned this pull request Jun 16, 2026

Add OrionBelt (OBML) converter #153

Merged

onkardahale mentioned this pull request Jul 1, 2026

python: accept free-form custom extension vendors #172

Merged

This was referenced Aug 12, 2026

Core spec: document root cannot carry ai_context or custom_extensions #322

Closed

Core spec: allow ai_context on the document root #323

Closed

kayemkim mentioned this pull request Sep 14, 2026

Core spec: root-level dialects/vendors are allowed by the schema since #297 but treated as non-spec everywhere else #393

Closed

This was referenced Sep 14, 2026

Keep converter outputs aligned with root schema #306

Closed

fix(core-spec): drop root-level dialects and vendors from the schema #397

Merged

christianeu-db closed this Oct 23, 2026

Navigation Menu

Repository navigation

fix(converters): emit schema-conformant OSI documents (drop non-spec root dialects/vendors) - #148

fix(converters): emit schema-conformant OSI documents (drop non-spec root dialects/vendors)#148
andreybavt wants to merge 1 commit into
apache:mainfrom
Kaelio:fix/schema-conformant-osi-documents

andreybavt commented Jun 8, 2026 •

edited

Loading

Uh oh!

andreybavt commented Jun 8, 2026

Uh oh!

khush-bhatia left a comment

Uh oh!

QMalcolm left a comment

Uh oh!

khush-bhatia commented Jun 11, 2026

Uh oh!

andreybavt commented Jun 11, 2026

Uh oh!

khush-bhatia commented Jul 13, 2026

Uh oh!

kayemkim commented Sep 14, 2026

Uh oh!

christianeu-db commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

andreybavt commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Root cause

Fix

Regression guard

Testing

Out of scope

Uh oh!

andreybavt commented Jun 8, 2026

Uh oh!

khush-bhatia left a comment

Choose a reason for hiding this comment

Uh oh!

QMalcolm left a comment

Choose a reason for hiding this comment

Uh oh!

khush-bhatia commented Jun 11, 2026

Uh oh!

andreybavt commented Jun 11, 2026

Uh oh!

khush-bhatia commented Jul 13, 2026

Uh oh!

kayemkim commented Sep 14, 2026

Uh oh!

christianeu-db commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

andreybavt commented Jun 8, 2026 •

edited

Loading