[christianeu-db]

Summary

Context

Discussion for this PR is in this thread: https://lists.apache.org/thread.html/vmgv5ffryw70wxbvtwnohhhz2zzx0ocf

Ossie is a vendor-neutral specification. Its extensibility model already reflects
this: a custom_extensions entry pairs a free-form vendor_name with a single
opaque data string. A vendor or an individual can attach any amount of extra
state, but that state is cordoned into one opaque value rather than promoted to
top-level spec fields. The core stays small and vendor-neutral, and everything
vendor-specific is opaque payload that the core carries but does not interpret.

Expressions are the one place where this principle is not yet applied. This PR
extends the same pattern to expressions and dialects. It is a spec-level change;
see the mailing-list threads under Related Issues.

1. Make the portable Ossie expression the primary field (backward-compatible)

Today an expression is a list of dialects, each a {dialect, expression} pair.
Ossie defines one portable dialect, OSSIE_SQL_2026
(core-spec/expression_language.md), but in practice it sits as just another
entry in the dialects list, peer to vendor dialects.
This is the only place in the spec where an Ossie concept and a vendor
concept appear at the same level, and the portable form is rarely authored as a
result (there are some workstreams in flight to close this gap).

This PR adds an optional top-level expression field to the expression object. It
is the primary, portable form, written in OSSIE_SQL_2026. The dialects list
remains for vendor-specific variants. When both the top-level expression and an
OSSIE_SQL_2026 entry in dialects are present, the top-level field is
authoritative. For portable-only expressions, this leads to a more concise representation
since the dialect sections / dialect type string can be dropped.

The field is optional and additive, so the change is backward-compatible: existing
documents remain valid.

2. Make dialect a free-form string, not an enum

dialect is currently a closed enum in core-spec/ossie-schema.json. Adding a
dialect requires a change to the core spec and a DISCUSS thread, and a dialect
outside the enum fails validation by default, so an outside party cannot register
its own dialect without a full spec change. Because the enum lists specific
products, it also reads as a tacit endorsement. Yet these dialects are explicitly
opaque: an implementation MAY ignore any non-OSSIE_SQL_2026 dialect, and
registering a dialect does not imply Ossie can query it.

vendor_name is already a free-form string for exactly these reasons. This PR
applies the same decoupling to dialect: enum to free-form string. One could argue
dialects belong entirely in custom_extensions, but expressions are fundamental
enough to keep as a first-class field; only the closed enumeration is removed.

Examples: expression portability before / after this PR

#	Job to be done	Before	After
1	Publish a measure that is already fully portable	✅ Portable form must be nested as a dialect entry. expression:   dialects:     - dialect: OSSIE_SQL_2026       expression: SUM(ss_sales_price)	✅ Portable form is a first-class top-level field. expression:   expression: SUM(ss_sales_price)
2	Keep vendor overrides alongside a portable form	✅ Portable and vendor forms are peers; nothing marks the primary. expression:   dialects:     - dialect: OSSIE_SQL_2026       expression: SUM(ss_sales_price)     - dialect: MAQL       expression: "SELECT SUM({fact/store_sales.ss_sales_price})"	✅ Portable form is primary; vendor forms are optional overrides. expression:   expression: SUM(ss_sales_price)   dialects:     - dialect: MAQL       expression: "SELECT SUM({fact/store_sales.ss_sales_price})"
3	Model an expression that cannot be made portable	✅ Vendor-only dialects, no portable form. expression:   dialects:     - dialect: DATABRICKS       expression: approx_count_distinct(session_id)     - dialect: SNOWFLAKE       expression: APPROX_COUNT_DISTINCT(session_id)	✅ Identical — omitting the top-level expression means "not portable." expression:   dialects:     - dialect: DATABRICKS       expression: approx_count_distinct(session_id)     - dialect: SNOWFLAKE       expression: APPROX_COUNT_DISTINCT(session_id)
4	Support an engine Ossie does not enumerate yet (FakeDB)	❌ Fails validation — FakeDB is not in the enum; valid only after editing core-spec/spec.yaml and regenerating the SDK types. expression:   dialects:     - dialect: FakeDB       expression: quantile_cont(latency_ms, 0.95)	✅ Valid — unknown dialect is opaque; optionally register it in extensions/dialects.md (docs only). expression:   dialects:     - dialect: DUCKDB       expression: quantile_cont(latency_ms, 0.95)

Rows 1–3 are additive: every existing document stays valid and dialects
keeps working. Row 4 is the new capability, mirroring how vendor_name
already works for custom_extensions.

Why this matters

Together, the future shape is an expression whose primary, portable form is defined
once at the top level, with dialects as optional overrides. The Hex converter calls
out the variety of expression types: not every dialect is a SQL dialect - some are richer
expression languages. More broadly, there is no reason an expression could not be a
code snippet or any other evaluation detail. A closed enum is too restrictive for
that range; a free-form string is not.

Extensions registry

The dialect and vendor example constants move out of the normative spec into a new
top-level extensions/ folder (README.md, dialects.md, vendors.md). This is a
community registry of well-known tokens: a place to align on their spelling and
meaning without making any token fundamental to the spec. Presence in a list does
not imply endorsement, and new tokens can be added without a full spec review.

Changes in this PR

• core-spec/ossie-schema.json: Dialect becomes type: string with illustrative
  examples and no enum; Vendor examples are genericized; the Expression
  object gains the optional expression field with its precedence rule.
• core-spec/spec.md, spec.yaml, expression_language.md: document the free-form
  dialect, the portable expression field, and the link to extensions/; the
  opaque / declaration-only text is retained.
• extensions/: new README.md, dialects.md, vendors.md.
• scripts/generate-spec-types.py: no longer generates a dialects enumeration
  (dialects are free-form, like vendors); it still syncs datatypes.
• python/ SDK: OssieDialectExpression.dialect is typed str; OssieExpression
  gains the optional expression; OssieDialect / OssieVendor remain as
  convenience constants.
• validation/validate.py: an unregistered dialect is treated as opaque and skipped
  rather than parsed as ANSI SQL; the portable expression is validated as
  OSSIE_SQL_2026.
• examples/tpcds_semantic_model.yaml: demonstrates the portable expression field.

Scope and follow-ups

The scope is deliberately minimal. The following refinements are left for follow-up
PRs:

• Converters: this will be a separate change in ossie-converters
• The vendor and dialect constant code and validation should move further from the
  core — into the converters or a shared mixins package — rather than living
  alongside the core models. This can be done incrementally.
• The OSSIE_SQL_2026 reading logic should be generalized to bias toward the primary
  top-level expression when it is defined. This PR documents that precedence in the
  spec and the reference validator honors it, but the converters are not yet updated
  to read the field.
• Pulling the Ossie -> vendor mappings from expressions will be a follow up

Related Issues

• Relates to Add measurements for model portability across converters #576 (finding that expressions were not ported in practice)
• Mailing-list discussion:
  • https://lists.apache.org/thread.html/0tql9g8myg255hhonhbzo906rdpgstcf (listing all the spots vendor-specific details have entered the spec)
  • https://lists.apache.org/thread.html/xh6o9mzhmrdpnjzxsn7kotjsg18yksvb (DISCUSS for adding an enum)
  • https://lists.apache.org/thread.html/46h1cr804mn6m4r35vrqn1tkdhrjxm7m (DISCUSS for adding 2 other enums)
  • https://lists.apache.org/thread.html/msbk9hml117ttyoxgqqz1h59s1rdpf3b (DISCUSS finding that listing the ossie dialect separately increases context size / vendor extensions are a huge part of payloads in practice)

Checklist

Specification

• Spec changes are included in core-spec/ and follow the existing structure
• Spec changes have been discussed on the mailing list or in a linked issue: https://lists.apache.org/thread.html/vmgv5ffryw70wxbvtwnohhhz2zzx0ocf
• Breaking changes to the spec are clearly called out in the summary (none — the change is additive and relaxes validation; the SDK dialect field type changed from an enum to str)

Ontology

• Ontology changes in ontology/ are consistent with spec changes (N/A — no ontology changes)
• New or modified terms are defined and documented (N/A)

Converters

• Converter change in f8d3508 will be opened in a new issue / PR in ossie-converters once this merges

Validation

• Validation rules in validation/ are updated if the spec changed
• New validation cases are covered by tests

Documentation

• docs/ is updated to reflect any user-facing changes (follow-up; see Scope)
• New features or behaviors are documented with examples where appropriate (in core-spec/ and examples/)
• CONTRIBUTING.md is updated if the contribution process changed (N/A)

Examples

• examples/ are added or updated for any new spec constructs or converter support. Examples in the spec and updated a store sample

Tests

• All existing tests pass (pytest / CI green)
• New functionality is covered by tests

Compliance

• ASF license headers are present on all new source files
• No third-party dependencies are added without PMC/IPMC approval
• If third-party source code is vendored/copied (not just declared as a dependency), NOTICE and/or LICENSE have been updated per ASF policy (N/A — no vendored third-party source)

[kayemkim]

I read this alongside the list thread and merged it locally onto current main (b42ce09), no conflicts. Three things on the code; the design points are on the list.

The hex check job fails at the head: ty check reports invalid-argument-type at load_ossie_dialect_expression.py:79, where _to_sqlglot_dialect(ossie_dialect: OssieDialect) now receives the str that OssieDialectExpression.dialect became. The runtime side matters more: that function ends in assert_never, so a document with a dialect outside the old enum, which this PR makes valid, raises AssertionError: Expected code to be unreachable, but got: 'DB_DIALECT' on export. I ran it directly with DB_DIALECT and SNOWFLKE; both raise. The wisdom test you added covers this case; hex needs the same, with the final branch returning None for anything unregistered and the parameter typed str.

Expression still lists dialects under required with minItems: 1, and OssieExpression.dialects keeps Field(..., min_length=1), so a portable-only document fails both the schema and the SDK:

[Schema] datasets -> 0 -> fields -> 0 -> expression: 'dialects' is a required property

A required of one of expression or dialects (anyOf) in the schema and a model validator in the SDK would match the behaviour the summary describes.

With #612 cherry-picked on top, the two tests that read $defs.Dialect.enum fail with KeyError (validation test_every_schema_dialect_is_mapped_or_skipped, Holistics test_every_schema_dialect_is_parsed_or_marked_not_sql); the table cross-check and the sqlglot reader tests pass. I can point those two at the table in extensions/dialects.md instead, in whichever order the two PRs land.