feat: add lightdash converter by Spissable · Pull Request #360 · apache/ossie

Spissable · 2026-09-04T09:26:12Z

Summary

Adds the bidirectional apache-ossie-lightdash converter and ossie-lightdash CLI for the current flat Ossie document format.

 Ossie ↔ Lightdash
- no Lightdash import/export support
+ import dbt metadata or native Lightdash models into a flat Ossie document
+ export a native Lightdash project or dbt schema.yml

Converts datasets, dimensions, metrics, joins, primary keys, datatypes, and AI hints; preserves Lightdash-specific metadata in the LIGHTDASH extension and reports unsupported/lossy constructs.
Derives safe model names from quoted identifiers or logical dataset names. Duplicate names, including case-only collisions, fail before either output format writes files. Models and joins share the validated names.
Qualifies bare SQL columns using SQLGlot, preserving types, date parts, qualified paths, comments, literals, and original formatting. Unsupported syntax is retained with an explicit issue.
Supports OSSIE_SQL_2026 expressions and dbt 1.10+ metadata placement. Documents include runnable local round-trip instructions.

Validation before commit:

104 converter tests passed on Python 3.14; 411 shared SDK/validation tests passed.
Spec enum synchronization and canonical TPC-DS validation passed.
Installed CLI: TPC-DS export → Lightdash compilation (5 explores, 0 errors) → import → schema/SQL validation; dbt-meta export/import also validated. Both round trips retain 5 datasets, 8 metrics, and 4 relationships.
Quoted-source/subquery/join fixture compiled (2 explores); cast/date-part fixture compiled (1 explore). CLI collision reproduction exited 2 without changing existing output.

Additional live verification on localhost after commit:

Imported native models into a flat document, exported and compiled them, and deployed two explores to an isolated preview using the existing Postgres connection.
Executed CAST, EXTRACT, and DATE_TRUNC against known fixture values; asserted both expected rows and verified the results in the browser.
Executed the quoted orders source joined to a SQL-subquery model; totals of 15 and 23 matched the original project's source results.
Retained the preview and generated local artifacts for inspection. No warehouse tables or local service configuration were changed. Snowflake/TPC-DS warehouse execution remains untested.
CI on the latest commit is awaiting maintainer approval; local results do not establish remote CI success.

Related Issues

Supersedes #253 with permission from @ota2000.

Checklist

Specification

Register LIGHTDASH as a well-known vendor without changing the document schema.

Converters

Use the current flat document format in the converter and CLI.
Cover conversion behavior and both review regressions in the existing test files.
Add the converter path to the repository labeler.

Validation

Run focused converter, shared SDK, schema, and SQL validation.
Trigger converter CI on shared Python, core-spec, and example changes.

Documentation

Document supported mappings, fidelity limits, and reproducible local checks.

Tests

Relevant local tests and CLI end-to-end checks pass.
Remote CI passes after a maintainer approves the workflows.

Compliance

ASF license headers are present on new source files.
No third-party source code is vendored; SQLGlot is an existing repository dependency reused by this converter.

Bidirectional converter between Ossie documents and Lightdash semantic definitions (dbt schema.yml meta blocks): datasets/fields/metrics/ relationships map to models/columns/meta.metrics/meta.joins, and Lightdash presentation attributes round-trip through custom_extensions with vendor_name "lightdash". Follows the apache-ossie-<vendor> uv packaging layout and ships tests off the TPC-DS example, including a structural round-trip.

…elity - Add converter-lightdash-ci.yml mirroring the other converters (tests were otherwise not exercised by CI) - Rewrite ${other_table.column} references into cross-dataset references on import, not just ${TABLE}.column - Keep inexpressible metric types (percentile) in the extension even when the metric carries custom SQL, on both column-level and model-level metrics - Drop the redundant per-converter .gitignore (covered by the root one) and unused helpers; document the not-yet-carried model-level meta keys in the README

…ip columns - Drop structural keys (sql/label on dimensions, sql/description on metrics) from the extension overlay on export, so a hand-authored extension can never override the OSI-derived definition; metric type stays overridable as the documented channel for inexpressible types - Skip relationships whose from_columns/to_columns lengths differ, reporting RELATIONSHIP_COLUMNS_MISMATCHED instead of silently truncating the join - Document that dataset names are not preserved when they differ from the source table name

… model metrics, add [tool.uv] - Surface unparseable lightdash extension data as an EXTENSION_DATA_INVALID issue instead of silently dropping it - Skip model-level metrics without sql on import, reporting METRIC_SQL_MISSING rather than emitting an empty OSI expression - Add the [tool.uv] block (required-version, default-groups) used by the other converters so uv sync reliably installs the dev group in CI

Mirror the change from the recent CI actions path cleanup: trigger on this workflow's own file instead of the whole .github tree.

* upstream/main: CI for databricks converter (apache#261) fix(dbt): resolve SUM_BOOLEAN semantic model for qualified columns (apache#292) [OSSIE][NVIDIA_GSF] Add bidirectional GSF semantic model converter (apache#247) Fix typo in python README.md (apache#284) Bump Java from 17 to 21 (apache#283) Adding AI disclosures updates (apache#259) Fix Omni pytest (apache#263) Updating CI Actions path (apache#276) Bump java version from 11 to 17 for Polaris converter (apache#278) Make the spec's own examples validate against osi-schema.json (apache#209) [Ontology] Flatten concept declarations in ontology spec (apache#257) feat(cli): add plugin invocation protocol (apache#155) Add datatype field to Field and Metric; reframe is_time as role marker (apache#113) Cleanup impl dir (apache#252)

Spec change apache#113 added `datatype` to fields and metrics and reframed `dimension.is_time` as a role marker rather than a type flag, which broke the TPC-DS round-trip: the converter inferred `is_time` from the Lightdash type and wrote `false` where the source left it unset. - Map Lightdash dimension types to and from `datatype`, so the type is carried in standard vocabulary instead of a vendor extension - Leave `is_time` unset on import (Lightdash has no equivalent marker) and report TIME_ROLE_NOT_REPRESENTABLE when a non-temporal field is flagged as a time axis on export - Emit a Lightdash type only for fields that are dimensions, so a measure-only field is not turned into a dimension by a round-trip - Document that datatypes round-trip by category rather than exact type

…name Upstream apache#288 renamed the Python models (OSIDocument -> OssieDocument, ...) and the osi_to_<vendor> / <vendor>_to_osi module convention. Apply the same rename to the Lightdash converter so it imports and runs against main.

The import command dumped the pydantic model without mode="json", so a .yaml target raised RepresenterError on OssieDialect. Take argv in main() and cover both output formats with a CLI test.

…port Real Lightdash projects alias most joins, reference sibling fields as ${column}, other metrics as ${metric} and use typed metrics over custom SQL. The importer matched joins on the model name only, left every non-${TABLE} reference in the Ossie expression, ignored the sql of typed metrics and emitted percentile and distinct metrics as a bare column. - resolve join aliases when parsing sql_on; stash the alias and other join attributes in the relationship's lightdash extension and restore them on export, aliasing repeated dataset pairs so Lightdash can compile them - rewrite ${column}, ${TABLE} and ${alias.column} references; inline ${metric} references; skip elements that use parameters or user attributes with an EXPRESSION_NOT_PORTABLE issue - express every typed metric as an aggregation over its operand, including SUM(DISTINCT ...), AVG(DISTINCT ...) and PERCENTILE_CONT ... WITHIN GROUP, and parse those shapes back into typed metrics on export On the 37-model jaffle-shop project this takes relationships from 1 of 21 to 21 of 21 and removes every unrewritten reference from the document.

…oins - export prefers --dialect, then ANSI_SQL, and reports DIALECT_UNAVAILABLE instead of silently taking whatever dialect comes first - import labels expressions with --dialect, since Lightdash SQL is written for the project's warehouse rather than ANSI - a metric spanning several datasets is hosted on the dataset that joins all the others directly, with ${joined_model.column} references, instead of being dropped; TPC-DS's two cross-dataset metrics now export - field expressions resolve joined datasets the same way and report FIELD_REFERENCE_UNJOINED when no join exists

Lightdash has vocabulary for more of the Ossie model than the README claimed: - dataset.primary_key <-> meta.primary_key (string or composite list) - ai_context <-> ai_hint on models, dimensions and metrics; multi-line instructions become a list of hints, structured synonyms and examples are rendered as extra hints on export - exported joins carry relationship: many-to-one, the cardinality an Ossie relationship defines, unless a stashed Lightdash value overrides it - time_intervals: OFF <-> an explicit is_time: false on a temporal column - metric datatype is derived from the aggregation and column type on import A label or ai_context on a measure-only field is now dropped with a FIELD_ATTRIBUTE_NOT_REPRESENTABLE issue instead of turning the field into a dimension on export.

Lightdash evaluates {% ... %} / {{ ... }} tags and bare ld.parameters, ld.query and ld.user references at query time; a dimension or metric using them is skipped with EXPRESSION_NOT_PORTABLE like ${ld....} references.

dbt 1.10+ nests meta under config; Lightdash merges meta and config.meta with config.meta winning. The importer only read the top-level key and saw an empty project when every model used the new placement. Export gains --meta-under-config to write that placement.

Lightdash scopes metric names per model, Ossie per semantic model, so any real project (count, total_events, unique_users on many models) produced an invalid document. The Ossie name is now Lightdash's own field id, <model>_<metric>, which is stable regardless of what other models define and matches the id users see in the API and URLs. The bare name travels in the lightdash extension and is restored on export; an Ossie metric without a stash exports under its name minus a <model>_ prefix. Names that still collide after qualification are suffixed with a METRIC_NAME_COLLISION issue.

…rgets Lightdash treats dbt seeds as models, so a schema file's seeds: entries become datasets too. A join whose target is not in the input would leave a dangling relationship (the validator rejects it); it is skipped with a JOIN_TARGET_UNKNOWN issue instead.

Lightdash SQL may name a column of the current model without ${TABLE} (SUM(budget_use)); the Ossie expression then carried no dataset qualifier and export could not place the metric in a multi-dataset document (67 of 390 jaffle-shop metrics were dropped). Bare identifiers that name one of the model's columns are now qualified, outside string literals and never when called as a function, and the hosting model is stashed in the extension so an expression that names no dataset at all is still placed.

…ield Lightdash explores compose joins per base model and carry model meta with no Ossie vocabulary; both were dropped silently. Now: - a join through another joined model (${projects.org_id} = ${organizations.org_id} on the queries explore) derives the edge between the two models it names, unless that model declares it itself, and the join is stashed verbatim on the dataset's lightdash extension; expression joins and joins with extra conditions are stashed the same way - export restores stashed joins, replacing the generated join to the same target and alias, and lets field and metric expressions resolve through them - model meta without Ossie vocabulary (label, hidden, sql_filter, group_details, default_time_dimension, required_filters, ...) and column meta outside dimension/metrics (additional_dimensions, ...) travel in the dataset's and field's extension and come back as they were - aliased joins on the same columns stay distinct relationships; pair and side order in sql_on no longer force a stash On the analytics project every one of 140 joins and all 60 models' meta now round-trip; jaffle-shop and demo-f1 likewise.

Lightdash treats every dbt column as a dimension by default; only meta.dimension.hidden withdraws it from grouping. The converter read the presence of meta.dimension as the dimension marker instead, so a real project exported most of its columns as measure-only fields (5,481 of 7,435 on the Lightdash analytics project). Import now emits dimension: {} for every column that is not hidden, and no dimension for a hidden one. Export writes a measure-only Ossie field as a hidden dimension, keeping its type, label and AI hint, and writes nothing for a dimension with nothing else to say. FIELD_ATTRIBUTE_NOT_REPRESENTABLE is gone: labels and hints always have a home now.

A metric's filters and a model's sql_filter / sql_where / required_filters travel in the lightdash extension and round-trip, but every other consumer of the document sees the unfiltered aggregate or the unrestricted dataset. Both are now reported on import (METRIC_FILTER_NOT_PORTABLE, ROW_FILTER_NOT_PORTABLE) instead of passing silently.

Add LIGHTDASH to the spec's vendor-name table, the schema examples, the Python OssieVendor enum and the converters README, and make the Lightdash converter emit the registered token. Documents written with the earlier lowercase name are still read.

Restructure the README around usage, the Python API, the mapping, and a section on why each loss is unavoidable (name scope, what a dimension is, joins versus relationships, query-time evaluation, types), followed by what is kept for Lightdash only, approximated, and not carried.

An Ossie document now becomes a deployable Lightdash project without a dbt project around it: export writes lightdash/models/<model>.yml per dataset (type: model, sql_from = the dataset's source, typed dimensions with their own sql, metrics and joins in place) plus a starter lightdash.config.yml whose warehouse type follows --dialect or --warehouse. Model meta the dbt flavour has to stash becomes ordinary top-level keys. --format dbt-meta keeps the single schema.yml output for dbt projects. lightdash compile accepts the exported TPC-DS project: 5 explores, 0 errors.

dbt projects keep one schema YAML per model, so import now accepts a directory as well as a file: it walks it in sorted order, merges every list-valued models: and seeds: entry, and ignores target/, dbt_packages/ and dbt_project.yml. load_schema is exported for the Python API.

The CLI printed nothing on success and bare issue codes. It now reports what it wrote (model files and config, or datasets, metrics and relationships), explains each issue type the first time it appears, and closes with a count. Everything goes to stderr, issues first, like the other converters; stdout stays clean.

Lightdash learns most dimension types from the warehouse, not from YAML, so a Lightdash project imports with most fields untyped and the model-file export has to assume string for them. --catalog reads the catalog.json that dbt docs generate writes and fills datatype for every column without an authored type, reducing the physical type to Ossie's vocabulary (INT64 -> Integer, NUMBER(12,2) -> Decimal, TIMESTAMP_TZ -> DateTimeTz). Authored types win. A model missing from the catalog is reported, which is how a stale catalog shows up. On demo-f1 this takes the model-file export from 78 assumed types to 1.

The positional form still works.

A config written without a warehouse type carried CHANGE_ME and was then never rewritten, so re-running with --dialect fixed the models but left lightdash compile refusing the placeholder. The config is now rewritten while it still holds the placeholder and kept once a real type is in it.

…y element A real project produced hundreds of one-line issues. The report is now one block per issue type: a header with the count, the explanation, and the affected elements wrapped to a readable width (the first few, or all of them with -v/--verbose).

…project A template with placeholders or a Jinja-only file crashed the walk; it is now skipped with a note naming the file. Virtualenv directories (env, venv, .venv, site-packages) are ignored like target/ and dbt_packages/.

import now reads Lightdash model files (type: model, sql_from, a dimensions list) as well as dbt schema files, mixed freely in one directory. A model file is folded into the dbt model shape: dimensions become columns, its sql_from is the dataset source verbatim (so --database/--schema are not needed for it), and the rest becomes model meta. This closes the round trip for dbt-less Lightdash projects and for anything export writes.

jbonofre · 2026-10-08T15:29:18Z

+        node["config"] = {"meta": meta}
+
+
+def _model_name_for(dataset: OssieDataset) -> str:


_model_name_for takes everything after the last . in dataset.source, which only works for plain a.b.c names:

A quoted source like proj.ds.tbl gives the name tbl, and a subquery source gives garbage. The CLI uses the model name directly as the file name (models/{model['name']}.yml), so these can produce invalid file names.

Two datasets with the same table name in different schemas (sales.orders, archive.orders) get the same model name and the same output file. The second silently overwrites the first.

Join targets and sql_on references use the same names, so they would point at the wrong model.

Could we strip quoting, fall back to the dataset name when the source is not a simple identifier path, and detect collisions? Failing with a clear error, or disambiguating the name, would be better than overwriting silently. A test with two same-named tables in different schemas would pin this down.

Thanks, fixed.
Simple identifier paths now have quoting stripped, and subqueries or other source expressions fall back to the logical dataset name. Model names and joins use the same validated mapping.

jbonofre · 2026-10-08T15:34:55Z

+_STRING_LITERAL_RE = re.compile(r"('(?:[^']|'')*')")
+
+
+def qualify_bare_columns(sql: str, dataset: str, columns) -> str:


Any identifier that matches a column name gets prefixed with the dataset, including SQL keywords and type names outside string literals. With a column named date or year:

CAST(x AS date) becomes CAST(x AS ds.date)

EXTRACT(year FROM d) becomes EXTRACT(ds.year FROM d)

The regex skips identifiers preceded by . but not ones followed by .. So other.col becomes ds.other.col if the model has a column named other. The result is invalid SQL in the imported Ossie expression, and columns like date and year are common enough that I would expect this to be hit in practice.

Could we also skip identifiers followed by . and the positions where a type or keyword is expected (after AS in a cast, the first argument of EXTRACT/DATE_TRUNC)? A tokenizer-based approach would be more robust than a regex. Tests with columns named date, year and other would cover these cases.

Fixed by replacing the regex with SQLGlot column detection, preserving types, date parts, and qualified references.

hamzahc1 · 2026-10-21T12:07:25Z

@jbonofre could we get a 2nd pass on this?

jbonofre · 2026-10-21T12:08:31Z

@hamzahc1 can you please move this PR to the new Ossie-converters repo?

jbonofre · 2026-10-21T12:08:48Z

I will close this one.

owlas · 2026-10-21T12:15:47Z

@jbonofre - i'm on it now

Adds the `apache-ossie-lightdash` converter and the `ossie-lightdash` CLI, migrated from apache/ossie#360 after the converters moved to this repository. Import reads dbt `meta` or native Lightdash model files and writes a flat Ossie document. Export reads an Ossie document and writes a native Lightdash project, which `lightdash deploy` reads without changes, or a dbt `schema.yml`. Lightdash-only attributes are stored in the `LIGHTDASH` custom extension; constructs with no equivalent on the other side are reported as a `ConverterIssue`. Compared with the branch reviewed on apache/ossie#360: - Malformed input raises `ConversionError` naming the element, in both directions, and the CLI exits with code 2 instead of a traceback. - Every round-tripped document is validated against `ossie/core-spec/ossie-schema.json`. - The M0-M3 portability fixtures from apache/ossie#576 round-trip with every metric intact; the dialect tag is `ANSI_SQL` by default and `OSSIE_SQL_2026` when the import is given that dialect. - CI installs the Lightdash CLI and runs `lightdash compile` over the exported TPC-DS project. The test skips when the CLI isn't on `PATH`. - The core-spec edits are removed; `Vendor` accepts any string. - The README is restyled for active voice and plain wording. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01VECsGjevfLr3kyZ6ToVycD

christianeu-db · 2026-10-22T06:06:01Z

closing (I see there is already an update in the ossie-converters)

Adds the `apache-ossie-lightdash` converter and the `ossie-lightdash` CLI, migrated from apache/ossie#360 after the converters moved to this repository. Import reads dbt `meta` or native Lightdash model files and writes a flat Ossie document. Export reads an Ossie document and writes a native Lightdash project, which `lightdash deploy` reads without changes, or a dbt `schema.yml`. Lightdash-only attributes are stored in the `LIGHTDASH` custom extension; constructs with no equivalent on the other side are reported as a `ConverterIssue`. Compared with the branch reviewed on apache/ossie#360: - Malformed input raises `ConversionError` naming the element, in both directions, and the CLI exits with code 2 instead of a traceback. - Every round-tripped document is validated against `ossie/core-spec/ossie-schema.json`. - The M0-M3 portability fixtures from apache/ossie#576 round-trip with every metric intact; the dialect tag is `ANSI_SQL` by default and `OSSIE_SQL_2026` when the import is given that dialect. - CI installs the Lightdash CLI and runs `lightdash compile` over the exported TPC-DS project. The test skips when the CLI isn't on `PATH`. - The core-spec edits are removed; `Vendor` accepts any string. - The README is restyled for active voice and plain wording. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01VECsGjevfLr3kyZ6ToVycD

ota2000 and others added 30 commits July 23, 2026 14:09

Align CI workflow path filter with the repo-wide update

41b8192

Mirror the change from the recent CI actions path cleanup: trigger on this workflow's own file instead of the whole .github tree.

Merge upstream main into add-lightdash-converter

adac280

fix(lightdash): serialize enums when importing to a YAML document

1611644

The import command dumped the pydantic model without mode="json", so a .yaml target raised RepresenterError on OssieDialect. Take argv in main() and cover both output formats with a CLI test.

fix(lightdash): treat Liquid templating as non-portable on import

b3a88b5

Lightdash evaluates {% ... %} / {{ ... }} tags and bare ld.parameters, ld.query and ld.user references at query time; a dimension or metric using them is skipped with EXPRESSION_NOT_PORTABLE like ${ld....} references.

docs(lightdash): installation without a checkout

93f2b9e

feat(lightdash): -i/--input and -o/--output like the other converters

25f9bdf

The positional form still works.

Spissable added 6 commits September 3, 2026 09:00

fix(lightdash): a missing input path is an error, not an empty document

f172517

chore(lightdash): remove workflow trailing whitespace

9eef6f2

chore: merge upstream main and resolve vendor conflicts

35f82cc

kayemkim mentioned this pull request Sep 17, 2026

Migrate converters and Python SDK to flat semantic model documents #396

Merged

5 tasks done

jbonofre self-requested a review October 8, 2026 13:32

jbonofre requested changes Oct 8, 2026

View reviewed changes

fix(lightdash): support flat documents and safe conversion

ab19d86

github-actions Bot added converter spec infra labels Oct 19, 2026

Spissable requested a review from jbonofre October 19, 2026 08:11

owlas mentioned this pull request Oct 21, 2026

Add lightdash converter apache/ossie-converters#7

Open

8 of 9 tasks

christianeu-db closed this Oct 22, 2026

Navigation Menu

Repository navigation

feat: add lightdash converter - #360

feat: add lightdash converter#360
Spissable wants to merge 37 commits into
apache:mainfrom
Spissable:add-lightdash-converter

Spissable commented Sep 4, 2026 •

edited

Loading

Uh oh!

jbonofre Oct 8, 2026

Uh oh!

Spissable Oct 19, 2026

Uh oh!

jbonofre Oct 8, 2026

Uh oh!

Spissable Oct 19, 2026

Uh oh!

hamzahc1 commented Oct 21, 2026

Uh oh!

jbonofre commented Oct 21, 2026

Uh oh!

jbonofre commented Oct 21, 2026

Uh oh!

owlas commented Oct 21, 2026

Uh oh!

christianeu-db commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

		node["config"] = {"meta": meta}


		def _model_name_for(dataset: OssieDataset) -> str:

		_STRING_LITERAL_RE = re.compile(r"('(?:[^']\|'')*')")


		def qualify_bare_columns(sql: str, dataset: str, columns) -> str:

Conversation

Spissable commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issues

Checklist

Specification

Converters

Validation

Documentation

Tests

Compliance

Uh oh!

jbonofre Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

Spissable Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

jbonofre Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

Spissable Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

hamzahc1 commented Oct 21, 2026

Uh oh!

jbonofre commented Oct 21, 2026

Uh oh!

jbonofre commented Oct 21, 2026

Uh oh!

owlas commented Oct 21, 2026

Uh oh!

christianeu-db commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Spissable commented Sep 4, 2026 •

edited

Loading