Skip to content

fix(dbt): preserve config.meta through Ossie conversion - #615

Closed
RyutoYoda wants to merge 1 commit into
apache:mainfrom
RyutoYoda:codex/dbt-config-meta-latest
Closed

RyutoYoda wants to merge 1 commit into
apache:mainfrom
RyutoYoda:codex/dbt-config-meta-latest

Conversation

@RyutoYoda

@RyutoYoda RyutoYoda commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Preserve non-empty dbt config.meta when converting between dbt MetricFlow semantic manifests and Ossie documents. The converter stores each metadata object in a DBT custom extension on its matching dataset, field, or metric, and restores it during reverse conversion. Empty metadata is omitted; malformed or duplicate DBT extensions are rejected with clear errors.

The default dbt converter test suite includes a real-dbt E2E: it runs dbt parse, reads the generated semantic manifest, invokes the ossie-dbt CLI, and verifies metadata through reverse conversion. The test uses the DuckDB adapter and needs no credentials or external database.

Related Issues

Fixes apache/ossie-converters#42

Verification

  • uv run --frozen dbt --version — dbt Core 1.12.6, DuckDB adapter 1.11.0
  • uv run --frozen pytest -q — 181 passed, 5 snapshots passed, 0 skipped
  • Real-dbt E2E — passed
  • Python 3.11, 3.12, 3.13, and 3.14 — full suite passed
  • uv lock --check and git diff --check — passed
  • GitHub Actions currently shows the labeling and welcome helper workflows queued; neither is a test workflow.

Checklist

  • Converter logic updated
  • New metadata validation cases covered by tests
  • User-facing behavior documented with an example
  • All tests pass, including the real-dbt E2E
  • New source and test files include ASF license headers
  • PMC/IPMC approval for the new dev-only dependency dbt-duckdb==1.11.0 — requested via [email protected] on 2026-10-20; approval is pending

Spec, ontology, validation package, examples, and contributor workflow are unchanged. No third-party source code is vendored.

@RyutoYoda
RyutoYoda marked this pull request as ready for review October 19, 2026 17:54
@RyutoYoda
RyutoYoda marked this pull request as draft October 19, 2026 17:57
@RyutoYoda
RyutoYoda marked this pull request as ready for review October 19, 2026 18:00

@christianeu-db christianeu-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this - the mapping looks correct, and the defensive reverse path (duplicate/malformed extension handling) and sort_keys serialization are nice.

My main concern is the test dependency. dbt-duckdb==1.11.0 pulls the full dbt Core tree into the lockfile (~700 lines, 37 packages) and gates the PR on a pending dev@ approval - all to drive one round-trip assertion. Could we drop the dependency and the real-dbt E2E, and cover the round-trip with fixtures? That removes the whole lock diff and the approval gate.

Worth noting #378 fixes the same issue (#303) with that fixture-based approach, no new dependencies. Since both PRs are converging on the same DBT custom-extension mechanism, it might be worth you and @georgea-tk coordinating on a single version - #615's validation and docs plus #378's lighter footprint would make a strong combined PR.

@RyutoYoda RyutoYoda closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

msi_to_osi silently discards config.meta instead of mapping it to custom_extensions

2 participants