[nicoloesch]

Summary

Depends on AustralianCancerDataNetwork/oa-configurator#37.

Requires pyproject.toml version update for oa-configurator

Breaking change

OrmLoaderConfig.test_orm_db is gone, replaced by test_orm_db_pg/test_orm_db_sqlite. Anyone with test_orm_db set in their omop.config will need to update it. There is no back-compat shim. Needed because dialect-specific integration tests require a guaranteed-correct-dialect test database each, which one shared field couldn't express.

Root cause

Four related ways schema_translate_map was silently bypassed across orm-loader:

1. merge_replace/merge_insert/merge_upsert built raw, unqualified text() SQL
2. staging-table index inspection used a schema-blind sa.inspect()
3. materialized-view lifecycle methods took a hand-threaded schema string instead of resolving through the connection, and
4. a staging-table reflection path could open a second connection that, on SQLite, silently discarded a session's own uncommitted work.

Fix

Loaded/merged data

• merge_replace/merge_insert/merge_upsert rewritten against real Table objects and
  dialect-specific Core constructs (sa.delete, insert().from_select(),
  postgresql.insert().on_conflict_do_nothing(), with a SQLite-specific counterpart), for both backends.
• Fixed the one remaining raw isolation_level="AUTOCOMMIT" call site.
• Staging-table reflection reflects now through the session's ownconnection() to prevent data loss

Materialized views

• create_mv/refresh_mv/drop_mv/index creation are fully role-aware
  • each resolves its physical schema from the bound connection's own schema_translate_map (__mv_schema_tag__, defaulting to Role.PRIMARY)
• The Postgres-only dialect guard is now a single check inside the shared _as_connection() context manager, replacing a narrower per-method check that only ever guarded direct/manual PostgresBackend() construction.
• Staging-table index management (manage_indices) now resolves its Inspector schema-aware, via physical_schema_of(session, schema_tag=cls.__table__.schema).

Split primary/vocab connections

• Loading and merging now works correctly against genuinely separate primary/vocab connections

Config, CI, test infrastructure

• STAGING_SCHEMA's reserved-schema registration moved into config.py (the actual omop.config entry point), so it's guaranteed to run whenever this package's config is resolvable
• OrmLoaderConfig.test_orm_db split into test_orm_db_pg/test_orm_db_sqlite for dialect-specific test configurations
• CI split into separate SQLite and Postgres build-test jobs
• engine_with_replica_role now uses autocommit_connection()'s Engine branch directly
• Removed own Dialect system in favour of the unified oa-configurator one
• Adopts oa-configurator's new test infrastructure (isolated_test_database, DIALECT_PARAMS, db_dialect marker).
• New tests:
  -test_schema_translate_map.py: the actual regression test for the originally-reported
  bug,
  • test_split_connection.py, test_shared_backend.py: merge-method contract tests unified across both dialects,
  • test_reserved_schema.py: proves STAGING_SCHEMA registration is actually
    enforced cross-package

Schema drift protection, primitive consolidation

• helpers/bootstrap.py::create_db/bootstrap() is now wrapped in guard_schema_provenance_for via oa-configurator's open_connection to prevent schema drift
• helpers/sql.py::qualify_identifier deleted entirely
  • consolidated into oa-configurator's qualified()
• backends/base.py::_as_connection now delegates its Engine/Connection branch to oa-configurator's open_connection.
• Adopts physical_schema_of stack-wide
  • fixes validate_schema_tag usage in materialised_view_mixin.py and schema_if_supported usage in backends/sqlite.py.

Checklist

• Applied exactly one label (breaking, feature, fix, dependencies, or chore)
• Tests pass locally (uv run pytest -q): 247 passed, 1 xfailed
• Tests pass locally (uv run pytest -q -m postgresql): 41 passed
• Lint passes (uv run ruff check .)

[gkennos]

raise dependency base for oa-configurator

also depending how you resolve the comment that remains here, you may need to update oa-configurator --> sql.py as well (validate_schema_tag)

[nicoloesch]

Nothing changed here since the last review beyond a stale-docstring fix: the validate_schema_tag() mention in the body and in tests/models.py is corrected to match the current contract. Body updated.

[Copilot]

🟡 Changes recommended

The dependency remains incompatible with required upstream APIs, and unresolved routing, provenance, and non-terminating pagination issues remain.

6 open findings

• Update oa-configurator dependency and lockfile for required APIs · New
• Reject non-positive batch sizes in PostgreSQL merges · New
• Reject non-positive batch sizes in SQLite merges · New
• Document or preserve compatibility for changed helper APIs · New
• Enforce schema provenance guard for bindable connections · New
• Route insert-if-empty query using the session mapper · New

What changed in this PR

Reworks loading, merging, schema routing, and materialized-view operations to honor schema_translate_map across PostgreSQL and SQLite.

Changes:

• Replaces raw merge SQL with schema-aware SQLAlchemy Core operations.
• Adds split-connection, schema-translation, materialized-view, and backend integration tests.
• Adopts oa-configurator’s schema claims, dialect utilities, and test infrastructure.

File	Description
.github/​workflows/​ci.yml	Splits SQLite and PostgreSQL CI jobs.
pyproject.toml	Excludes dialect tests by default.
docs/​loaders/​context.md	Clarifies quote-mode behavior.
docs/​registry/​validation.md	Documents validator presets.
docs/​tables/​mat_view.md	Documents schema-tagged materialized views.
src/​orm_loader/​backends/​__init__.py	Exports schema-claim support.
src/​orm_loader/​backends/​base.py	Consolidates dialect and schema-aware backend behavior.
src/​orm_loader/​backends/​postgres.py	Rewrites PostgreSQL merges and view lifecycle handling.
src/​orm_loader/​backends/​resolve.py	Resolves backends and physical staging schemas.
src/​orm_loader/​backends/​sqlite.py	Rewrites SQLite merges using Core constructs.
src/​orm_loader/​config.py	Splits PostgreSQL and SQLite test configuration.
src/​orm_loader/​helpers/​__init__.py	Updates helper exports.
src/​orm_loader/​helpers/​bootstrap.py	Adds topology-aware database creation.
src/​orm_loader/​helpers/​bulk.py	Refines context-manager typing.
src/​orm_loader/​helpers/​sql.py	Removes local identifier qualification.
src/​orm_loader/​helpers/​sqlite.py	Adds mapper-aware FK diagnostics.
src/​orm_loader/​loaders/​data_classes.py	Renames staging schema state to a schema tag.
src/​orm_loader/​loaders/​loader_interface.py	Adds mapper-aware staging inserts.
src/​orm_loader/​loaders/​loading_helpers.py	Makes PostgreSQL COPY schema- and mapper-aware.
src/​orm_loader/​mappers/​__init__.py	Removes the obsolete dialect exception export.
src/​orm_loader/​mappers/​materialised_view_contracts.py	References shared qualification utilities.
src/​orm_loader/​mappers/​materialised_view_errors.py	Removes the specialized dialect error.
src/​orm_loader/​mappers/​materialised_view_mixin.py	Resolves view schemas from schema tags.
src/​orm_loader/​tables/​loadable_table.py	Threads mapper and schema-tag routing through loads.
src/​orm_loader/​tables/​typing.py	Updates loader protocol signatures.
tests/​backends/​test_postgres_backend.py	Expands real PostgreSQL backend coverage.
tests/​backends/​test_reserved_schema.py	Tests reserved staging-schema enforcement.
tests/​backends/​test_shared_backend.py	Adds cross-dialect merge contracts.
tests/​backends/​test_sqlite_backend.py	Uses registered computed-column models.
tests/​conftest.py	Adopts isolated dialect-specific fixtures.
tests/​helpers/​test_bootstrap.py	Tests topology-aware table creation.
tests/​loaders/​test_dedupe.py	Schema-tags the test model.
tests/​loaders/​test_loader_e2e.py	Adds SQLite routing and rollback coverage.
tests/​loaders/​test_parquet_loader.py	Schema-tags parquet test models.
tests/​loaders/​test_pg_loader.py	Migrates PostgreSQL tests to schema tags.
tests/​loaders/​test_schema_translate_map.py	Adds schema-translation regressions.
tests/​loaders/​test_split_connection.py	Tests separate primary/vocab connections.
tests/​mappers/​test_materialised_view_mixin.py	Adds role-aware view lifecycle tests.
tests/​models.py	Adds schema-tagged test models.
tests/​pg_db.py	Removes the legacy PostgreSQL fixture.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.