Skip to content

Add DuckDB backend - #42

Open
nicoloesch wants to merge 7 commits into
mainfrom
4-duckdb-support
Open

nicoloesch wants to merge 7 commits into
mainfrom
4-duckdb-support

Conversation

@nicoloesch

@nicoloesch nicoloesch commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Changes

  • Adds a DuckDB backend through a new duckdb extra, with staging tables and batched replace, upsert and insert merges.
  • Backends declare foreign keys they can't enforce through deferred_fk_constraints().
  • load_csv(validate_deferred_fks=True) checks those keys with an anti-join after each merge.
  • The new check does nothing on PostgreSQL and SQLite, which have no deferred keys.
  • On DuckDB, replace updates rows in place, because DuckDB rejects deleting a referenced row.
  • check_replace_feasible() lets callers reject an impossible replace before loading anything.
  • replace with index_strategy="keep" is rejected up front on backends that can't support it.
  • Pagination lives in one shared _paginate_batches helper instead of a copy per backend.
  • Each backend now declares whether it commits between batches.
  • load_csv always drops its staging table, including when the merge fails.
  • Depends on the DuckDB dialect in , so CI fails until that is released.

Checklist

  • Applied exactly one label (breaking, feature, fix, dependencies, or chore)
  • Tests pass locally (uv run pytest -q)
  • Lint passes (uv run ruff check .)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New backwards-compatible functionality. MINOR: x.y+1.z

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feasibility review for addition of duckdb backend

1 participant