Skip to content

ES: convert province by province, then merge - #338

Open
ivorbosloper wants to merge 3 commits into
mainfrom
es-per-file
Open

ivorbosloper wants to merge 3 commits into
mainfrom
es-per-file

Conversation

@ivorbosloper

Copy link
Copy Markdown
Collaborator

es now converts each of its 50 provincial GeoPackages on its own and merges the parts, because the whole of Spain does not fit in memory in one run. Closes #336.

  • PerFileConverterMixin (fiboa_cli/conversion/per_file.py) runs the normal convert once per source file into a temporary part, then merges the parts with vecorel-cli's DuckDBBaseConverter.merge_parquet, which checks the ids and Hilbert-sorts over the whole set. With a single source file it is plain convert. It replaces the 450-line PerFileBaseConverter of PerFileBaseConverter: convert file by file, then merge #281.
  • The merge runs on a standalone DuckDBBaseConverter rather than mixing that class in, so the converter's own convert chain stays as it is.
  • Full 2025 edition (17.8 million rows): peak RSS 10 GB, against more than 27 GB before the one-run attempt was stopped; the output validates.
  • Not in this PR: in several provinces about 0.1% of the rows repeat an id (the SIGPAC key plus the declaration line). They get the ~n suffix, as before.

🤖 Generated with Claude Code

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ES: convert all of Spain in one run, or per file

2 participants