Repository navigation
Add raw property with unmapped source columns - #5
Merged
Merged
Conversation
…eature Each output feature gets properties.raw, a JSON object of the source row's columns, so downstream can map any field without further fork changes. Columns the conform maps directly (a column name, or a list of column names, for any SCHEMA key or lat/lon) are left out because their values are already output properties. Columns used only inside conform functions stay, with their original pre-function values. Columns starting with oa: and empty values are skipped. The ESRI cache now writes every field from the layer metadata instead of only the fields the conform names. Conform fields missing from the metadata are still written as empty columns and logged once. Claude-Session: https://claude.ai/code/session_01M9dZ4UWYaMwr9e9cwwKpyS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
ChopChop will map new source fields itself through a layer-level
extract, so the fork never needs another per-field change. The first user is UPRN/USRN for gb/countrywide (ENG-1529). For that, every output feature carries the source row's columns.What changed
openaddr/conform.py:row_transform_and_convertcopies the unmapped source columns with the newrow_raw_columnsbefore any conform function runs (row_fxn_chainwrites back into the row), and setsproperties.rawon the feature. Applies to every layer type.openaddr/cache.py: the ESRI cache writes every field from the layer metadata instead of only the fields the conform names. Conform fields missing from the metadata are still written as empty columns (so functions that index them keep working) and logged once.test_raw_properties; expected values updated whererawnow appears, and where ESRI row hashes and fingerprints change (see below).Exclusion rule
For every key in the layer's SCHEMA plus
latandlon: if the conform value is a string, that column is excluded; if it is a list, each column name in it is excluded. Columns referenced only inside function objects (regexp,format,map,join,chain,first_non_empty,constant, ...) stay inrawwith their original values. Columns whose name starts withoa:(any case) and empty values (None or '') are skipped. Names are verbatim and values are the strings the CSV round trip produced.Testing
python3 -m unittest openaddr.tests.conform: 83 tests, OK.python3 test.pyin Docker): 129 tests, OK (master: 128, OK).hashvalues and cache fingerprints in 5 existing tests changed because the ESRI CSV now holds all fields (hashcovers the whole row). Nothing downstream readshash.Rollout
ChopChop must ship its decorate strip of
rawand Combine file-streaming first. Then: amd64 build, pushregistry.geocod.io/geocodio/batch-machine:latest, anddocker pullon the proxy host.https://claude.ai/code/session_01M9dZ4UWYaMwr9e9cwwKpyS