Repository navigation
feat(progress): report machine-readable progress from every batch-machine step - #6
Merged
Merged
Conversation
…lines Adds openaddr.progress.Reporter, which writes throttled progress lines to stdout outside the logging module. OPENADDR_PROGRESS_INTERVAL sets the interval in seconds (default 15, 0 disables). Tests default it to 0.
… mapping steps Emits progress lines for the ArcGIS and URL downloads, unzipping, source extraction and column mapping, with H3 cell deltas for the ArcGIS and mapping phases. Extract steps now return the number of rows written so the mapping phase knows its total.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch machine now prints machine-readable progress lines while it downloads, unzips, reads and maps a source. ChopChop reads them from the container's stdout to show a progress bar, an ETA and a live map of which areas were processed and where rows failed. Turning progress off (
OPENADDR_PROGRESS_INTERVAL=0) leaves output exactly as before.Changes
openaddr/progress.py(new): aReporterthat prints{"progress":{...}}lines to stdout withprint(..., flush=True), outside the logging module. One line when a phase starts, throttled lines while it runs (OPENADDR_PROGRESS_INTERVAL, default 15 s,0disables), and one when it ends. Each line carriesphase,done,total(ornull),unit,counters, and H3 resolution 6 cell deltas as[ok, fallback, error].openaddr/cache.py:totalis theget_feature_count()value the code already fetched and only logged.totalfromContent-Lengthwhen the server sends it.openaddr/conform.py:setup.py: addsh3 >= 4, < 5. The Dockerfile already installs throughpip3 install ..openaddr/tests/progress.pycovers the line shape, the null total, throttling, cell deltas, the disabled interval and non-finite coordinates.openaddr/tests/__init__.pydefaults the interval to 0, so existing tests print nothing new.Decisions
phase_index/phase_countare left out, because the sequence of phases is not known up front.representative_point()for non-point geometry, so polygon features land inside their shape.Review notes
openaddr/progress.py, then the ArcGIS call site incache.py, which is where most real sources spend their time.donecounts written features andcounters.skippedcounts skipped ones, sodonecan end belowtotal.Follow-ups
h3wheel should be confirmed in the first amd64 build.