Skip to content

feat(progress): report machine-readable progress from every batch-machine step - #6

Merged
MiniCodeMonkey merged 2 commits into
masterfrom
feat/progress-lines
Oct 6, 2026
Merged

MiniCodeMonkey merged 2 commits into
masterfrom
feat/progress-lines

Conversation

@MiniCodeMonkey

Copy link
Copy Markdown
Member

Batch machine now prints machine-readable progress lines while it downloads, unzips, reads and maps a source. ChopChop reads them from the container's stdout to show a progress bar, an ETA and a live map of which areas were processed and where rows failed. Turning progress off (OPENADDR_PROGRESS_INTERVAL=0) leaves output exactly as before.


Changes

  • openaddr/progress.py (new): a Reporter that prints {"progress":{...}} lines to stdout with print(..., flush=True), outside the logging module. One line when a phase starts, throttled lines while it runs (OPENADDR_PROGRESS_INTERVAL, default 15 s, 0 disables), and one when it ends. Each line carries phase, done, total (or null), unit, counters, and H3 resolution 6 cell deltas as [ok, fallback, error].
  • openaddr/cache.py:
    • ArcGIS download: total is the get_feature_count() value the code already fetched and only logged.
    • URL download: counts bytes, with total from Content-Length when the server sends it.
  • openaddr/conform.py:
    • Unzip: counts the uncompressed bytes.
    • Reading the source: OGR feature count; for GeoJSON, bytes of the file size; for CSV, rows with no total.
    • Mapping columns: rows, with the extract step's row count as the total.
    • Extract functions now return the number of rows they wrote.
  • setup.py: adds h3 >= 4, < 5. The Dockerfile already installs through pip3 install ..
  • Tests:
    • New openaddr/tests/progress.py covers the line shape, the null total, throttling, cell deltas, the disabled interval and non-finite coordinates.
    • A new ArcGIS test checks the first and final lines.
    • openaddr/tests/__init__.py defaults the interval to 0, so existing tests print nothing new.

Decisions

  • Rate and ETA are not computed here. ChopChop computes them over a rolling window, so every tool only reports counts.
  • phase_index/phase_count are left out, because the sequence of phases is not known up front.
  • Map cells use representative_point() for non-point geometry, so polygon features land inside their shape.

Review notes

  • Start with openaddr/progress.py, then the ArcGIS call site in cache.py, which is where most real sources spend their time.
  • In the ArcGIS phase, done counts written features and counters.skipped counts skipped ones, so done can end below total.

Follow-ups

  • The image build was only run on arm64. The amd64 h3 wheel should be confirmed in the first amd64 build.

…lines

Adds openaddr.progress.Reporter, which writes throttled progress lines to
stdout outside the logging module. OPENADDR_PROGRESS_INTERVAL sets the
interval in seconds (default 15, 0 disables). Tests default it to 0.
… mapping steps

Emits progress lines for the ArcGIS and URL downloads, unzipping, source
extraction and column mapping, with H3 cell deltas for the ArcGIS and
mapping phases. Extract steps now return the number of rows written so the
mapping phase knows its total.
@MiniCodeMonkey
MiniCodeMonkey merged commit 96bb232 into master Oct 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant