Skip to content

Remove the deprecated nwis module - #457

Draft
thodson-usgs wants to merge 3 commits into
DOI-USGS:mainfrom
thodson-usgs:remove-nwis
Draft

thodson-usgs wants to merge 3 commits into
DOI-USGS:mainfrom
thodson-usgs:remove-nwis

Conversation

@thodson-usgs

Copy link
Copy Markdown
Collaborator

Summary

This removes dataretrieval.nwis before its published 2027-05-06 horizon, as part of the next major release. USGS is decommissioning the waterservices endpoints the module called, and every getter in it already has a waterdata or nwdc replacement.

Breaking:

  • import dataretrieval.nwis now raises ModuleNotFoundError.
  • NoSitesError is removed, both dataretrieval.NoSitesError and exceptions.NoSitesError. Only the waterservices path raised it. The modern getters return an empty DataFrame when nothing matches.

Code

  • Deleted nwis.py, tests/nwis_test.py, tests/waterservices_test.py, and 9 fixtures used only by those tests.
  • Deleted REMOVALS["nwis"], the "No sites/data" body check in _querying, and _validation.require_any_of, whose only caller was nwis.
  • .importlinter: dropped nwis from the layers contract and deleted the nwis-quarantine contract. Also updated the pyproject comment, the pre-commit comments, and the pyscn clone baseline.
  • Rewrote the generic tests that used nwis only to reach shared code (utils.query, the deprecation mechanism) so they call that code directly.
  • rdb.read_rdb's HTML-response message used to tell callers to migrate to waterdata. Only waterdata.get_ratings reaches it now, so the message asks them to report the moved endpoint instead.

Notebooks and demos

  • hydroshare/USGS_NWIS_WaterUse_Examples.ipynb was defunct and couldn't run. It is rewritten on nwdc.get_wateruse and renamed USGS_NWDC_WaterUse_Examples.ipynb.
  • nwqn_data_pull scripts now use waterdata.get_monitoring_locations and get_daily. Approval filtering happens on the server (approval_status="Approved") instead of matching the old NWIS _cd strings.
  • Copy-edited the notebooks that still called nwis current or said waterdata "will replace" it, and removed the unused nwis import from the R-vignette notebook.
  • Added nblinks for the two notebooks that weren't in the docs: USGS_WaterUse_Examples and hydroshare/USGS_NLDI_Examples. Updated examples/index.rst.

Prose

  • Updated README, NEWS (new 10/22/2026 entry with a migration map), CONTEXT (NWIS entry now marked retired), AGENTS, CONTRIBUTING, the architecture overview, and userguide/dataportals.rst.
  • ADR 0005 is marked completed. Where an ADR's Compliance section named the deleted contract or tests, I added a short dated note and left the historical reasoning alone.

Other deprecations evaluated

None of the other published horizons is earlier than May 2027, so they all stay. They are listed in the NEWS entry:

Surface Horizon
get_time_series_metadata v0 filters 2027-06-01 (set by the service's v0 retirement)
get_cql(service=) 2027-08-09
wateruse alias 2027-08-11
ogc.interruptions 2027-08-25

The camelCase get_samples kwargs have no horizon published.

Remaining "NWIS" mentions (intentional)

  • Domain vocabulary: WQP providers=NWIS, NLDI nwissite/nwisgw, NGWMN organization_type="NWIS", the USGS NWIS time-zone code list, and docstrings saying site numbers are assigned by NWIS.
  • R readNWIS* calls quoted in the R-vignette notebook comments.
  • Historical record: earlier NEWS entries, ADR narrative, and the CONTEXT glossary entry.

Testing

  • ruff check, ruff format --check, lint-imports (7 kept), xenon, and complexipy all pass, as do the pre-commit hooks.
  • pytest: 1650 passed. Branch coverage is 99.07%, above fail_under 98.9.
  • mypy --python-version 3.12 is clean. Locally, the default 3.10 target fails only inside numpy's installed stub (type statement), which is unrelated to this change. CI should confirm.
  • make html succeeds and the executed notebooks render. The only cell errors are NetworkErrors on WQP calls in rvignettes and DiscreteSamples, caused by a local SSL problem that also reproduces with a bare httpx.get.
  • Ran against live services: the NWDC water-use notebook, WaterData_demo, Introduction, and peak_streamflow_trends. Ran retrieve_nwqn_streamflow.map_retrieval on one site with lithops and S3 stubbed out.
  • Not verified end to end: the full nwqn lithops/S3 pipeline.

Remove dataretrieval.nwis ahead of its 2027-05-06 horizon. USGS is
decommissioning the waterservices endpoints it called, and every getter
has a waterdata (or nwdc) replacement.

- Delete dataretrieval/nwis.py, its tests and fixtures, and its
  reference page.
- Remove NoSitesError (only the waterservices path raised it) and the
  "No sites/data" body check in _querying; remove REMOVALS["nwis"] and
  the now-unused _validation.require_any_of.
- Drop nwis from .importlinter (layers + nwis-quarantine contract),
  pyproject, pre-commit comments, and the pyscn clone baseline.
- Port demos: the defunct hydroshare water-use notebook is rewritten on
  nwdc.get_wateruse (USGS_NWDC_WaterUse_Examples); the nwqn_data_pull
  scripts use waterdata.get_monitoring_locations / get_daily; copy-edit
  notebooks that still described nwis as current.
- Docs: add missing nblinks for USGS_WaterUse_Examples and
  USGS_NLDI_Examples; update README, NEWS, CONTEXT, AGENTS,
  CONTRIBUTING, the architecture overview, and mark ADR 0005 completed.

BREAKING CHANGE: `import dataretrieval.nwis` raises ModuleNotFoundError,
and dataretrieval.NoSitesError / exceptions.NoSitesError are removed.
Prose only (Markdown, notebook Markdown cells, docstrings, and comments);
no identifiers, runtime strings, or doctest blocks changed.

- 'while' for contrast -> whereas/although (STA8 p. 60); 'since' for
  cause -> because (p. 45); 'data' plural (p. 48); 'and/or' -> 'or
  both' (p. 44)
- Hyphenate unit modifiers: water-quality, surface-water (p. 28)
- Numbers: thousands separators, 'from X to Y' ranges, words for
  isolated small counts (pp. 145-146)
- State names spelled out at first use and when standing alone (p. 127);
  'State' capitalized for a U.S. State and 'U.S.' as an adjective (GPO)
- Fix typos in notebooks and docstrings
Prose only; no identifiers, runtime strings, or doctest blocks changed.

- Capitalize 'State' consistently for a U.S. State, including
  attributive uses ('State code', 'State FIPS'); lowercase for Mexican
  states and Canadian provinces
- More 'since'/'while' -> because/whereas (STA8 pp. 45, 60); 'data'
  plural (p. 48)
- Commas: before 'whereas' and nonrestrictive 'which' (pp. 22, 35),
  serial comma (p. 26), after the year in a full date (p. 145)
- Related numbers in figures (p. 146); 'that is' for i.e. in running
  text (p. 125); water-quality unit modifier (p. 28)
- American spelling (behavior, judgment, neighboring), 'gage', Water
  Quality Portal capitalization, and a garbled sentence in
  configuration_test

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant