Repository navigation
ENH: Decode NEXRAD Level 2 message types 32 and 33 - #4147
thaliamontreux wants to merge 6 commits into
Conversation
Message 32 (RDA PRF Data, added in RDA Build 23 per ICD 2620002 Table XVIII) carries the pulse repetition frequency in use for each waveform type (CS/CD/SPP) in units of 0.001 Hz, and is decoded into Level2File.prf_data. Message 33 carries named RDA log files, optionally gzip/bzip2 compressed, and is decoded into Level2File.rda_log. Both messages previously emitted 'Unknown message' warnings on every Build 23+ volume.
|
|
Newer matplotlib on the conda macOS runners trips this RuntimeWarning during the image-test sweep; with filterwarnings=['error'] it fails ~176 unrelated tests on every open PR (see Unidata#4148).
dopplershift
left a comment
dopplershift
left a comment
There was a problem hiding this comment.
Do you have a volume (or site ID/datetime) that generated the warnings originally? My preference for the test would be to add a new volume to staticdata and test that the new fields are properly parsed without warnings (as in done in the other tests). That way we're testing authentic data rather than a synthesized binary stream.
e07898a to
f010567
Compare
TOML literal strings do not process escapes, so '\d' was a regex matching a literal backslash rather than digits -- the filter never matched 'More than 20 figures'. Newer shapely also emits 'invalid value encountered in create_collection' inside cartopy on the macOS conda runners.
f010567 to
af6106d
Compare
Adds KLSX20261008_235320_V06 to staticdata and exercises it through the standard level2 file test plus a dedicated assertion on prf_data, per review feedback preferring authentic data over synthesized streams.
|
Good call — added It now runs through the standard For message 33: it's the event-driven RDA log dump and doesn't appear in routine volumes — we scanned current volumes across many sites plus all the existing Level 2 files in |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
rmz-oz
left a comment
rmz-oz
left a comment
There was a problem hiding this comment.
a gzip payload with a bad deflate block raises zlib.error, which isn't an OSError, so one corrupt msg 33 makes Level2File fail on the whole volume. on main and 1.7.1 the same file still opens with just the Unknown message: 33 warning.
| # Compression type 3 is a ZIP container, which needs a member | ||
| # index to unpack -- keep the raw payload in that case. | ||
| text = data.decode('utf-8', 'replace') | ||
| except (OSError, EOFError): |
There was a problem hiding this comment.
this misses zlib.error (e.g. "invalid block type" from gzip.decompress), probably worth adding with an import zlib. separately, msg 33 reads its payload directly instead of going through _buffer_segment like 13/15/18, so if it ever comes in more than one segment the second segment gets parsed as a new header. I only checked that with a synthetic two segment file though, not a real volume.
Address review feedback on Unidata#4147: - Route both decoders through _buffer_segment like the other bulk messages (13/15/18) so a message split across segments is reassembled before parsing instead of a second segment being parsed as a new header. Adds a two-segment message 33 test. - Catch zlib.error alongside OSError/EOFError when decompressing message 33 payloads -- gzip.decompress raises it for corrupt deflate streams ("invalid block type"). - Move gzip/bz2 imports to module level; read the compressed payload via the header's compressed_size rather than blindly consuming the buffer. - Wrap the real-volume test's Level2File in a context manager so the test file is closed (CodeQL file-not-closed finding).
| def test_level2_msg32_real(caplog): | ||
| """Check that message 32 is parsed without warnings from a real Build 23+ volume.""" | ||
| caplog.set_level(logging.WARNING, 'metpy.io.nexrad') | ||
| with open(get_test_data('KLSX20261008_235320_V06', as_file_obj=False), |
Follow-up audit of the new decoders: - Handle the ICD 2620002P size_hw==65535 encoding, where the segment fields carry the total message size in bytes. Previously a large message 33 would have sat in the segment buffer forever, its header fields misread as segment numbers. - Guard both decoders against truncated bodies -- a short read (end of a damaged or partial volume) now logs a warning instead of raising struct.error mid-file. - Merge rather than replace message 32 PRF tables so a second message in the same volume cannot silently drop earlier waveform data. Adds tests for all three cases.
This branch has not been deployed
5 participants
Description Of Changes
RDA Build 23+ volume scans emit two message types that MetPy currently reports as
Unknown message: 32 and 33.Message 32 — RDA PRF Data (ICD 2620002 Table XVIII, new in RDA Build 23): carries the pulse repetition frequencies in use for each waveform type — Contiguous Surveillance (CS), Contiguous Doppler (CD), and Staggered Pulse Pair (SPP) — as unsigned 32-bit values scaled by 0.001 Hz. This is the ground truth for each cut's PRF, and thus its Nyquist velocity and unambiguous range. Decoded into
Level2File.prf_dataas{waveform_name: [prf_hz, ...]}; unrecognized waveform codes are kept under their numeric value. Validated against real Build 23+ volumes (e.g. KLSX: CS 321.9–862.1 Hz, CD 446.4–1282.1 Hz, SPP 574.3–1135.6 Hz).Message 33 — RDA Log Data: a named RDA diagnostic log file (e.g.
AzServoLog) with a 68-byte header (version, 26-byte identifier, data version, compression type, compressed/decompressed sizes), where the payload may be uncompressed, gzip, bzip2, or zip-compressed. Gzip/bzip2 payloads are decompressed; zip payloads are kept raw (container format needs a member index). Entries are appended toLevel2File.rda_logas dicts withidentifier,version,data_version,compression, and decodedtext. Message 33 is sent on demand rather than in every volume, so it is rare in practice — but it no longer warns.Since these messages now have decoders, the
Unknown message: 32warnings that fire hundreds of times per Build 23+ volume are gone.Checklist