Repository navigation
Conversation
Introduce PTBXLDataset (BaseDataset + YAML) with optional wfdb extra, root-keyed metadata cache, and a resolved cache YAML so file_path is correct before BaseDataset init. Co-authored-by: Cursor <[email protected]>
Add PTBXLSuperclassClassification and split_by_strat_fold (folds 1-8 / 9 / 10) in separate modules so they can move to benchmarks later. Co-authored-by: Cursor <[email protected]>
Cover helpers, metadata cache keyed by root, event reads via PTBXLDataset, waveform IO (wfdb), and the 5-superclass task path. Co-authored-by: Cursor <[email protected]>
Document the dataset/task Overview RST pages and a minimal ECG superclass example for CPBench users. Co-authored-by: Cursor <[email protected]>
…cally BaseDataset keys global_event_df on root+dataset_name; hash data-root path and CSV bytes into dataset_name (EEGBCI pattern) and write CSV/YAML via tmp+os.replace under FileLock so DDP cannot read a truncated file. Co-authored-by: Cursor <[email protected]>
Move the SCP class map to functools.cache so vars(task) is unchanged after __call__. Emit age=-1 when missing and clip HIPAA-censored 300 to 90 so litdata can serialize the field. Co-authored-by: Cursor <[email protected]>
Accept a precomputed folds sequence and a generic fold_field so callers need not materialize 12-lead signals just to read an integer. Error messages no longer mention PTB-XL. Co-authored-by: Cursor <[email protected]>
Exercise the real litdata path without mocking wfdb, assert MultiLabelProcessor vocab order, and pin cache-dir uniqueness plus vars(task) stability. Co-authored-by: Cursor <[email protected]>
v1.0.3 is 21,799 records / 18,869 patients. Document that MultiLabelProcessor emits CD, HYP, MI, NORM, STTC. Harmonize the datasets toctree and reuse the ptbxl extra in the pixi test env. Co-authored-by: Cursor <[email protected]>
The redundant-looking 'as' is required by ruff's unused-import rule for module re-exports, same as EEGBCIDataset / PTBXLDataset in this file. Co-authored-by: Cursor <[email protected]>
BaseDataset.load_table concatenates root/file_path and rejects absolute paths, so override load_table to read the derived CSV from metadata_cache_dir while leaving self.root equal to the constructor path. Co-authored-by: Cursor <[email protected]>
Official PTB-XL folds are patient-disjoint; this flag verifies that property via patient_to_index. Default False keeps current callers working. The synthetic fixture leaks patient 15709 across folds 1 and 9 so the check is actually exercised. Co-authored-by: Cursor <[email protected]>
AxelNoun
force-pushed
the
feat/ptbxl-dataset
branch
from
August 18, 2026 22:57
47aa3a4 to
fe1007f
Compare
…ataset BaseDataset._init_cache_dir now also hashes the resolved config and fingerprints the table sources. The source fingerprint resolves the derived CSV against the user data root, where it does not live, so it is empty for PTBXLDataset; say so, and say what the cache identity rests on instead.
Contributor
Author
|
@jhnwu3 This PR (PTB-XL dataset and 5-superclass task) has not been reviewed yet. I merged current One question following #1264: with Could you review this PR, or indicate who should? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contributor: AxelNoun (GitHub: @AxelNoun), external contributor (no NetID)
Contribution Type: New Dataset + Task
Description: Adds PTB-XL (12-lead ECG, PhysioNet v1.0.3, open access) as a
PyHealth dataset, with a 5-superclass multi-label diagnosis task and a
split helper using the official stratified folds. Motivated by the CPBench
effort: it gives conformal prediction a multi-label ECG benchmark that needs
no credentialing, with metadata suited to covariate/label-shift splits.
Based on Wagner et al., Scientific Data 2020
(https://www.nature.com/articles/s41597-020-0495-6).
Relation to
CardiologyDataset:masterlistsptb-xlamong the sixsubsets of
CardiologyDataset, which reads the PhysioNet/CinC Challenge 2020copy (
.mat/.heafiles underptb-xl/g*/, one patient id perg*folder)for the binary
cardiology_is*_fntasks. It subclasses the 1.xBaseSignalDataset, now a deprecation stub, so an instance on currentmasterhas noset_taskorstat.PTBXLDatasetreads the PTB-XL v1.0.3release itself on the
BaseDataset/YAML API and exposes what that path doesnot: PTB-XL patient ids,
strat_fold, SCP statements with likelihoods,site/device, and a choice of 100 or 500 Hz records. (The
cardiology_is*_fntasks read only age, sex and Dx codes from the
.heaheaders and assume500 Hz.)
Output label order:
MultiLabelProcessor.fitsorts observed labelsalphabetically. On the full v1.0.3 corpus the multi-hot vector is
CD, HYP, MI, NORM, STTC. The task does not pin a custom vocabulary.Files to Review:
pyhealth/datasets/ptbxl.py: dataset implementationpyhealth/datasets/configs/ptbxl.yaml: table config (v1.0.3 template)pyhealth/tasks/ptbxl.py: 5-superclass multi-label taskpyhealth/datasets/splitter.py:split_by_strat_foldhelpertests/core/test_ptbxl.py: tests with synthetic fixturesdocs/api/datasets/pyhealth.datasets.PTBXLDataset.rst,docs/api/tasks/pyhealth.tasks.ptbxl.rstexamples/ecg/ptbxl/ptbxl_superclass_quickstart.pyImplementation notes:
wfdbis an optional extra (pip install 'pyhealth[ptbxl]'), lazily imported.PTBXLDataset.rootis the user-provided data path. Derived metadata CSVslive under
metadata_cache_dir(notroot) so read-only mounts stayuntouched.
BaseDataset.load_tableconcatenatesroot / file_pathandcannot take an absolute path, so
load_tableis overridden to scan thecache directory. Cache identity hashes the data-root path and the derived
CSV bytes into
dataset_name(EEGBCI pattern) so two roots or an in-placeptbxl_database.csvreplace cannot shareglobal_event_df.BaseDatasetnow also hashes the resolved configand fingerprints table sources, but it resolves
file_pathagainstroot,where the derived CSV does not live. On the test fixture the fingerprint is
empty (
key["sources"] == {"records": [[]]}), so identity rests on theconfig hash (the derived filename encodes the root and
ptbxl_database.csvhashes) and ondataset_name. The code comment inptbxl.pynow says this.PYHEALTH_REQUIRE_CACHE_DIR(Warn about the default dataset cache; allow requiring cache_dir #1264): the derived CSV and the resolvedYAML are written before
super().__init__(). With the variable set and nocache_dir, they are written tometadata_cache_dir(default~/.cache/pyhealth/datasets/ptbxl/) and thenBaseDatasetraises(observed on the test fixture). Not changed in this PR.
BaseDatasetonly acceptsconfig_path, so PTB-XLwrites a rate/root/source-keyed YAML next to the derived CSV before
super().__init__(). CSV and YAML writes use tmp+os.replaceunderfilelock.FileLock.# noqa: E402on the EEGBCI import line inpyhealth/datasets/__init__.py: left untouched (outside the diff).(n_leads, n_samples), matching EEGBCI's convention.ageis always an int: a missing age becomes-1(
age_is_missing=True), and the HIPAA-censored raw value 300 is clipped to90(age_is_censored=True). The raw 300 remains in the derived metadataCSV.
mainly pacemaker ECGs) are dropped by default via
drop_empty_labels.split_by_strat_fold(..., folds=)avoids decoding 12-lead signals just toread an integer.
check_patient_disjoint=Trueraises if apatient_idappears in more than one split (official PTB-XL folds are patient-disjoint;
the synthetic fixture leaks patient 15709 across folds 1 and 9 to exercise
this).
test-resources/core/ptbxl/is synthetic CSVs; WFDBwaveforms are generated at test time (not committed).
Testing / CI:
masterate94ca1d(includes Truncate nested inputs longer than the fitted width #1263 to Key the dataset cache on the config and source files #1265 andFit processors on the training split in set_task (split=PatientSplit) #1273), head
f4d5d07(Windows, Python 3.13.5,
wfdb4.3.1), each file run withpython -m unittest discover -t tests -s tests/core -p '<file>' -v:test_ptbxl.py28 passed,test_dataset_cache_key.py4 passed,test_default_cache_dir.py5 passed.python tools/check_pr_rules.py --base e94ca1d --head HEADpassed(ruff 0.16.10).
wfdb/pyhealth[ptbxl], waveform I/O andset_taske2e tests skip.pyhealth = { extras = ["ptbxl"] }somake testallunder pixi covers waveform I/O.Follow-up: 71-SCP-code classification and age regression are anticipated
in the structure but not implemented here; they are planned as a separate PR.
Test plan
python -m unittest discover -t tests -s tests/core -p 'test_ptbxl.py' -v: 28 passed (withwfdb), onf4d5d07test_dataset_cache_key.py(4 passed) andtest_default_cache_dir.py(5 passed), onf4d5d07python tools/check_pr_rules.py --base e94ca1d --head HEAD: passedmake testall/ contribution-rules on the PR: green onfe1007f(18 Aug)make testall/ contribution-rules onf4d5d07(after themastermerge)Recreates #1186, which was closed on 12 Aug when the fork hosting its branch
was detached during an account cleanup on my side. This was my error, and I
apologise for the noise. #1186 received no review; its only comment was my note
on the pre-existing
# noqa: E402, repeated above.