Integrate Synthea with PyHealth

[heraclitus007]

Summary
Add first-class support for generating synthetic patient populations with Synthea and loading the resulting CSV data through PyHealth’s standard dataset API.

What this adds
This feature provides:

• A SyntheaGenerator wrapper around the Synthea CLI.
• Automatic use of the pinned Synthea v4.0.0 release with SHA-256 verification.
• Support for Synthea CLI arguments and validated synthea.properties overrides.
• Deterministic, fingerprinted output directories for different generation configurations.
• Lazy generation and reuse of existing output.
• SyntheaCSVDataset, which loads generated data through BaseDataset.
• Schema mappings for Synthea patient, encounter, clinical, payer-transition, and claims tables.
• Configuration discovery and validation against the active Synthea JAR.
• An end-to-end notebook demonstrating generation, loading, and schema comparison with the official Synthea source release.

Files added or modified

• pyhealth/datasets/synthea_generator.py: Synthea process management, configuration handling, JAR resolution, checksum verification, output fingerprinting, and lazy generation.
• pyhealth/datasets/synthea_csv.py: PyHealth dataset integration for generated Synthea CSV files.
• pyhealth/datasets/configs/synthea_csv.yaml: Table schemas, patient identifiers, timestamps, attributes, and preprocessing mappings.
• pyhealth/datasets/configs/synthea_release.yaml: Supported Synthea release metadata and exporter configuration.
• pyhealth/datasets/init.py: Public exports for SyntheaGenerator and SyntheaCSVDataset.
• tests/core/test_synthea_generator.py: Unit coverage for generator validation, CLI construction, configuration precedence, output reuse, lazy loading, table selection, and dataset behavior.
• examples/synthea_end_to_end.ipynb: Complete usage walkthrough and schema-equivalence demonstration.

Design decisions

• Dataset construction is lightweight; Synthea runs only when data is first loaded.
• Existing generated output is reused unless regenerate=True is specified.
• Generation parameters are included in the output fingerprint to keep different populations isolated.
• CSV export is managed by the wrapper. Other Synthea exporters are disabled for this integration.
• Thirteen clinical and demographic tables are enabled by default.
• Claims and claim transactions are supported but remain opt-in because of their size.
• Legacy procedure exports using date are normalized to the current start field.
• The integration validates user-provided configuration keys against the selected Synthea JAR.

Out of scope

• FHIR, JSON, CCDA, text, and other non-CSV Synthea exporters.
• Loading organizations, payers, and providers as patient event tables.
• Defining a default downstream prediction task or model.
• Installing or managing Java.
• Guaranteeing byte-for-byte reproducibility between the published Synthea JAR and independently compiled source builds.
• Vendoring Synthea source code or generated patient populations.

Validation

• Added 26 focused unit tests, including an opt-in live generation test.
• Added an end-to-end notebook covering the public API.
• Verified the new files compile successfully.
• Verified the new source and test files with Ruff.

[jhnwu3]

I like the idea of a Synthea dataset here, but it feels to me we need a SyntheaGenerator(BaseModel) instead here.

[heraclitus007]

@jhnwu3 Thanks for the review. I've restructured along those lines:

Generator is now a model: pyhealth.models.Synthea lives in pyhealth/models/generators/ next to HALO/MedGAN and subclasses BaseModel. Since Synthea is a rule-based Java simulator, it has no trainable parameters; forward() raises with a pointer to run() / ensure_generated(), which write CSVs to disk.

Dataset stays a dataset: pyhealth.datasets.SyntheaCSVDataset(root) loads any Synthea CSV export and no longer depends on the model.

Java: Java 17+ is a documented prerequisite, and the version is checked up front with a clear error.

Housekeeping: API docs added for both classes; generated artifacts removed from the PR.

[chufangao]

Thank you for this!

The synthea dataset format is a bit different from the existing dataset formats, would it be possible to write some more quick docs / examples to emphasize the differences so that people don't get confused?

I think the existing dataset format is like a dataframe with columns: patient_id, time, code. SYNTHEA looks like it generates a lot more than that. Could be potentially confusing if documentation isn't clear,

[heraclitus007]

Thanks for the feedback, everyone. I've pushed an update that addresses all three points.

1. Setup vs. generation (constructor / generate())

You're right that ensure_generated() was confusing, and that Synthea() looked like it generated data (it didn't). The constructor now only sets up the generator, and population settings go on generate(), so one instance can produce as many populations as you want:

synthea = Synthea("./synthea-output")                     # setup only, nothing runs
small = synthea.generate(population=10, seed=1)            # returns the CSV directory
large = synthea.generate(population=500, seed=2, state="Ohio")
dataset = SyntheaCSVDataset(small)

• run(), ensure_generated() and with_config() are removed.
• Calling generate() again with the same settings returns the existing output instead of rerunning Synthea; overwrite=True forces a fresh run.
• If seed is omitted, a random seed is chosen and logged, so each call gives a new population.

@john, on your question: Synthea is a stateless command-line program. Each generate() call runs java -jar synthea.jar -p 10 -s 42 ..., which writes CSVs and exits. There's no server and no state kept between runs, so it is now effectively one function. The settings were only on the constructor for historical reasons.

2. Setup docs

The pyhealth.models.Synthea API page now has a Setup section. Users don't need to download Synthea; the pinned JAR is downloaded, checksum-verified and cached on first use. The only manual step is installing Java 17+, and the section gives install commands for conda, Homebrew, apt and Temurin. It also covers offline use with jar_path=.

3. How Synthea data differs (@andy)

The SyntheaCSVDataset page has a new "How Synthea Differs From Other PyHealth Datasets" section:

• Synthea's raw output is about 18 wide CSV tables, but after loading it's the same event table as every other PyHealth dataset (patient_id, event_type, timestamp plus attributes).
• The main difference is the codes: SNOMED-CT, RxNorm, LOINC and CVX instead of ICD/NDC. That means MIMIC tasks and ICD code mappings don't apply directly.
• It also lists which tables load and other gotchas (birth-date patient events, visits keyed by encounter, string lab values).

The end-to-end notebook has been rebuilt for the new API and re-run.