Repository navigation
Conversation
Refits the calibration intercept, and optionally the slope, of a trained binary or multilabel model on the logit scale. This is a common way to update a clinical prediction model for a new setting with few local events (Steyerberg et al. 2004, Vergouwe et al. 2017). TemperatureScaling fixes the intercept at 0, so it cannot fit an additive shift of the log-odds. Labels that take a single value on the calibration set are left unchanged with a warning, perfect separation under intercept_slope gives a warning, and non-finite parameters raise an error. Includes unit tests, a synthetic two-site example, and API docs. Closes sunlabuiuc#1250
Collaborator
|
Running workflows, don't hesitate to shoot us a visit in the Discord |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refits the calibration intercept, and optionally the slope, of a trained binary or multilabel model on the logit scale. This is a common way to update a clinical prediction model for a new setting with few local events (Steyerberg et al. 2004, Vergouwe et al. 2017). TemperatureScaling fixes the intercept at 0, so it cannot fit an additive shift of the log-odds.
Labels that take a single value on the calibration set are left unchanged with a warning, perfect separation under intercept_slope gives a warning, and non-finite parameters raise an error. Includes unit tests, a synthetic two-site example, and API docs.
Closes #1250
When a clinical prediction model moves to a new hospital, it often still ranks patients well but predicts the wrong event rate. This PR adds
pyhealth.calib.calibration.LogisticRecalibration, which updates a trained binary or multilabel model's logit tointercept + slope * logit, with both fitted by maximum likelihood on a calibration set. None of the existing calibrators fits an intercept, and for binary tasks TemperatureScaling is the slope-only case with the intercept fixed at 0.Interface
method="intercept"fits the intercept with the slope fixed at 1.method="intercept_slope"fits both.TemperatureScalinginterface,calibrate(cal_dataset=...)and thenforward. Happy to change the name or interface if you prefer something else.Edge cases
intercept_slope, a label whose positives and negatives are perfectly separated on the logit gets a warning, since its slope has no finite estimate.Relation to #1268
#1268 adds
calibration_slopeandcalibration_intercepttobinary_metrics_fn, fitted on the same modely ~ a + b * logit, and its intercept definition (slope fixed at 1) matchesmethod="intercept"here. The two fit well together, one measures miscalibration and the other corrects it. Once both are in, they could share one fitting function, and I am glad to make that change in whichever PR lands second.Changes
pyhealth/calib/calibration/logistic_recalibration.py, the new calibrator.pyhealth/calib/calibration/__init__.py, export.tests/core/test_logistic_recalibration.py, 12 tests. They cover recovering known intercept and slope values from synthetic miscalibrated logits, leaving calibrated logits alone,forwardreturningintercept + slope * logitwith matching probabilities and loss, evaluation throughTrainer, multilabel fitting with astate_dictround trip, the edge cases above, no false warning for a large real shift or a large real slope, and errors for multiclass and unknown methods.examples/logistic_recalibration_new_site.py, trains on a synthetic site and recalibrates on 150 patients from a second site with a higher event rate. On the second site's test set, ECE drops from about 0.30 to about 0.17 with the intercept and about 0.15 with intercept and slope, while AUROC stays at 0.79. Values move by a few hundredths between runs.docs/api/calib.rstanddocs/api/calib/pyhealth.calib.calibration.rst.