Skip to content

Add LogisticRecalibration for intercept and slope updating - #1272

Open
yfeng3212 wants to merge 1 commit into
sunlabuiuc:masterfrom
yfeng3212:logistic-recalibration
Open

yfeng3212 wants to merge 1 commit into
sunlabuiuc:masterfrom
yfeng3212:logistic-recalibration

Conversation

@yfeng3212

Copy link
Copy Markdown

Refits the calibration intercept, and optionally the slope, of a trained binary or multilabel model on the logit scale. This is a common way to update a clinical prediction model for a new setting with few local events (Steyerberg et al. 2004, Vergouwe et al. 2017). TemperatureScaling fixes the intercept at 0, so it cannot fit an additive shift of the log-odds.

Labels that take a single value on the calibration set are left unchanged with a warning, perfect separation under intercept_slope gives a warning, and non-finite parameters raise an error. Includes unit tests, a synthetic two-site example, and API docs.

Closes #1250

When a clinical prediction model moves to a new hospital, it often still ranks patients well but predicts the wrong event rate. This PR adds pyhealth.calib.calibration.LogisticRecalibration, which updates a trained binary or multilabel model's logit to intercept + slope * logit, with both fitted by maximum likelihood on a calibration set. None of the existing calibrators fits an intercept, and for binary tasks TemperatureScaling is the slope-only case with the intercept fixed at 0.

Interface

  • method="intercept" fits the intercept with the slope fixed at 1. method="intercept_slope" fits both.
  • Binary and multilabel tasks, with one intercept and slope per label. Multiclass raises an error.
  • It follows the TemperatureScaling interface, calibrate(cal_dataset=...) and then forward. Happy to change the name or interface if you prefer something else.

Edge cases

  • A label that takes a single value on the calibration set is left at intercept 0 and slope 1 with a warning, which matters for multilabel tasks with many rare labels. If every label is like this, it raises an error.
  • With intercept_slope, a label whose positives and negatives are perfectly separated on the logit gets a warning, since its slope has no finite estimate.
  • Non-finite fitted parameters raise an error.

Relation to #1268

#1268 adds calibration_slope and calibration_intercept to binary_metrics_fn, fitted on the same model y ~ a + b * logit, and its intercept definition (slope fixed at 1) matches method="intercept" here. The two fit well together, one measures miscalibration and the other corrects it. Once both are in, they could share one fitting function, and I am glad to make that change in whichever PR lands second.

Changes

  • pyhealth/calib/calibration/logistic_recalibration.py, the new calibrator.
  • pyhealth/calib/calibration/__init__.py, export.
  • tests/core/test_logistic_recalibration.py, 12 tests. They cover recovering known intercept and slope values from synthetic miscalibrated logits, leaving calibrated logits alone, forward returning intercept + slope * logit with matching probabilities and loss, evaluation through Trainer, multilabel fitting with a state_dict round trip, the edge cases above, no false warning for a large real shift or a large real slope, and errors for multiclass and unknown methods.
  • examples/logistic_recalibration_new_site.py, trains on a synthetic site and recalibrates on 150 patients from a second site with a higher event rate. On the second site's test set, ECE drops from about 0.30 to about 0.17 with the intercept and about 0.15 with intercept and slope, while AUROC stays at 0.79. Values move by a few hundredths between runs.
  • docs/api/calib.rst and docs/api/calib/pyhealth.calib.calibration.rst.

Refits the calibration intercept, and optionally the slope, of a trained
binary or multilabel model on the logit scale. This is a common way to
update a clinical prediction model for a new setting with few local events
(Steyerberg et al. 2004, Vergouwe et al. 2017). TemperatureScaling fixes
the intercept at 0, so it cannot fit an additive shift of the log-odds.

Labels that take a single value on the calibration set are left unchanged
with a warning, perfect separation under intercept_slope gives a warning,
and non-finite parameters raise an error. Includes unit tests, a synthetic
two-site example, and API docs.

Closes sunlabuiuc#1250
@jhnwu3

jhnwu3 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Running workflows, don't hesitate to shoot us a visit in the Discord

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: logistic recalibration (intercept and slope updating)

2 participants