Making Conformal Predictors Robust in Healthcare Settings: a Case Study on EEG Classification

TL;DR

This paper evaluates conformal prediction methods for EEG seizure classification in healthcare, addressing distribution shift challenges and demonstrating personalized calibration strategies that significantly improve coverage.

Contribution

It introduces personalized calibration strategies for conformal predictors that enhance coverage in healthcare settings with distribution shifts.

Findings

Personalized calibration improves coverage by over 20 percentage points.

Conformal prediction maintains comparable prediction set sizes.

Implementation available in open-source PyHealth framework.

Abstract

Quantifying uncertainty in clinical predictions is critical for high-stakes diagnosis tasks. Conformal prediction offers a principled approach by providing prediction sets with theoretical coverage guarantees. However, in practice, patient distribution shifts violate the i.i.d. assumptions underlying standard conformal methods, leading to poor coverage in healthcare settings. In this work, we evaluate several conformal prediction approaches on EEG seizure classification, a task with known distribution shift challenges and label uncertainty. We demonstrate that personalized calibration strategies can improve coverage by over 20 percentage points while maintaining comparable prediction set sizes. Our implementation is available via PyHealth, an open-source healthcare AI framework: https://github.com/sunlabuiuc/PyHealth.