UNICON: UNIfied CONtinual Learning for Medical Foundational Models

TL;DR

This paper introduces UNICON, a unified continual learning framework that enables medical foundation models to adapt across multiple domains, tasks, and modalities without forgetting, demonstrated by improved performance on CT and PET imaging tasks.

Contribution

UNICON provides a novel, unified continual learning approach that allows foundation models to expand across diverse medical imaging domains and tasks without catastrophic forgetting.

Findings

Improved performance on prognosis and segmentation tasks after adaptation.

Achieved a 5% increase in Dice score with PET scans.

Validated the framework's ability to prevent task interference.

Abstract

Foundational models are trained on extensive datasets to capture the general trends of a domain. However, in medical imaging, the scarcity of data makes pre-training for every domain, modality, or task challenging. Continual learning offers a solution by fine-tuning a model sequentially on different domains or tasks, enabling it to integrate new knowledge without requiring large datasets for each training phase. In this paper, we propose UNIfied CONtinual Learning for Medical Foundational Models (UNICON), a framework that enables the seamless adaptation of foundation models to diverse domains, tasks, and modalities. Unlike conventional adaptation methods that treat these changes in isolation, UNICON provides a unified, perpetually expandable framework. Through careful integration, we show that foundation models can dynamically expand across imaging modalities, anatomical regions, and clinical objectives without catastrophic forgetting or task interference. Empirically, we validate our approach by adapting a chest CT foundation model initially trained for classification to a prognosis and segmentation task. Our results show improved performance across both additional tasks. Furthermore, we continually incorporated PET scans and achieved a 5\% improvement in Dice score compared to respective baselines. These findings establish that foundation models are not inherently constrained to their initial training scope but can evolve, paving the way toward generalist AI models for medical imaging.