Continual Retinal Vision-Language Pre-training upon Incremental Imaging Modalities

TL;DR

RetCoP is a novel continual vision-language pre-training framework for fundus imaging that incrementally integrates multiple modalities, effectively mitigating catastrophic forgetting and enhancing model generalization in dynamic environments.

Contribution

This work introduces the first continual fundus vision-language pre-training framework, RetCoP, with a rehearsal strategy and off-diagonal information distillation to handle incremental modalities.

Findings

RetCoP outperforms existing methods in generalization.

RetCoP achieves the lowest forgetting rate.

RetCoP effectively integrates multiple fundus modalities.

Abstract

Traditional fundus image analysis models focus on single-modal tasks, ignoring fundus modality complementarity, which limits their versatility. Recently, retinal foundation models have emerged, but most still remain modality-specific. Integrating multiple fundus imaging modalities into a single foundation model is valuable. However, in dynamic environments, data from different modalities often arrive incrementally, necessitating continual pre-training. To address this, we propose RetCoP, the first continual vision-language pre-training framework in the fundus domain, which incrementally integrates image and text features from different imaging modalities into a single unified foundation model. To mitigate catastrophic forgetting in continual pre-training, we introduce a rehearsal strategy utilizing representative image-text pairs and an off-diagonal information distillation approach. The former allows the model to revisit knowledge from previous stages, while the latter explicitly preserves the alignment between image and text representations. Experiments show that RetCoP outperforms all the compared methods, achieving the best generalization and lowest forgetting rate. The code can be found at https://github.com/Yuang-Yao/RetCoP.