Stable-Drift: A Patient-Aware Latent Drift Replay Method for Stabilizing Representations in Continual Learning

TL;DR

This paper introduces Stable-Drift, a patient-aware replay method that uses latent drift to identify and replay unstable samples, significantly reducing catastrophic forgetting in continual learning for medical imaging.

Contribution

It proposes a novel latent drift-guided replay approach that leverages internal feature changes to improve continual learning stability in medical imaging tasks.

Findings

Reduces forgetting in COVID-19 CT classification across hospitals

Outperforms naive fine-tuning and random replay methods

Effective with CNN and Vision Transformer backbones

Abstract

When deep learning models are sequentially trained on new data, they tend to abruptly lose performance on previously learned tasks, a critical failure known as catastrophic forgetting. This challenge severely limits the deployment of AI in medical imaging, where models must continually adapt to data from new hospitals without compromising established diagnostic knowledge. To address this, we introduce a latent drift-guided replay method that identifies and replays samples with high representational instability. Specifically, our method quantifies this instability via latent drift, the change in a sample internal feature representation after naive domain adaptation. To ensure diversity and clinical relevance, we aggregate drift at the patient level, our memory buffer stores the per patient slices exhibiting the greatest multi-layer representation shift. Evaluated on a cross-hospital COVID-19 CT classification task using state-of-the-art CNN and Vision Transformer backbones, our method substantially reduces forgetting compared to naive fine-tuning and random replay. This work highlights latent drift as a practical and interpretable replay signal for advancing robust continual learning in real world medical settings.