Few-Shot Continual Learning for 3D Brain MRI with Frozen Foundation Models

TL;DR

This paper introduces a method for few-shot continual learning on 3D brain MRI that uses frozen foundation models with task-specific LoRA adapters, achieving zero forgetting and efficient adaptation across multiple tasks.

Contribution

The paper proposes combining frozen pretrained backbone models with task-specific LoRA modules for continual learning on 3D MRI, avoiding catastrophic forgetting and requiring minimal trainable parameters.

Findings

LoRA adapters prevent catastrophic forgetting in sequential tasks.

The method achieves balanced performance on tumor segmentation and brain age estimation.

Sequential fine-tuning suffers severe forgetting, while linear probing fails on some tasks.

Abstract

Foundation models pretrained on large-scale 3D medical imaging data face challenges when adapted to multiple downstream tasks under continual learning with limited labeled data. We address few-shot continual learning for 3D brain MRI by combining a frozen pretrained backbone with task-specific Low-Rank Adaptation (LoRA) modules. Tasks arrive sequentially -- tumor segmentation (BraTS) and brain age estimation (IXI) -- with no replay of previous task data. Each task receives a dedicated LoRA adapter; only the adapter and task-specific head are trained while the backbone remains frozen, thereby eliminating catastrophic forgetting by design (BWT=0). In continual learning, sequential full fine-tuning suffers severe forgetting (T1 Dice drops from 0.80 to 0.16 after T2), while sequential linear probing achieves strong T1 (Dice 0.79) but fails on T2 (MAE 1.45). Our LoRA approach achieves the best balanced performance across both tasks: T1 Dice 0.62$\pm$0.07, T2 MAE 0.16$\pm$0.05, with zero forgetting and $<$0.1\% trainable parameters per task, though with noted systematic age underestimation in T2 (Wilcoxon $p<0.001$). Frozen foundation models with task-specific LoRA adapters thus offer a practical solution when both tasks must be maintained under few-shot continual learning.