Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retention

TL;DR

This paper introduces a dual-phase continual learning approach that combines supervised adaptation with unsupervised test-time data to mitigate forgetting in vision-language models without replay, demonstrating strong empirical results.

Contribution

It presents a novel memory-free method leveraging unlabeled test data and a teacher-student framework to reduce forgetting in class-incremental continual learning for VLMs.

Findings

Effective mitigation of forgetting without replay.

Strong empirical results on class-incremental tasks.

Memory-free approach using test-time data.

Abstract

Foundational Vision-Language Models (VLMs) excel across diverse tasks, but adapting them to new domains without forgetting prior knowledge remains a critical challenge. Continual Learning (CL) addresses this challenge by enabling models to learn sequentially from new data while mitigating the forgetting of prior information, typically under supervised settings involving label shift. Nonetheless, abrupt distribution shifts can still cause substantial forgetting, potentially nullifying the benefits of supervised updates, especially when storing or replaying past data is infeasible. In this work, we propose leveraging unlabeled testtime data in an unsupervised manner to reinforce prior task performance without requiring replay or stored examples. Unlike traditional Test Time Adaptation (TTA), which primarily focuses on domain shift or corruption, our method improves performance on earlier tasks by exploiting representative test samples encountered during deployment. We introduce a simple Teacher-Student framework with gradient-based sparse parameter updates, and show that it effectively mitigates forgetting in class-incremental CL for VLMs, offering a memory-free alternative to episodic replay with strong empirical results.