CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning

TL;DR

This paper introduces a generative replay method using VAEs to enhance incremental learning in CLIP, effectively mitigating forgetting and improving zero-shot performance across diverse domains.

Contribution

It presents a novel generative replay approach with VAEs for CLIP, maintaining zero-shot abilities and adapting to new tasks without catastrophic forgetting.

Findings

Improves zero-shot capabilities in incremental learning

Bridges gap with joint prompt tuning

Effective across multiple domain shifts

Abstract

With the emergence of Transformers and Vision-Language Models (VLMs) such as CLIP, fine-tuning large pre-trained models has recently become a prevalent strategy in Continual Learning. This has led to the development of numerous prompting strategies to adapt transformer-based models without incurring catastrophic forgetting. However, these strategies often compromise the original zero-shot capabilities of the pre-trained CLIP model and struggle to adapt to domains that significantly deviate from the pre-training data. In this work, we propose Continual Generative training for Incremental prompt-Learning, a simple and novel approach to mitigate forgetting while adapting CLIP. Briefly, we employ Variational Autoencoders (VAEs) to learn class-conditioned distributions within the embedding space of the visual encoder. We then exploit these distributions to sample new synthetic visual embeddings and train the corresponding class-specific textual prompts during subsequent tasks. Through extensive experiments on different domains, we show that such a generative replay approach can adapt to new tasks while improving zero-shot capabilities, evaluated using a novel metric tailored for CL scenarios. Notably, further analysis reveals that our approach can bridge the gap with joint prompt tuning. The codebase is available at https://github.com/aimagelab/mammoth.