CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models

TL;DR

CLAP4CLIP introduces a probabilistic finetuning framework for vision-language models like CLIP, enhancing continual learning by improving uncertainty estimation and reducing forgetting, thus enabling safer and more reliable downstream tasks.

Contribution

The paper proposes CLAP, a novel probabilistic finetuning method for CLIP in continual learning, which improves uncertainty calibration and mitigates forgetting compared to deterministic approaches.

Findings

CLAP outperforms deterministic finetuning methods in continual learning tasks.

It provides better uncertainty estimation for high-risk applications.

The approach enables effective data detection and exemplar selection.

Abstract

Continual learning (CL) aims to help deep neural networks learn new knowledge while retaining what has been learned. Owing to their powerful generalizability, pre-trained vision-language models such as Contrastive Language-Image Pre-training (CLIP) have lately gained traction as practical CL candidates. However, the domain mismatch between the pre-training and the downstream CL tasks often calls for finetuning of the CLIP on the latter. Most existing finetuning methods exhibit deterministic nature. This makes them overlook the many possible interactions across the input modalities and deems them unsafe for high-risk tasks requiring reliable uncertainty estimation. To address these, our work proposes Continual LeArning with Probabilistic finetuning (CLAP) - a probabilistic modeling framework over visual-guided text features per task, thus providing more calibrated CL finetuning. Unlike recent data-hungry anti-forgetting CL techniques, CLAP alleviates forgetting by exploiting the rich pre-trained knowledge of CLIP for weight initialization and distribution regularization of task-specific parameters. Cooperating with the diverse range of existing prompting methods, CLAP can surpass the predominant deterministic finetuning approaches for CL with CLIP. We conclude with out-of-the-box applications of superior uncertainty estimation abilities of CLAP including novel data detection and exemplar selection within the existing CL setups. Our code is available at \url{https://github.com/srvCodes/clap4clip}.