Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models

TL;DR

This paper introduces AFA, a novel framework for continual learning in vision-language models that enhances zero-shot recognition and few-shot learning capabilities, outperforming existing methods in multi-domain incremental tasks.

Contribution

The paper proposes a dual-adapter framework, AFA, that addresses forward and backward forgetting in continual learning of VLMs, improving zero-shot and few-shot performance.

Findings

AFA significantly outperforms state-of-the-art methods in few-shot MTIL tasks.

AFA surpasses CLIP's inherent zero-shot transferability.

Extensive experiments validate the effectiveness of the proposed framework.

Abstract

This study aims to address the problem of multi-domain task incremental learning~(MTIL), which requires that vision-language models~(VLMs) continuously acquire new knowledge while maintaining their inherent zero-shot recognition capability. Existing paradigms delegate the testing of unseen-domain samples to the original CLIP, which only prevents the degradation of the model's zero-shot capability but fails to enhance the generalization of the VLM further. To this end, we propose a novel MTIL framework, named AFA, which comprises two core modules: (1) an against forward-forgetting adapter that learns task-invariant information for each dataset in the incremental tasks to enhance the zero-shot recognition ability of VLMs; (2) an against backward-forgetting adapter that strengthens the few-shot learning capability of VLMs while supporting incremental learning. Extensive experiments demonstrate that the AFA method significantly outperforms existing state-of-the-art approaches, especially in few-shot MTIL tasks, and surpasses the inherent zero-shot performance of CLIP in terms of transferability. The code is provided in the Supplementary Material.