ATLAS: Adapter-Based Multi-Modal Continual Learning with a Two-Stage Learning Strategy

TL;DR

This paper introduces ATLAS, a two-stage adapter-based continual learning method for multi-modal vision-and-language models that reduces forgetting and improves downstream task generalization by leveraging experience and knowledge expansion.

Contribution

It proposes a novel two-stage learning paradigm that integrates multi-modal and uni-modal tasks, enhancing continual learning and knowledge transfer in vision-language models.

Findings

Effective in reducing catastrophic forgetting.

Improves downstream task generalization.

Enhances representation diversity through upstream learning.

Abstract

While vision-and-language models significantly advance in many fields, the challenge of continual learning is unsolved. Parameter-efficient modules like adapters and prompts present a promising way to alleviate catastrophic forgetting. However, existing works usually learn individual adapters for each task, which may result in redundant knowledge among adapters. Moreover, they continue to use the original pre-trained model to initialize the downstream model, leading to negligible changes in the model's generalization compared to the original model. In addition, there is still a lack of research investigating the consequences of integrating a multi-modal model into the updating procedure for both uni-modal and multi-modal tasks and the subsequent impacts it has on downstream tasks. In this paper, we propose an adapter-based two-stage learning paradigm, a multi-modal continual learning scheme that consists of experience-based learning and novel knowledge expansion, which helps the model fully use experience knowledge and compensate for novel knowledge. Extensive experiments demonstrate that our method is proficient for continual learning. It expands the distribution of representation upstream while also minimizing the negative impact of forgetting previous tasks. Additionally, it enhances the generalization capability for downstream tasks. Furthermore, we incorporate both multi-modal and uni-modal tasks into upstream continual learning. We observe that learning from upstream tasks can help with downstream tasks. Our code will be available at: https://github.com/lihong2303/ATLAS.