Meta-attention for ViT-backed Continual Learning

TL;DR

This paper introduces MEAT, a novel attention-based method for adapting pre-trained vision transformers to continual learning tasks, achieving higher accuracy and efficiency than CNN-based approaches.

Contribution

The paper proposes MEAT, a new mask-based continual learning method for ViTs that masks only a portion of parameters, improving efficiency and accuracy over prior CNN-based methods.

Findings

MEAT outperforms CNN-based methods with 4-6% higher accuracy.

MEAT is more efficient with less parameter overhead.

Extensive experiments validate MEAT's superiority in continual learning for ViTs.

Abstract

Continual learning is a longstanding research topic due to its crucial role in tackling continually arriving tasks. Up to now, the study of continual learning in computer vision is mainly restricted to convolutional neural networks (CNNs). However, recently there is a tendency that the newly emerging vision transformers (ViTs) are gradually dominating the field of computer vision, which leaves CNN-based continual learning lagging behind as they can suffer from severe performance degradation if straightforwardly applied to ViTs. In this paper, we study ViT-backed continual learning to strive for higher performance riding on recent advances of ViTs. Inspired by mask-based continual learning methods in CNNs, where a mask is learned per task to adapt the pre-trained ViT to the new task, we propose MEta-ATtention (MEAT), i.e., attention to self-attention, to adapt a pre-trained ViT to new tasks without sacrificing performance on already learned tasks. Unlike prior mask-based methods like Piggyback, where all parameters are associated with corresponding masks, MEAT leverages the characteristics of ViTs and only masks a portion of its parameters. It renders MEAT more efficient and effective with less overhead and higher accuracy. Extensive experiments demonstrate that MEAT exhibits significant superiority to its state-of-the-art CNN counterparts, with 4.0~6.0% absolute boosts in accuracy. Our code has been released at https://github.com/zju-vipa/MEAT-TIL.