Attention Retention for Continual Learning with Vision Transformers

TL;DR

This paper introduces a novel attention-retaining framework for Vision Transformers to mitigate catastrophic forgetting in continual learning by explicitly constraining attention drift through gradient masking, inspired by neuroscientific insights.

Contribution

It proposes a new method that preserves learned visual concepts in Vision Transformers during continual learning by controlling attention drift with a gradient masking technique.

Findings

Achieves state-of-the-art performance in continual learning tasks.

Effectively mitigates catastrophic forgetting across diverse scenarios.

Preserves attention to previously learned visual concepts.

Abstract

Continual learning (CL) empowers AI systems to progressively acquire knowledge from non-stationary data streams. However, catastrophic forgetting remains a critical challenge. In this work, we identify attention drift in Vision Transformers as a primary source of catastrophic forgetting, where the attention to previously learned visual concepts shifts significantly after learning new tasks. Inspired by neuroscientific insights into the selective attention in the human visual system, we propose a novel attention-retaining framework to mitigate forgetting in CL. Our method constrains attention drift by explicitly modifying gradients during backpropagation through a two-step process: 1) extracting attention maps of the previous task using a layer-wise rollout mechanism and generating instance-adaptive binary masks, and 2) when learning a new task, applying these masks to zero out gradients associated with previous attention regions, thereby preventing disruption of learned visual concepts. For compatibility with modern optimizers, the gradient masking process is further enhanced by scaling parameter updates proportionally to maintain their relative magnitudes. Experiments and visualizations demonstrate the effectiveness of our method in mitigating catastrophic forgetting and preserving visual concepts. It achieves state-of-the-art performance and exhibits robust generalizability across diverse CL scenarios.