FCL-ViT: Task-Aware Attention Tuning for Continual Learning

TL;DR

FCL-ViT introduces a feedback mechanism in Vision Transformers for continual learning, dynamically tuning attention features to adapt to new tasks while outperforming existing methods.

Contribution

The paper proposes FCL-ViT, a novel feedback-based Vision Transformer architecture with task-aware attention tuning for improved continual learning performance.

Findings

Surpasses state-of-the-art in continual learning benchmarks.

Uses fewer trainable parameters than existing methods.

Effectively adapts attention features to new tasks.

Abstract

Continual Learning (CL) involves adapting the prior Deep Neural Network (DNN) knowledge to new tasks, without forgetting the old ones. However, modern CL techniques focus on provisioning memory capabilities to existing DNN models rather than designing new ones that are able to adapt according to the task at hand. This paper presents the novel Feedback Continual Learning Vision Transformer (FCL-ViT) that uses a feedback mechanism to generate real-time dynamic attention features tailored to the current task. The FCL-ViT operates in two Phases. In phase 1, the generic image features are produced and determine where the Transformer should attend on the current image. In phase 2, task-specific image features are generated that leverage dynamic attention. To this end, Tunable self-Attention Blocks (TABs) and Task Specific Blocks (TSBs) are introduced that operate in both phases and are responsible for tuning the TABs attention, respectively. The FCL-ViT surpasses state-of-the-art performance on Continual Learning compared to benchmark methods, while retaining a small number of trainable DNN parameters.