Semantic Prompting with Image-Token for Continual Learning

TL;DR

This paper introduces I-Prompt, a task-agnostic prompt-based method for continual learning that leverages image tokens' semantic information to improve performance and efficiency without relying on task prediction.

Contribution

The paper proposes I-Prompt, a novel approach that uses semantic prompt matching and image token-level prompting to eliminate task prediction in continual learning.

Findings

Achieves competitive performance on four benchmarks.

Reduces training time significantly compared to state-of-the-art methods.

Demonstrates robustness across various scenarios.

Abstract

Continual learning aims to refine model parameters for new tasks while retaining knowledge from previous tasks. Recently, prompt-based learning has emerged to leverage pre-trained models to be prompted to learn subsequent tasks without the reliance on the rehearsal buffer. Although this approach has demonstrated outstanding results, existing methods depend on preceding task-selection process to choose appropriate prompts. However, imperfectness in task-selection may lead to negative impacts on the performance particularly in the scenarios where the number of tasks is large or task distributions are imbalanced. To address this issue, we introduce I-Prompt, a task-agnostic approach focuses on the visual semantic information of image tokens to eliminate task prediction. Our method consists of semantic prompt matching, which determines prompts based on similarities between tokens, and image token-level prompting, which applies prompts directly to image tokens in the intermediate layers. Consequently, our method achieves competitive performance on four benchmarks while significantly reducing training time compared to state-of-the-art methods. Moreover, we demonstrate the superiority of our method across various scenarios through extensive experiments.