Less is more: Summarizing Patch Tokens for efficient Multi-Label Class-Incremental Learning

TL;DR

This paper introduces MULTI-LANE, a method that summarizes patch tokens to enable efficient multi-label class-incremental learning with disentangled task-specific representations, achieving state-of-the-art results.

Contribution

It proposes a novel approach to multi-label class-incremental learning by summarizing patch tokens, eliminating prompt selection, and maintaining task-specific pathways for efficiency and disentanglement.

Findings

MULTI-LANE achieves state-of-the-art performance in MLCIL benchmarks.

The method is also competitive in class-incremental learning scenarios.

Summarizing patch tokens reduces computational complexity significantly.

Abstract

Prompt tuning has emerged as an effective rehearsal-free technique for class-incremental learning (CIL) that learns a tiny set of task-specific parameters (or prompts) to instruct a pre-trained transformer to learn on a sequence of tasks. Albeit effective, prompt tuning methods do not lend well in the multi-label class incremental learning (MLCIL) scenario (where an image contains multiple foreground classes) due to the ambiguity in selecting the correct prompt(s) corresponding to different foreground objects belonging to multiple tasks. To circumvent this issue we propose to eliminate the prompt selection mechanism by maintaining task-specific pathways, which allow us to learn representations that do not interact with the ones from the other tasks. Since independent pathways in truly incremental scenarios will result in an explosion of computation due to the quadratically complex multi-head self-attention (MSA) operation in prompt tuning, we propose to reduce the original patch token embeddings into summarized tokens. Prompt tuning is then applied to these fewer summarized tokens to compute the final representation. Our proposed method Multi-Label class incremental learning via summarising pAtch tokeN Embeddings (MULTI-LANE) enables learning disentangled task-specific representations in MLCIL while ensuring fast inference. We conduct experiments in common benchmarks and demonstrate that our MULTI-LANE achieves a new state-of-the-art in MLCIL. Additionally, we show that MULTI-LANE is also competitive in the CIL setting. Source code available at https://github.com/tdemin16/multi-lane