A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters

TL;DR

This paper introduces SimE, a simple and efficient incremental learning framework using vision-language models with nonlinear adapter connections, improving training efficiency and performance without relying on large memory banks or strong backbones.

Contribution

Proposes SimE, a novel incremental learning framework with nonlinear adapter connections, demonstrating improved efficiency and performance over traditional methods and CLIP-based approaches.

Findings

SimE surpasses traditional methods by 9.6% on TinyImageNet.

SimE outperforms other CLIP-based methods by 5.3% on CIFAR-100.

Nonlinear correlation exists between adapter connections and IL capabilities.

Abstract

Incremental Learning (IL) aims to learn new tasks while preserving previously acquired knowledge. Integrating the zero-shot learning capabilities of pre-trained vision-language models into IL methods has marked a significant advancement. However, these methods face three primary challenges: (1) the need for improved training efficiency; (2) reliance on a memory bank to store previous data; and (3) the necessity of a strong backbone to augment the model's capabilities. In this paper, we propose SimE, a Simple and Efficient framework that employs a vision-language model with adapters designed specifically for the IL task. We report a remarkable phenomenon: there is a nonlinear correlation between the number of adaptive adapter connections and the model's IL capabilities. While increasing adapter connections between transformer blocks improves model performance, adding more adaptive connections within transformer blocks during smaller incremental steps does not enhance, and may even degrade the model's IL ability. Extensive experimental results show that SimE surpasses traditional methods by 9.6% on TinyImageNet and outperforms other CLIP-based methods by 5.3% on CIFAR-100. Furthermore, we conduct a systematic study to enhance the utilization of the zero-shot capabilities of CLIP. We suggest replacing SimE's encoder with a CLIP model trained on larger datasets (e.g., LAION2B) and stronger architectures (e.g., ViT-L/14).