Big-model Driven Few-shot Continual Learning

TL;DR

This paper introduces a novel framework that leverages large pre-trained models to enhance few-shot continual learning, improving accuracy and avoiding overfitting by adaptive transfer and distillation techniques.

Contribution

The paper proposes a big-model driven framework with an adaptive decision mechanism for effective knowledge transfer in few-shot continual learning.

Findings

Outperforms state-of-the-art FSCL methods on CIFAR100, miniImageNet, and CUB200.

Utilizes big-model transfer learning to adapt to new samples effectively.

Employs an adaptive distillation process to optimize continual model parameters.

Abstract

Few-shot continual learning (FSCL) has attracted intensive attention and achieved some advances in recent years, but now it is difficult to again make a big stride in accuracy due to the limitation of only few-shot incremental samples. Inspired by distinctive human cognition ability in life learning, in this work, we propose a novel Big-model driven Few-shot Continual Learning (B-FSCL) framework to gradually evolve the model under the traction of the world's big-models (like human accumulative knowledge). Specifically, we perform the big-model driven transfer learning to leverage the powerful encoding capability of these existing big-models, which can adapt the continual model to a few of newly added samples while avoiding the over-fitting problem. Considering that the big-model and the continual model may have different perceived results for the identical images, we introduce an instance-level adaptive decision mechanism to provide the high-level flexibility cognitive support adjusted to varying samples. In turn, the adaptive decision can be further adopted to optimize the parameters of the continual model, performing the adaptive distillation of big-model's knowledge information. Experimental results of our proposed B-FSCL on three popular datasets (including CIFAR100, minilmageNet and CUB200) completely surpass all state-of-the-art FSCL methods.