Few-Shot and Continual Learning with Attentive Independent Mechanisms

TL;DR

This paper introduces Attentive Independent Mechanisms (AIM), a modular approach that enhances few-shot and continual learning in deep neural networks by enabling independent concept learning and fast adaptation.

Contribution

AIM is a novel modular component that decouples feature extraction from higher-order learning, improving adaptability and reducing forgetting in neural networks.

Findings

AIM significantly improves few-shot learning performance on MiniImageNet and CIFAR-FS.

AIM enhances continual learning capabilities on Omniglot, CIFAR-100, and MiniImageNet.

The approach is modular and can be integrated into existing frameworks.

Abstract

Deep neural networks (DNNs) are known to perform well when deployed to test distributions that shares high similarity with the training distribution. Feeding DNNs with new data sequentially that were unseen in the training distribution has two major challenges -- fast adaptation to new tasks and catastrophic forgetting of old tasks. Such difficulties paved way for the on-going research on few-shot learning and continual learning. To tackle these problems, we introduce Attentive Independent Mechanisms (AIM). We incorporate the idea of learning using fast and slow weights in conjunction with the decoupling of the feature extraction and higher-order conceptual learning of a DNN. AIM is designed for higher-order conceptual learning, modeled by a mixture of experts that compete to learn independent concepts to solve a new task. AIM is a modular component that can be inserted into existing deep learning frameworks. We demonstrate its capability for few-shot learning by adding it to SIB and trained on MiniImageNet and CIFAR-FS, showing significant improvement. AIM is also applied to ANML and OML trained on Omniglot, CIFAR-100 and MiniImageNet to demonstrate its capability in continual learning. Code made publicly available at https://github.com/huang50213/AIM-Fewshot-Continual.