Incremental Meta-Learning via Episodic Replay Distillation for Few-Shot Image Recognition

TL;DR

This paper introduces Episodic Replay Distillation (ERD), a novel method for incremental meta-learning that effectively mitigates catastrophic forgetting by mixing current and past class exemplars during episodic training, outperforming existing approaches.

Contribution

The paper proposes ERD, a new incremental meta-learning approach that combines class replay with knowledge distillation to improve continual learning performance.

Findings

ERD surpasses state-of-the-art methods on four datasets.

Reduces the gap between IML and joint-training upper bound significantly.

Effective in one-shot, long task sequence scenarios.

Abstract

Most meta-learning approaches assume the existence of a very large set of labeled data available for episodic meta-learning of base knowledge. This contrasts with the more realistic continual learning paradigm in which data arrives incrementally in the form of tasks containing disjoint classes. In this paper we consider this problem of Incremental Meta-Learning (IML) in which classes are presented incrementally in discrete tasks. We propose an approach to IML, which we call Episodic Replay Distillation (ERD), that mixes classes from the current task with class exemplars from previous tasks when sampling episodes for meta-learning. These episodes are then used for knowledge distillation to minimize catastrophic forgetting. Experiments on four datasets demonstrate that ERD surpasses the state-of-the-art. In particular, on the more challenging one-shot, long task sequence incremental meta-learning scenarios, we reduce the gap between IML and the joint-training upper bound from 3.5% / 10.1% / 13.4% with the current state-of-the-art to 2.6% / 2.9% / 5.0% with our method on Tiered-ImageNet / Mini-ImageNet / CIFAR100, respectively.