# Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild

## TL;DR

This paper introduces a novel lifelong learning approach that leverages unlabeled data in the wild to significantly reduce catastrophic forgetting and improve accuracy on sequential tasks.

## Contribution

It proposes a new class-incremental learning scheme with global distillation, overfitting avoidance, and confidence-based sampling for unlabeled data, advancing continual learning methods.

## Key findings

- Up to 15.8% higher accuracy on CIFAR and ImageNet.
- 46.5% less forgetting compared to prior methods.
- Effective utilization of unlabeled data in lifelong learning scenarios.

## Abstract

Lifelong learning with deep neural networks is well-known to suffer from catastrophic forgetting: the performance on previous tasks drastically degrades when learning a new task. To alleviate this effect, we propose to leverage a large stream of unlabeled data easily obtainable in the wild. In particular, we design a novel class-incremental learning scheme with (a) a new distillation loss, termed global distillation, (b) a learning strategy to avoid overfitting to the most recent task, and (c) a confidence-based sampling method to effectively leverage unlabeled external data. Our experimental results on various datasets, including CIFAR and ImageNet, demonstrate the superiority of the proposed methods over prior methods, particularly when a stream of unlabeled data is accessible: our method shows up to 15.8% higher accuracy and 46.5% less forgetting compared to the state-of-the-art method. The code is available at https://github.com/kibok90/iccv2019-inc.