Less-forgetting Learning in Deep Neural Networks

TL;DR

This paper introduces a novel method to reduce catastrophic forgetting in deep neural networks without relying on source domain data, improving retention of learned information and overall recognition performance.

Contribution

The proposed less-forgetting learning method effectively mitigates forgetting without source data and addresses mini-batch forgetting during training, enhancing neural network generalization.

Findings

Effective in reducing source domain information loss

Improves recognition rates in neural networks

Addresses mini-batch forgetting during training

Abstract

A catastrophic forgetting problem makes deep neural networks forget the previously learned information, when learning data collected in new environments, such as by different sensors or in different light conditions. This paper presents a new method for alleviating the catastrophic forgetting problem. Unlike previous research, our method does not use any information from the source domain. Surprisingly, our method is very effective to forget less of the information in the source domain, and we show the effectiveness of our method using several experiments. Furthermore, we observed that the forgetting problem occurs between mini-batches when performing general training processes using stochastic gradient descent methods, and this problem is one of the factors that degrades generalization performance of the network. We also try to solve this problem using the proposed method. Finally, we show our less-forgetting learning method is also helpful to improve the performance of deep neural networks in terms of recognition rates.