Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors

TL;DR

This paper introduces a fusion mechanism that incorporates all previous feature extractors to mitigate catastrophic forgetting in continual learning, demonstrating improved performance and preservation of earlier knowledge.

Contribution

It proposes a novel fusion approach that includes all past feature extractors and uses pruning to control model size, enhancing continual learning effectiveness.

Findings

Reduces forgetting of earlier learned knowledge

Achieves state-of-the-art continual learning results

Effectively maintains model performance over time

Abstract

Continual learning of new knowledge over time is one desirable capability for intelligent systems to recognize more and more classes of objects. Without or with very limited amount of old data stored, an intelligent system often catastrophically forgets previously learned old knowledge when learning new knowledge. Recently, various approaches have been proposed to alleviate the catastrophic forgetting issue. However, old knowledge learned earlier is commonly less preserved than that learned more recently. In order to reduce the forgetting of particularly earlier learned old knowledge and improve the overall continual learning performance, we propose a simple yet effective fusion mechanism by including all the previously learned feature extractors into the intelligent model. In addition, a new feature extractor is included to the model when learning a new set of classes each time, and a feature extractor pruning is also applied to prevent the whole model size from growing rapidly. Experiments on multiple classification tasks show that the proposed approach can effectively reduce the forgetting of old knowledge, achieving state-of-the-art continual learning performance.