Anti-Retroactive Interference for Lifelong Learning

TL;DR

This paper introduces a lifelong learning paradigm inspired by cognitive science, combining meta-learning and associative mechanisms to improve knowledge retention and task performance in machine learning models.

Contribution

It proposes a novel approach that disrupts background distributions and adaptively fuses knowledge based on similarity, enhancing lifelong learning capabilities.

Findings

Improved performance on MNIST, CIFAR100, CUB200, and ImageNet100 datasets.

Theoretically proven convergence to the same optimum across tasks.

Effective mitigation of catastrophic forgetting in continual learning.

Abstract

Humans can continuously learn new knowledge. However, machine learning models suffer from drastic dropping in performance on previous tasks after learning new tasks. Cognitive science points out that the competition of similar knowledge is an important cause of forgetting. In this paper, we design a paradigm for lifelong learning based on meta-learning and associative mechanism of the brain. It tackles the problem from two aspects: extracting knowledge and memorizing knowledge. First, we disrupt the sample's background distribution through a background attack, which strengthens the model to extract the key features of each task. Second, according to the similarity between incremental knowledge and base knowledge, we design an adaptive fusion of incremental knowledge, which helps the model allocate capacity to the knowledge of different difficulties. It is theoretically analyzed that the proposed learning paradigm can make the models of different tasks converge to the same optimum. The proposed method is validated on the MNIST, CIFAR100, CUB200 and ImageNet100 datasets.