ErrorEraser: Unlearning Data Bias for Improved Continual Learning

TL;DR

ErrorEraser introduces a universal plugin for continual learning that identifies and erases biased, erroneous memories caused by data biases, significantly improving knowledge retention and transfer across tasks.

Contribution

The paper presents ErrorEraser, a novel method with modules for identifying and erasing biased data, enhancing continual learning by addressing data bias issues.

Findings

Significantly reduces forgetting rates in continual learning.

Improves accuracy across multiple CL methods.

Effectively mitigates negative impacts of data biases.

Abstract

Continual Learning (CL) primarily aims to retain knowledge to prevent catastrophic forgetting and transfer knowledge to facilitate learning new tasks. Unlike traditional methods, we propose a novel perspective: CL not only needs to prevent forgetting, but also requires intentional forgetting.This arises from existing CL methods ignoring biases in real-world data, leading the model to learn spurious correlations that transfer and amplify across tasks. From feature extraction and prediction results, we find that data biases simultaneously reduce CL's ability to retain and transfer knowledge. To address this, we propose ErrorEraser, a universal plugin that removes erroneous memories caused by biases in CL, enhancing performance in both new and old tasks. ErrorEraser consists of two modules: Error Identification and Error Erasure. The former learns the probability density distribution of task data in the feature space without prior knowledge, enabling accurate identification of potentially biased samples. The latter ensures only erroneous knowledge is erased by shifting the decision space of representative outlier samples. Additionally, an incremental feature distribution learning strategy is designed to reduce the resource overhead during error identification in downstream tasks. Extensive experimental results show that ErrorEraser significantly mitigates the negative impact of data biases, achieving higher accuracy and lower forgetting rates across three types of CL methods. The code is available at https://github.com/diadai/ErrorEraser.