UER: A Heuristic Bias Addressing Approach for Online Continual Learning

TL;DR

This paper introduces UER, a heuristic method for online continual learning that decomposes logits into angle and norm factors, using the norm to balance learning new and old knowledge, thereby reducing bias and improving performance.

Contribution

The paper proposes a novel bias correction approach in online continual learning by decomposing logits and leveraging the norm factor, which is simpler and more effective than existing methods.

Findings

UER outperforms state-of-the-art methods on three datasets.

Decomposition of logits into angle and norm factors reveals bias primarily in the angle.

Using the norm factor helps retain historical knowledge while learning new data.

Abstract

Online continual learning aims to continuously train neural networks from a continuous data stream with a single pass-through data. As the most effective approach, the rehearsal-based methods replay part of previous data. Commonly used predictors in existing methods tend to generate biased dot-product logits that prefer to the classes of current data, which is known as a bias issue and a phenomenon of forgetting. Many approaches have been proposed to overcome the forgetting problem by correcting the bias; however, they still need to be improved in online fashion. In this paper, we try to address the bias issue by a more straightforward and more efficient method. By decomposing the dot-product logits into an angle factor and a norm factor, we empirically find that the bias problem mainly occurs in the angle factor, which can be used to learn novel knowledge as cosine logits. On the contrary, the norm factor abandoned by existing methods helps remember historical knowledge. Based on this observation, we intuitively propose to leverage the norm factor to balance the new and old knowledge for addressing the bias. To this end, we develop a heuristic approach called unbias experience replay (UER). UER learns current samples only by the angle factor and further replays previous samples by both the norm and angle factors. Extensive experiments on three datasets show that UER achieves superior performance over various state-of-the-art methods. The code is in https://github.com/FelixHuiweiLin/UER.