Continual Normalization: Rethinking Batch Normalization for Online Continual Learning

TL;DR

This paper introduces Continual Normalization, a new normalization technique designed to improve online continual learning by addressing the limitations of Batch Normalization in non-i.i.d data scenarios, leading to better task retention.

Contribution

The paper proposes Continual Normalization, a simple replacement for Batch Normalization that mitigates its negative effects in online continual learning, enhancing performance across tasks.

Findings

CN outperforms BN in online continual learning scenarios.

Extensive experiments show CN improves task retention and reduces catastrophic forgetting.

CN is compatible with various continual learning algorithms.

Abstract

Existing continual learning methods use Batch Normalization (BN) to facilitate training and improve generalization across tasks. However, the non-i.i.d and non-stationary nature of continual learning data, especially in the online setting, amplify the discrepancy between training and testing in BN and hinder the performance of older tasks. In this work, we study the cross-task normalization effect of BN in online continual learning where BN normalizes the testing data using moments biased towards the current task, resulting in higher catastrophic forgetting. This limitation motivates us to propose a simple yet effective method that we call Continual Normalization (CN) to facilitate training similar to BN while mitigating its negative effect. Extensive experiments on different continual learning algorithms and online scenarios show that CN is a direct replacement for BN and can provide substantial performance improvements. Our implementation is available at \url{https://github.com/phquang/Continual-Normalization}.