CLeAN: Continual Learning Adaptive Normalization in Dynamic Environments

TL;DR

This paper introduces CLeAN, an adaptive normalization method for continual learning on tabular data, which dynamically adjusts to evolving data distributions and improves model performance while reducing forgetting.

Contribution

CLeAN is a novel normalization technique that estimates feature scales with learnable parameters updated via EMA, tailored for continual learning in dynamic environments.

Findings

CLeAN enhances model performance on sequential data tasks.

It mitigates catastrophic forgetting in continual learning scenarios.

Demonstrated effectiveness across multiple datasets and strategies.

Abstract

Artificial intelligence systems predominantly rely on static data distributions, making them ineffective in dynamic real-world environments, such as cybersecurity, autonomous transportation, or finance, where data shifts frequently. Continual learning offers a potential solution by enabling models to learn from sequential data while retaining prior knowledge. However, a critical and underexplored issue in this domain is data normalization. Conventional normalization methods, such as min-max scaling, presuppose access to the entire dataset, which is incongruent with the sequential nature of continual learning. In this paper we introduce Continual Learning Adaptive Normalization (CLeAN), a novel adaptive normalization technique designed for continual learning in tabular data. CLeAN involves the estimation of global feature scales using learnable parameters that are updated via an Exponential Moving Average (EMA) module, enabling the model to adapt to evolving data distributions. Through comprehensive evaluations on two datasets and various continual learning strategies, including Resevoir Experience Replay, A-GEM, and EwC we demonstrate that CLeAN not only improves model performance on new data but also mitigates catastrophic forgetting. The findings underscore the importance of adaptive normalization in enhancing the stability and effectiveness of tabular data, offering a novel perspective on the use of normalization to preserve knowledge in dynamic learning environments.