Efficient Learning Under Density Shift in Incremental Settings Using Cram\'er-Rao-Based Regularization

TL;DR

This paper introduces a novel regularization method using Fisher information to improve neural network robustness against distribution shifts in sequential, distributed data settings, achieving significant accuracy gains.

Contribution

The paper proposes Covariate Shift Correction (C²A), a Fisher information-based regularizer that enhances neural network training under density shift in incremental data scenarios.

Findings

C²A achieves up to 19% higher accuracy than state-of-the-art methods.

The method effectively adapts to natural and sequential covariate shifts.

Density estimation via Fisher information improves robustness to non-iid data.

Abstract

The continuous surge in data volume and velocity is often dealt with using data orchestration and distributed processing approaches, abstracting away the machine learning challenges that exist at the algorithmic level. With growing interest in automating the learning loop, training with data that arrive in a sequence rather than in the classical in-memory training data form will face a machine learning challenge because of evolving feature distributions across batches of training data biasing the cross-validation step (\cite{sugiyama2012machine}). This work takes a distributed density estimation angle to the problem where data are temporally distributed. It processes data in batches and allows a neural network to treat a batch as training data. The method accumulates knowledge about the data density via posterior probability absorption using the Fisher Information Matrix, which contains information about the local optimization gradients for the batch. This is then used as a regularizer for the loss in the following batch, and therefore the density estimate for the entire dataset constructively gets more robust to the non-iid distribution shift. This needs the presence of a pair of batches in memory at a time, so the space cost is not a function of the size of the complete, distributed dataset. We proposed a novel regularization-based approach Covariate Shift Correction $C^{2}A$ that leverages Fisher information and Kullback-Leibler divergence to adapt to both natural and sequential covariate shift caused by dataset fragmentation. $C^{2}A$ achieves $19\%$ accuracy at maximum against state-of-the-art methods.