Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset

TL;DR

This paper introduces a novel adaptive soft reset method for neural networks that effectively handles non-stationary data distributions, improving learning in dynamic environments such as reinforcement learning and continual learning.

Contribution

It proposes an Ornstein-Uhlenbeck process with an adaptive drift for automatic modeling and adaptation to non-stationarity, acting as a soft parameter reset.

Findings

Performs well in non-stationary supervised learning

Effective in off-policy reinforcement learning

Outperforms traditional methods in dynamic settings

Abstract

Neural networks are traditionally trained under the assumption that data come from a stationary distribution. However, settings which violate this assumption are becoming more popular; examples include supervised learning under distributional shifts, reinforcement learning, continual learning and non-stationary contextual bandits. In this work we introduce a novel learning approach that automatically models and adapts to non-stationarity, via an Ornstein-Uhlenbeck process with an adaptive drift parameter. The adaptive drift tends to draw the parameters towards the initialisation distribution, so the approach can be understood as a form of soft parameter reset. We show empirically that our approach performs well in non-stationary supervised and off-policy reinforcement learning settings.