An Exponential Averaging Process with Strong Convergence Properties

TL;DR

This paper introduces a modified exponential averaging method called p-EMA with improved stochastic convergence properties, applicable to noisy observations in random dynamical systems and adaptive SGD step size control.

Contribution

The paper proposes p-EMA, an adaptation of EMA with weights decreasing to zero, and provides convergence guarantees under mild autocorrelation assumptions.

Findings

p-EMA achieves strong stochastic convergence.

Applicable to noisy observations in dynamical systems.

Implications for adaptive SGD step size control.

Abstract

Averaging, or smoothing, is a fundamental approach to obtain stable, de-noised estimates from noisy observations. In certain scenarios, observations made along trajectories of random dynamical systems are of particular interest. One popular smoothing technique for such a scenario is exponential moving averaging (EMA), which assigns observations a weight that decreases exponentially in their age, thus giving younger observations a larger weight. However, EMA fails to enjoy strong stochastic convergence properties, which stems from the fact that the weight assigned to the youngest observation is constant over time, preventing the noise in the averaged quantity from decreasing to zero. In this work, we consider an adaptation to EMA, which we call $p$-EMA, where the weights assigned to the last observations decrease to zero at a subharmonic rate. We provide stochastic convergence guarantees for this kind of averaging under mild assumptions on the autocorrelations of the underlying random dynamical system. We further discuss the implications of our results for a recently introduced adaptive step size control for Stochastic Gradient Descent (SGD), which uses $p$-EMA for averaging noisy observations.