Asymptotic Bias of Stochastic Gradient Search

TL;DR

This paper analyzes the long-term bias of stochastic gradient algorithms with biased estimators, providing bounds and insights applicable to high-dimensional nonlinear methods like reinforcement learning and Monte Carlo sampling.

Contribution

It introduces a theoretical framework using dynamic systems and differential geometry to bound the asymptotic bias of biased stochastic gradient algorithms under mild conditions.

Findings

Derived tight bounds on asymptotic bias for biased stochastic gradients.

Applied results to policy-gradient reinforcement learning.

Analyzed asymptotic behavior in hidden Markov model estimation.

Abstract

The asymptotic behavior of the stochastic gradient algorithm with a biased gradient estimator is analyzed. Relying on arguments based on the dynamic system theory (chain-recurrence) and the differential geometry (Yomdin theorem and Lojasiewicz inequality), tight bounds on the asymptotic bias of the iterates generated by such an algorithm are derived. The obtained results hold under mild conditions and cover a broad class of high-dimensional nonlinear algorithms. Using these results, the asymptotic properties of the policy-gradient (reinforcement) learning and adaptive population Monte Carlo sampling are studied. Relying on the same results, the asymptotic behavior of the recursive maximum split-likelihood estimation in hidden Markov models is analyzed, too.