Stochastic Gradient Descent under Markovian Sampling Schemes

TL;DR

This paper analyzes the convergence of stochastic gradient descent algorithms that operate under Markovian sampling schemes, providing theoretical bounds and introducing a variance-reduced variant for improved efficiency.

Contribution

It establishes the first lower bounds for Markov chain-based SGD and proposes MC-SAG, a variance-reduced, communication-efficient algorithm under minimal assumptions.

Findings

Derived the theoretical lower bound involving Markov chain hitting time.

Proved convergence results under mild regularity assumptions.

Introduced MC-SAG with variance reduction and improved efficiency.

Abstract

We study a variation of vanilla stochastic gradient descent where the optimizer only has access to a Markovian sampling scheme. These schemes encompass applications that range from decentralized optimization with a random walker (token algorithms), to RL and online system identification problems. We focus on obtaining rates of convergence under the least restrictive assumptions possible on the underlying Markov chain and on the functions optimized. We first unveil the theoretical lower bound for methods that sample stochastic gradients along the path of a Markov chain, making appear a dependency in the hitting time of the underlying Markov chain. We then study Markov chain SGD (MC-SGD) under much milder regularity assumptions than prior works (e.g., no bounded gradients or domain, and infinite state spaces). We finally introduce MC-SAG, an alternative to MC-SGD with variance reduction, that only depends on the hitting time of the Markov chain, therefore obtaining a communication-efficient token algorithm.