Gradient-Free Approaches is a Key to an Efficient Interaction with Markovian Stochasticity

TL;DR

This paper introduces a new derivative-free optimization method for Markovian stochastic problems, demonstrating that it can efficiently handle noise with mixing times less than the problem dimension, and proves its optimality.

Contribution

A novel derivative-free method for Markovian stochastic optimization that is optimal and independent of noise mixing time when it is less than the problem dimension.

Findings

Convergence rates independent of mixing time when τ < d.

Method is optimal with matching lower bounds.

Zero-order oracle outperforms first-order in this setting.

Abstract

This paper deals with stochastic optimization problems involving Markovian noise with a zero-order oracle. We present and analyze a novel derivative-free method for solving such problems in strongly convex smooth and non-smooth settings with both one-point and two-point feedback oracles. Using a randomized batching scheme, we show that when mixing time $\tau$ of the underlying noise sequence is less than the dimension of the problem $d$, the convergence estimates of our method do not depend on $\tau$. This observation provides an efficient way to interact with Markovian stochasticity: instead of invoking the expensive first-order oracle, one should use the zero-order oracle. Finally, we complement our upper bounds with the corresponding lower bounds. This confirms the optimality of our results.