Stochastic Optimization Algorithms for Problems with Controllable Biased Oracles

TL;DR

This paper develops stochastic optimization algorithms that adaptively control bias in gradient estimates, addressing applications like stochastic composition and policy optimization, and analyzes their convergence and complexity.

Contribution

It introduces algorithms that dynamically adjust bias-control parameters in stochastic gradient methods for biased oracles, with theoretical convergence guarantees.

Findings

Algorithms achieve convergence to stationary points in nonconvex settings.

Theoretical bounds on sample complexity and bias-control complexity.

Numerical experiments demonstrate effectiveness across three applications.

Abstract

Motivated by emerging applications in machine learning, we consider an optimization problem in a general form where the gradient of the objective function is available through a biased stochastic oracle. We assume a bias-control parameter can reduce the bias magnitude; however, a lower bias requires more computation/samples. For instance, in two applications on stochastic composition optimization and policy optimization for infinite-horizon Markov decision processes, we show that the bias follows a power law and exponential decay, respectively, as functions of their corresponding bias control parameters. For problems with such gradient oracles, the paper proposes stochastic algorithms that adjust the bias-control parameter throughout the iterations. We analyze the nonasymptotic performance of the proposed algorithms in the nonconvex regime and establish their sample or bias-control computational complexities to obtain a stationary point in expectation or with high probability. Finally, we numerically evaluate the performance of the proposed algorithms over three applications.