Multi-Point Bandit Algorithms for Nonstationary Online Nonconvex Optimization

TL;DR

This paper develops and analyzes bandit algorithms for nonstationary online nonconvex optimization, introducing novel regret measures based on stationary solutions and Hessian estimation, with applications to avoiding saddle points and structured nonconvex functions.

Contribution

It proposes new bandit algorithms for nonstationary nonconvex problems, including Hessian estimation via Gaussian Stein's identity and regret bounds for structured nonconvex functions.

Findings

Regret bounds for nonstationary nonconvex optimization using stationary solutions.

Hessian estimation in bandit setting via Gaussian Stein's identity.

Algorithms for avoiding saddle points in nonstationary bandit problems.

Abstract

Bandit algorithms have been predominantly analyzed in the convex setting with function-value based stationary regret as the performance measure. In this paper, motivated by online reinforcement learning problems, we propose and analyze bandit algorithms for both general and structured nonconvex problems with nonstationary (or dynamic) regret as the performance measure, in both stochastic and non-stochastic settings. First, for general nonconvex functions, we consider nonstationary versions of first-order and second-order stationary solutions as a regret measure, motivated by similar performance measures for offline nonconvex optimization. In the case of second-order stationary solution based regret, we propose and analyze online and bandit versions of the cubic regularized Newton's method. The bandit version is based on estimating the Hessian matrices in the bandit setting, based on second-order Gaussian Stein's identity. Our nonstationary regret bounds in terms of second-order stationary solutions have interesting consequences for avoiding saddle points in the bandit setting. Next, for weakly quasi convex functions and monotone weakly submodular functions we consider nonstationary regret measures in terms of function-values; such structured classes of nonconvex functions enable one to consider regret measure defined in terms of function values, similar to convex functions. For this case of function-value, and first-order stationary solution based regret measures, we provide regret bounds in both the low- and high-dimensional settings, for some scenarios.