Stochastic Zeroth-order Functional Constrained Optimization: Oracle Complexity and Applications

TL;DR

This paper introduces new stochastic zeroth-order algorithms for functionally constrained optimization problems in machine learning, providing the first known oracle complexity bounds for such problems with noisy function evaluations.

Contribution

It proposes and analyzes the first oracle complexity bounds for stochastic zeroth-order constrained optimization, applicable to convex and nonconvex problems.

Findings

Oracle complexity of O((m+1)n/ε^2) in convex case

Oracle complexity of O((m+1)n/ε^3) in nonconvex case

Demonstrated superior performance on hyperparameter tuning and neural network training

Abstract

Functionally constrained stochastic optimization problems, where neither the objective function nor the constraint functions are analytically available, arise frequently in machine learning applications. In this work, assuming we only have access to the noisy evaluations of the objective and constraint functions, we propose and analyze stochastic zeroth-order algorithms for solving the above class of stochastic optimization problem. When the domain of the functions is $\mathbb{R}^n$, assuming there are $m$ constraint functions, we establish oracle complexities of order $\mathcal{O}((m+1)n/\epsilon^2)$ and $\mathcal{O}((m+1)n/\epsilon^3)$ respectively in the convex and nonconvex setting, where $\epsilon$ represents the accuracy of the solutions required in appropriately defined metrics. The established oracle complexities are, to our knowledge, the first such results in the literature for functionally constrained stochastic zeroth-order optimization problems. We demonstrate the applicability of our algorithms by illustrating its superior performance on the problem of hyperparameter tuning for sampling algorithms and neural network training.