A single-loop SPIDER-type stochastic subgradient method for expectation-constrained nonconvex nonsmooth optimization

TL;DR

This paper introduces a novel single-loop SPIDER-type stochastic subgradient method for nonconvex nonsmooth optimization with expectation constraints, achieving improved complexity and practical efficiency over existing methods.

Contribution

It develops a new exact penalty model and a single-loop stochastic subgradient algorithm that handles complex expectation constraints with weaker regularity conditions.

Findings

Achieves an $O(\epsilon^{-4})$ iteration complexity for near-$\epsilon$ stationary points.

Obtains an $(\epsilon,\epsilon)$-KKT point of the original problem.

Significantly faster performance (up to 466x) on fairness and Neyman-Pearson problems compared to state-of-the-art algorithms.

Abstract

Many real-world problems, such as those with fairness constraints, involve complex expectation constraints and large datasets, necessitating the design of efficient stochastic methods to solve them. Most existing research focuses on cases with no {constraint} or easy-to-project constraints or deterministic constraints. In this paper, we consider nonconvex nonsmooth stochastic optimization problems with expectation constraints, for which we build a novel exact penalty model. We first show the relationship between the penalty model and the original problem. Then on solving the penalty problem, we present a single-loop SPIDER-type stochastic subgradient method, which utilizes the subgradients of both the objective and constraint functions, as well as the constraint function value at each iteration. Under certain regularity conditions (weaker than Slater-type constraint qualification or strong feasibility assumed in existing works), we establish an iteration complexity result of $O(\epsilon^{-4})$ to reach a near-$\epsilon$ stationary point of the penalized problem in expectation, matching the lower bound for such tasks. Building on the exact penalization, an $(\epsilon,\epsilon)$-KKT point of the original problem is obtained. For a few scenarios, our complexity of either the {objective} sample subgradient or the constraint sample function values can be lower than the state-of-the-art results by a factor of $\epsilon^{-2}$. Moreover, on solving two fairness-constrained problems and a multi-class Neyman-Pearson classification problem, our method is significantly (up to 466 times) faster than the state-of-the-art algorithms, including switching subgradient method and inexact proximal point methods.