Nonconvex Stochastic Nested Optimization via Stochastic ADMM

TL;DR

This paper introduces a stochastic ADMM method for nonconvex stochastic nested optimization problems, achieving optimal sample complexity and handling more general cases than existing proximal algorithms.

Contribution

The paper proposes a stochastic ADMM algorithm for nonconvex nested problems with improved complexity and broader applicability compared to prior proximal methods.

Findings

Achieves $ ilde{O}(rac{1}{ ext{epsilon}^3})$ sample complexity for online case.

Achieves $ ilde{O}(rac{1}{ ext{epsilon}^2})$ complexity for finite sum case.

Handles problems where proximal mapping of the penalty is difficult.

Abstract

We consider the stochastic nested composition optimization problem where the objective is a composition of two expected-value functions. We proposed the stochastic ADMM to solve this complicated objective. In order to find an $\epsilon$ stationary point where the expected norm of the subgradient of corresponding augmented Lagrangian is smaller than $\epsilon$, the total sample complexity of our method is $\mathcal{O}(\epsilon^{-3})$ for the online case and $\mathcal{O} \Bigl((2N_1 + N_2) + (2N_1 + N_2)^{1/2}\epsilon^{-2}\Bigr)$ for the finite sum case. The computational complexity is consistent with proximal version proposed in \cite{zhang2019multi}, but our algorithm can solve more general problem when the proximal mapping of the penalty is not easy to compute.