First-order methods for stochastic and finite-sum convex optimization with deterministic constraints

TL;DR

This paper introduces stochastic first-order methods that ensure solutions satisfy deterministic constraints within a specified tolerance, improving reliability in practical convex optimization applications.

Contribution

It proposes novel accelerated stochastic gradient methods that find solutions with deterministic constraint satisfaction, unlike existing approaches focusing on expected feasibility.

Findings

Established first-order oracle complexity bounds for the proposed methods.

Derived complexity results for sample average approximation using these methods.

Demonstrated the effectiveness of the methods in achieving deterministic constraint satisfaction.

Abstract

In this paper, we study a class of stochastic and finite-sum convex optimization problems with deterministic constraints. Existing methods typically aim to find an $\epsilon$-$expectedly\ feasible\ stochastic\ optimal$ solution, in which the expected constraint violation and expected optimality gap are both within a prescribed tolerance $\epsilon$. However, in many practical applications, constraints must be nearly satisfied with certainty, rendering such solutions potentially unsuitable due to the risk of substantial violations. To address this issue, we propose stochastic first-order methods for finding an $\epsilon$-$surely\ feasible\ stochastic\ optimal$ ($\epsilon$-SFSO) solution, where the constraint violation is deterministically bounded by $\epsilon$ and the expected optimality gap is at most $\epsilon$. Our methods apply an accelerated stochastic gradient (ASG) scheme or a modified variance-reduced ASG scheme $only\ once$ to a sequence of quadratic penalty subproblems with appropriately chosen penalty parameters. We establish first-order oracle complexity bounds for the proposed methods in computing an $\epsilon$-SFSO solution. As a byproduct, we also derive first-order oracle complexity results for sample average approximation method in computing an $\epsilon$-SFSO solution of the stochastic optimization problem using our proposed methods to solve the sample average problem.