New Penalized Stochastic Gradient Methods for Linearly Constrained Strongly Convex Optimization

TL;DR

This paper introduces a novel penalized stochastic gradient method for strongly convex optimization with linear constraints, achieving optimal complexity and enabling constraint elimination for efficiency.

Contribution

It proposes a new nested accelerated stochastic gradient algorithm with adaptive penalty smoothing, improving convergence and computational efficiency for large-scale constrained problems.

Findings

Achieves $ ilde O(1/\sqrt{ ext{epsilon}})$ complexity for solution accuracy

Provides bounds on the distance between original and penalized solutions

Demonstrates effectiveness through computational experiments

Abstract

For minimizing a strongly convex objective function subject to linear inequality constraints, we consider a penalty approach that allows one to utilize stochastic methods for problems with a large number of constraints and/or objective function terms. We provide upper bounds on the distance between the solutions to the original constrained problem and the penalty reformulations, guaranteeing the convergence of the proposed approach. We give a nested accelerated stochastic gradient method and propose a novel way for updating the smoothness parameter of the penalty function and the step-size. The proposed algorithm requires at most $\tilde O(1/\sqrt{\epsilon})$ expected stochastic gradient iterations to produce a solution within an expected distance of $\epsilon$ to the optimal solution of the original problem, which is the best complexity for this problem class to the best of our knowledge. We also show how to query an approximate dual solution after stochastically solving the penalty reformulations, leading to results on the convergence of the duality gap. Moreover, the nested structure of the algorithm and upper bounds on the distance to the optimal solutions allows one to safely eliminate constraints that are inactive at an optimal solution throughout the algorithm, which leads to improved complexity results. Finally, we present computational results that demonstrate the effectiveness and robustness of our algorithm.