Hinge-Proximal Stochastic Gradient Methods for Convex Optimization with Functional Constraints

TL;DR

This paper introduces a hinge-proximal stochastic gradient framework for high-dimensional convex optimization with functional constraints, improving efficiency and empirical performance in streaming and online settings.

Contribution

It develops a novel hinge-proximal approach that uses an exact penalty reformulation, enabling updates with only one constraint gradient per iteration and relaxing the need for Lipschitz continuity.

Findings

New hinge-proximal algorithms outperform existing methods.

Variance-reduced and nested versions achieve near-SGD complexity.

Empirical results demonstrate superior performance on robust regression tasks.

Abstract

This paper considers stochastic convex optimization problems with smooth functional constraints arising in constrained estimation and robust signal recovery. We operate in the high-dimensional and highly-constrained setting, where oracle access is restricted to one or a few objective and constraint gradients per-iteration, as in streaming or online estimation. Existing approaches to solve such problems are based on either the stochastic primal-dual or stochastic subgradient methods, and require globally Lipschitz continuous constraint functions. In this work, we develop a hinge-proximal framework that utilizes an exact penalty reformulation to yield updates involving only one linearized constraint (and hence accessing one constraint gradient) per-iteration. The updates also admit a novel hinge-proximal three-point inequality relying on smoothness rather than global Lipschitz continuity of the constraint functions. The framework leads to three algorithms: a baseline hinge-proximal SGD (HPS), a variance-reduced HPS version for finite-sum settings, and a nested HPS version whose performance depends on a geometric regularity constant of the constraint region rather than explicitly on the number of constraints, while achieving near-SGD sample complexity. The superior empirical performance of the proposed algorithms is demonstrated on a robust regression problem with noisy features, representative of errors-in-variables estimation.