Stochastic Penalty-Barrier Methods for Constrained Machine Learning

TL;DR

This paper introduces SPBM, a novel stochastic penalty-barrier method for constrained deep learning that handles non-convex, non-smooth problems efficiently, outperforming existing baselines.

Contribution

The paper develops a new stochastic penalty-barrier algorithm with exponential dual averaging and the Moreau envelope, addressing non-convex, non-smooth constraints in deep learning.

Findings

SPBM matches or outperforms existing constrained optimization methods.

SPBM incurs only linear runtime overhead compared to unconstrained Adam.

Effective for problems with up to 10,000 constraints.

Abstract

Constrained machine learning enables fairness-aware training, physics-informed neural networks, and integration of symbolic domain knowledge into statistical models. Despite its practical importance, no general method exists for the non-convex, non-smooth, stochastic setting that arises naturally in deep learning. We propose the Stochastic Penalty-Barrier Method (SPBM), which extends classical penalty and barrier methods to this setting via exponential dual averaging, a stabilized penalty schedule, and the Moreau envelope to handle non-smoothness. Experiments across multiple settings show that SPBM matches or outperforms existing constrained optimization baselines while incurring only linear runtime overhead compared to unconstrained Adam for up to 10,000 constraints.