ASPEN: An Additional Sampling Penalty Method for Finite-Sum Optimization Problems with Nonlinear Equality Constraints

TL;DR

This paper introduces ASPEN, a novel algorithm for non-convex finite-sum optimization with nonlinear equality constraints, combining adaptive sampling, penalty methods, and non-monotone line search to improve efficiency and convergence.

Contribution

It develops a new sampling penalty framework that adapts sample size and penalty parameters, extending existing methods to more complex nonlinear constrained problems.

Findings

Proves almost sure convergence under standard assumptions.

Demonstrates computational savings through adaptive sampling.

Shows effectiveness on academic and real-world machine learning problems.

Abstract

We propose a novel algorithm for solving non-convex, nonlinear equality-constrained finite-sum optimization problems. The proposed algorithm incorporates an additional sampling strategy for sample size update into the well-known framework of quadratic penalty methods. Thus, depending on the problem at hand, the resulting method may exhibit a sample size strategy ranging from a mini-batch on one end, to increasing sample size that achieves the full sample eventually, on the other end of the spectrum. A non-monotone line search is used for the step size update, while the penalty parameter is also adaptive. The proposed algorithm avoids costly projections, which, together with the sample size update, may yield significant computational cost savings. Also, the proposed method can be viewed as a transition of an additional sampling approach for unconstrained and linear constrained problems, to a more general class with non-linear constraints. The almost sure convergence is proved under a standard set of assumptions for this framework, while numerical experiments on both academic and real-data based machine learning problems demonstrate the effectiveness of the proposed approach.