AN-SPS: Adaptive Sample Size Nonmonotone Line Search Spectral Projected Subgradient Method for Convex Constrained Optimization Problems

TL;DR

This paper introduces AN-SPS, an adaptive sample size spectral projected subgradient method with a nonmonotone line search for convex constrained optimization, ensuring almost sure convergence without boundedness assumptions.

Contribution

It proposes a novel adaptive sampling scheme combined with spectral and nonmonotone line search techniques for convex optimization with convergence guarantees.

Findings

Preliminary results show advantages of the adaptive scheme on hinge loss problems.

The method achieves almost sure convergence without boundedness assumptions.

Different nonmonotone line search strategies are compared within the framework.

Abstract

We consider convex optimization problems with a possibly nonsmooth objective function in the form of a mathematical expectation. The proposed framework (AN-SPS) employs Sample Average Approximations (SAA) to approximate the objective function, which is either unavailable or too costly to compute. The sample size is chosen in an adaptive manner, which eventually pushes the SAA error to zero almost surely (a.s.). The search direction is based on a scaled subgradient and a spectral coefficient, both related to the SAA function. The step size is obtained via a nonmonotone line search over a predefined interval, which yields a theoretically sound and practically efficient algorithm. The method retains feasibility by projecting the resulting points onto a feasible set. The a.s. convergence of AN-SPS method is proved without the assumption of a bounded feasible set or bounded iterates. Preliminary numerical results on Hinge loss problems reveal the advantages of the proposed adaptive scheme. In addition, a study of different nonmonotone line search strategies in combination with different spectral coefficients within AN-SPS framework is also conducted, yielding some hints for future work.