Incremental Quasi-Newton Algorithms for Solving Nonconvex, Nonsmooth, Finite-Sum Optimization Problems

TL;DR

This paper introduces incremental quasi-Newton algorithms, particularly IBFGS, for nonconvex, nonsmooth, finite-sum optimization problems, demonstrating their effectiveness in semi-supervised machine learning and outperforming existing methods.

Contribution

The paper proposes novel incremental quasi-Newton algorithms, including IBFGS variants, for challenging nonconvex, nonsmooth optimization, with extensive empirical validation.

Findings

All IBFGS approaches outperform a state-of-the-art bundle method.

Algorithms perform well in semi-supervised machine learning tasks.

Incremental quasi-Newton strategies are effective for nonconvex, nonsmooth problems.

Abstract

Algorithms for solving nonconvex, nonsmooth, finite-sum optimization problems are proposed and tested. In particular, the algorithms are proposed and tested in the context of an optimization problem formulation arising in semi-supervised machine learning. The common feature of all algorithms is that they employ an incremental quasi-Newton (IQN) strategy, specifically an incremental BFGS (IBFGS) strategy. One applies an IBFGS strategy to the problem directly, whereas the others apply an IBFGS strategy to a difference-of-convex reformulation, smoothed approximation, or (strongly) convex local approximation. Experiments show that all IBFGS approaches fare well in practice, and all outperform a state-of-the-art bundle method.