LSOS: Line-search Second-Order Stochastic optimization methods for nonconvex finite sums

TL;DR

This paper introduces a novel line-search second-order stochastic optimization framework for nonconvex finite sums, achieving convergence without convexity assumptions and demonstrating competitive empirical performance.

Contribution

It presents a new second-order method with line searches for nonconvex finite sums, controlling errors via two-step sampling, and proves convergence properties.

Findings

Proves almost-sure stationarity of limit points.

Shows convergence to solutions under strong convexity.

Numerical experiments outperform some existing methods.

Abstract

We develop a line-search second-order algorithmic framework for minimizing finite sums. We do not make any convexity assumptions, but require the terms of the sum to be continuously differentiable and have Lipschitz-continuous gradients. The methods fitting into this framework combine line searches and suitably decaying step lengths. A key issue is a two-step sampling at each iteration, which allows us to control the error present in the line-search procedure. Stationarity of limit points is proved in the almost-sure sense, while almost-sure convergence of the sequence of approximations to the solution holds with the additional hypothesis that the functions are strongly convex. Numerical experiments, including comparisons with state-of-the art stochastic optimization methods, show the efficiency of our approach.