# Quasi-Newton Methods for Machine Learning: Forget the Past, Just Sample

## TL;DR

This paper introduces sampled quasi-Newton methods that use random sampling around the current point to build more reliable Hessian approximations, improving efficiency and convergence in machine learning optimization tasks.

## Contribution

The paper proposes novel sampled LBFGS and sampled LSR1 algorithms that utilize local sampling for Hessian approximation, enhancing convergence and parallelization capabilities.

## Key findings

- Outperform classical quasi-Newton methods in classification and neural network training.
- Require fewer data points (epochs) for convergence.
- Effective in parallel and distributed computing environments.

## Abstract

We present two sampled quasi-Newton methods (sampled LBFGS and sampled LSR1) for solving empirical risk minimization problems that arise in machine learning. Contrary to the classical variants of these methods that sequentially build Hessian or inverse Hessian approximations as the optimization progresses, our proposed methods sample points randomly around the current iterate at every iteration to produce these approximations. As a result, the approximations constructed make use of more reliable (recent and local) information, and do not depend on past iterate information that could be significantly stale. Our proposed algorithms are efficient in terms of accessed data points (epochs) and have enough concurrency to take advantage of parallel/distributed computing environments. We provide convergence guarantees for our proposed methods. Numerical tests on a toy classification problem as well as on popular benchmarking binary classification and neural network training tasks reveal that the methods outperform their classical variants.