Adaptive First- and Second-Order Algorithms for Large-Scale Machine Learning

TL;DR

This paper introduces adaptive first- and second-order optimization algorithms for large-scale machine learning, improving efficiency and performance in nonconvex deep learning tasks through novel stochastic regularization and L-BFGS methods.

Contribution

It presents a new framework for adaptive first-order methods with stochastic regularization and a stochastic damped L-BFGS algorithm tailored for deep learning.

Findings

Algorithms show promising performance on deep learning datasets

Adaptive sampling and step size improve convergence

Second-order method enhances optimization in nonconvex settings

Abstract

In this paper, we consider both first- and second-order techniques to address continuous optimization problems arising in machine learning. In the first-order case, we propose a framework of transition from deterministic or semi-deterministic to stochastic quadratic regularization methods. We leverage the two-phase nature of stochastic optimization to propose a novel first-order algorithm with adaptive sampling and adaptive step size. In the second-order case, we propose a novel stochastic damped L-BFGS method that improves on previous algorithms in the highly nonconvex context of deep learning. Both algorithms are evaluated on well-known deep learning datasets and exhibit promising performance.