Approximate Newton Methods

TL;DR

This paper introduces a unifying framework to analyze the convergence of stochastic second-order optimization methods, bridging the gap between theory and practical performance in large-scale machine learning problems.

Contribution

It provides a comprehensive theoretical analysis of local and global convergence for subsampled Newton and related methods, aligning theory with real-world applications.

Findings

The framework explains convergence behavior of subsampled Newton methods.

Theoretical results match observed performance in large-scale problems.

Addresses gaps between existing convergence theory and practical efficiency.

Abstract

Many machine learning models involve solving optimization problems. Thus, it is important to deal with a large-scale optimization problem in big data applications. Recently, subsampled Newton methods have emerged to attract much attention due to their efficiency at each iteration, rectified a weakness in the ordinary Newton method of suffering a high cost in each iteration while commanding a high convergence rate. Other efficient stochastic second order methods are also proposed. However, the convergence properties of these methods are still not well understood. There are also several important gaps between the current convergence theory and the performance in real applications. In this paper, we aim to fill these gaps. We propose a unifying framework to analyze both local and global convergence properties of second order methods. Based on this framework, we present our theoretical results which match the performance in real applications well.