Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batches

TL;DR

This paper introduces SVRN, a stochastic variance-reduced Newton method that significantly accelerates finite-sum minimization, especially for large datasets, by reducing the number of data passes needed compared to existing methods.

Contribution

The paper proposes SVRN, a novel stochastic variance-reduced Newton algorithm that accelerates second-order methods for convex finite-sum problems, with improved complexity bounds and scalability.

Findings

SVRN reduces data passes from O(α log(1/ε)) to O(log(1/ε)/log(n))

Acceleration effect increases with larger data size n

SVRN compares favorably to first-order variance-reduction methods

Abstract

Stochastic variance reduction has proven effective at accelerating first-order algorithms for solving convex finite-sum optimization tasks such as empirical risk minimization. Incorporating second-order information has proven helpful in further improving the performance of these first-order methods. Yet, comparatively little is known about the benefits of using variance reduction to accelerate popular stochastic second-order methods such as Subsampled Newton. To address this, we propose Stochastic Variance-Reduced Newton (SVRN), a finite-sum minimization algorithm that provably accelerates existing stochastic Newton methods from $O(\alpha\log(1/\epsilon))$ to $O\big(\frac{\log(1/\epsilon)}{\log(n)}\big)$ passes over the data, i.e., by a factor of $O(\alpha\log(n))$, where $n$ is the number of sum components and $\alpha$ is the approximation factor in the Hessian estimate. Surprisingly, this acceleration gets more significant the larger the data size $n$, which is a unique property of SVRN. Our algorithm retains the key advantages of Newton-type methods, such as easily parallelizable large-batch operations and a simple unit step size. We use SVRN to accelerate Subsampled Newton and Iterative Hessian Sketch algorithms, and show that it compares favorably to popular first-order methods with variance~reduction.