Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence

TL;DR

The paper introduces the Newton Sketch, a randomized second-order optimization method that achieves linear-time complexity and super-linear convergence, applicable to various convex problems with high probability guarantees.

Contribution

It presents a novel randomized Newton method using Hessian sketching, providing convergence guarantees independent of condition numbers and extending to constrained convex optimization.

Findings

Achieves super-linear convergence with high probability

Lower complexity than traditional Newton's method when using randomized projections

Extends to convex constrained problems with self-concordant barriers

Abstract

We propose a randomized second-order method for optimization known as the Newton Sketch: it is based on performing an approximate Newton step using a randomly projected or sub-sampled Hessian. For self-concordant functions, we prove that the algorithm has super-linear convergence with exponentially high probability, with convergence and complexity guarantees that are independent of condition numbers and related problem-dependent quantities. Given a suitable initialization, similar guarantees also hold for strongly convex and smooth objectives without self-concordance. When implemented using randomized projections based on a sub-sampled Hadamard basis, the algorithm typically has substantially lower complexity than Newton's method. We also describe extensions of our methods to programs involving convex constraints that are equipped with self-concordant barriers. We discuss and illustrate applications to linear programs, quadratic programs with convex constraints, logistic regression and other generalized linear models, as well as semidefinite programs.