Adaptive Newton Sketch: Linear-time Optimization with Quadratic Convergence and Effective Hessian Dimensionality

TL;DR

This paper introduces an adaptive randomized Newton sketch algorithm that achieves quadratic convergence for convex optimization by dynamically adjusting the sketch size based on the effective Hessian dimension, improving efficiency and convergence.

Contribution

The paper presents a novel adaptive sketch size method that does not require prior knowledge of the effective dimension, ensuring quadratic convergence in convex optimization.

Findings

Achieves quadratic convergence with a sketch size proportional to the effective dimension.

Demonstrates the adaptive method's ability to maintain optimal sketch size throughout optimization.

Provides state-of-the-art computational complexity for strongly convex convex programs.

Abstract

We propose a randomized algorithm with quadratic convergence rate for convex optimization problems with a self-concordant, composite, strongly convex objective function. Our method is based on performing an approximate Newton step using a random projection of the Hessian. Our first contribution is to show that, at each iteration, the embedding dimension (or sketch size) can be as small as the effective dimension of the Hessian matrix. Leveraging this novel fundamental result, we design an algorithm with a sketch size proportional to the effective dimension and which exhibits a quadratic rate of convergence. This result dramatically improves on the classical linear-quadratic convergence rates of state-of-the-art sub-sampled Newton methods. However, in most practical cases, the effective dimension is not known beforehand, and this raises the question of how to pick a sketch size as small as the effective dimension while preserving a quadratic convergence rate. Our second and main contribution is thus to propose an adaptive sketch size algorithm with quadratic convergence rate and which does not require prior knowledge or estimation of the effective dimension: at each iteration, it starts with a small sketch size, and increases it until quadratic progress is achieved. Importantly, we show that the embedding dimension remains proportional to the effective dimension throughout the entire path and that our method achieves state-of-the-art computational complexity for solving convex optimization programs with a strongly convex component.