Global linear convergence of Newton's method without strong-convexity or Lipschitz gradients

TL;DR

This paper proves that Newton's method achieves global linear convergence for a broad class of functions with stable Hessians, including non-strongly convex problems like logistic regression, even with approximate Hessians and subproblem solutions.

Contribution

It establishes affine-invariant linear convergence of Newton's method without requiring strong convexity or Lipschitz gradients, extending its applicability.

Findings

Newton's method converges linearly for functions with stable Hessians.

The convergence holds even with approximate Hessians and subproblem solutions.

Newton's method outperforms first-order methods under the studied conditions.

Abstract

We show that Newton's method converges globally at a linear rate for objective functions whose Hessians are stable. This class of problems includes many functions which are not strongly convex, such as logistic regression. Our linear convergence result is (i) affine-invariant, and holds even if an (ii) approximate Hessian is used, and if the subproblems are (iii) only solved approximately. Thus we theoretically demonstrate the superiority of Newton's method over first-order methods, which would only achieve a sublinear $O(1/t^2)$ rate under similar conditions.