Improved global performance guarantees of second-order methods in convex minimization

TL;DR

This paper compares two second-order optimization approaches for convex minimization, introduces improved path-following schemes with better complexity bounds, and applies these methods to various constrained problems.

Contribution

It develops new theoretical analyses and path-following schemes that improve global performance guarantees for second-order methods on self-concordant functions.

Findings

Path-following scheme has better complexity than Damped Newton Method

New predictor-corrector scheme improves constant factors in complexity

CRNMs achieve superior bounds on strongly convex functions with Lipschitz continuous second derivative

Abstract

In this paper, we attempt to compare two distinct branches of research on second-order optimization methods. The first one studies self-concordant functions and barriers, the main assumption being that the third derivative of the objective is bounded by the second derivative. The second branch studies cubic regularized Newton methods (CRNMs) with the main assumption that the second derivative is Lipschitz continuous. We develop a new theoretical analysis for a path-following scheme (PFS) for general self-concordant functions, as opposed to the classical path-following scheme developed for self-concordant barriers. We show that the complexity bound for this scheme is better than that of the Damped Newton Method (DNM) and show that our method has global superlinear convergence. We propose also a new predictor-corrector path-following scheme (PCPFS) that leads to further improvement of constant factors in the complexity guarantees for minimizing general self-concordant functions. We also apply path-following schemes to different classes of constrained optimization problems and obtain the resulting complexity bounds. Finally, we analyze an important subclass of general self-concordant functions, namely a class of strongly convex functions with Lipschitz continuous second derivative, and show that for this subclass CRNMs give even better complexity bounds.