Augmented Newton Method for Optimization: Global Linear Rate and Momentum Interpretation

TL;DR

This paper introduces two new variants of the Newton method for unconstrained optimization, incorporating penalty and augmented Lagrangian techniques, with interpretations as momentum methods and proven global convergence.

Contribution

It proposes the Penalty Newton and Augmented Newton methods, unifying existing variants and providing a new momentum interpretation with global convergence guarantees.

Findings

The Augmented Newton method can be seen as Newton with adaptive heavy ball momentum.

Global convergence results are established under mild assumptions.

The methods generalize several well-known Newton variants.

Abstract

We propose two variants of Newton method for solving unconstrained minimization problem. Our method leverages optimization techniques such as penalty and augmented Lagrangian method to generate novel variants of the Newton method namely the Penalty Newton method and the Augmented Newton method. In doing so, we recover several well-known existing Newton method variants such as Damped Newton, Levenberg, and Levenberg-Marquardt methods as special cases. Moreover, the proposed Augmented Newton method can be interpreted as Newton method with adaptive heavy ball momentum. We provide global convergence results for the proposed methods under mild assumptions that hold for a wide variety of problems. The proposed methods can be sought as the penalty and augmented extensions of the results obtained by Karimireddy et. al [24].