Faithful-Newton Framework: Bridging Inner and Outer Solvers for Enhanced Optimization

TL;DR

The Faithful-Newton framework introduces simple, integrated Newton-type methods that improve global convergence guarantees by directly assessing subproblem solution quality, achieving faster convergence rates for convex optimization.

Contribution

It presents a novel framework that tightly integrates inner and outer iterations, maintaining simplicity while enhancing convergence guarantees for Newton-type methods.

Findings

Achieves global superlinear convergence under standard assumptions.

Attains condition-number-independent linear convergence.

Demonstrates competitive performance in numerical experiments.

Abstract

Newton-type methods enjoy fast local convergence and strong empirical performance, but achieving global guarantees comparable to first-order methods remains challenging. Even for simple strongly convex problems, no straightforward variant of Newton's method matches the global complexity of gradient descent. While more sophisticated variants can improve iteration complexity, they typically require solving difficult subproblems with high per-iteration costs, leading to worse overall complexity. These limitations stem from treating the subproblem as an afterthought, either as a black box, yielding overly complex and impractical formulations, or in isolation, without regard to its role in advancing the optimization of the main objective. By tightening the integration between the inner iterations of the subproblem solvers and the outer iterations of the optimization algorithm, we introduce simple Newton-type variants, called Faithful-Newton framework, which, in a sense, remain faithful to the overall simplicity of classical Newton's method by retaining simple linear system subproblems. The key conceptual difference, however, is that the quality of the subproblem solution is directly assessed based on its effectiveness in reducing optimality, which in turn enables desirable convergence complexities across a variety of settings. Under standard assumptions, we show that our variants, depending on parameter choices, achieve global superlinear convergence, condition-number-independent linear convergence, and/or local quadratic convergence, even when using inexact Newton steps, for strongly convex problems; and competitive iteration complexity for general convex problems. Numerical experiments further demonstrate that our proposed methods perform competitively compared with several alternative Newton-type approaches.