Accelerating Trust-Region Methods: An Attempt to Balance Global and Local Efficiency

TL;DR

This paper introduces accelerated trust-region methods that balance global acceleration with local quadratic convergence, providing new theoretical insights and practical algorithms for second-order optimization.

Contribution

It proposes the first accelerated trust-region methods leveraging primal-dual info, achieving a balance between global acceleration and local convergence guarantees.

Findings

Achieves a global complexity of (psilon^{-1/3}) with quadratic local convergence.

Introduces a near-optimal global rate of (psilon^{-2/7}) with loss of quadratic local convergence.

Reveals a phase transition in the trade-off between global acceleration and local convergence.

Abstract

Historically speaking, it is hard to balance the global and local efficiency of second-order optimization algorithms. For instance, the classical Newton's method possesses excellent local convergence but lacks global guarantees, often exhibiting divergence when the starting point is far from the optimal solution~\cite{more1982newton,dennis1996numerical}. In contrast, accelerated second-order methods offer strong global convergence guarantees, yet they tend to converge with slower local rate~\cite{carmon2022optimal,chen2022accelerating,jiang2020unified}. Existing second-order methods struggle to balance global and local performance, leaving open the question of how much we can globally accelerate the second-order methods while maintaining excellent local convergence guarantee. In this paper, we tackle this challenge by proposing for the first time the accelerated trust-region-type methods, and leveraging their unique primal-dual information. Our primary technical contribution is \emph{Accelerating with Local Detection}, which utilizes the Lagrange multiplier to detect local regions and achieves a global complexity of $\tilde{O}(\epsilon^{-1/3})$, while maintaining quadratic local convergence. We further explore the trade-off when pushing the global convergence to the limit. In particular, we propose the \emph{Accelerated Trust-Region Extragradient Method} that has a global near-optimal rate of $\tilde{O}(\epsilon^{-2/7})$ but loses the quadratic local convergence. This reveals a phase transition in accelerated trust-region type methods: the excellent local convergence can be maintained when achieving a moderate global acceleration but becomes invalid when pursuing the extreme global efficiency. Numerical experiments further confirm the results indicated by our convergence analysis.