Faster Newton Methods for Convex and Nonconvex Optimization in Gradient Complexity

TL;DR

This paper introduces faster second-order optimization methods that significantly reduce gradient complexity for large-scale convex and nonconvex problems, advancing the state-of-the-art in efficiency.

Contribution

The authors propose new methods that improve gradient complexity bounds for both convex and nonconvex optimization, surpassing recent results in the field.

Findings

Achieved gradient complexity of O(d + d^{1/3} ε^{-3/2}) for nonconvex optimization.

Achieved gradient complexity of O((d + d^{13/21} ε^{-2/7}) log d) for convex optimization.

Improved the theoretical bounds compared to previous methods.

Abstract

Second-order optimization methods are computationally expensive for large-scale problems. Recently, Doikov, Chayti, and Jaggi (ICML 2023) proposed the LazyCRN method that reduces computation by studying the gradient complexity of second-order methods. Their method can achieve a gradient complexity of $\mathcal{O}( d + d^{1/2} \epsilon^{-3/2})$ and $\mathcal{O}( d + d^{1/2} \epsilon^{-1/2})$ for nonconvex and convex optimization, respectively, where $d$ is the effective dimension and $\epsilon$ is the target precision. Very recently, Adil, Bullins, Sidford, and Zhang (NeurIPS 2025) improved the gradient complexity to $\mathcal{O}( d + d^{1/3} \epsilon^{-3/2} \ln^{18} \epsilon^{-1})$ for nonconvex optimization. However, the tightness of these methods remains open. In this work, we propose new methods that achieve an improved complexity of $\mathcal{O}( d + d^{1/3} \epsilon^{-3/2})$ and $\mathcal{O}( (d + d^{13/21} \epsilon^{-2/7}) \ln d)$ for nonconvex and convex optimization, respectively, improving best-known results for both setups.