Faster stochastic cubic regularized Newton methods with momentum

TL;DR

This paper introduces faster stochastic cubic regularized Newton methods that incorporate momentum-based variance reduction for Hessian estimation, achieving improved iteration complexity and performance in large-scale optimization.

Contribution

The paper proposes novel SCRN algorithms with momentum-based Hessian estimation and analyzes their iteration complexity under mild assumptions.

Findings

Achieve comparable performance to deterministic CRN methods.

Significantly outperform first-order methods in iteration counts.

Effective for problems with accurate gradient estimation and costly Hessian computation.

Abstract

Cubic regularized Newton (CRN) methods have attracted signiffcant research interest because they offer stronger solution guarantees and lower iteration complexity. With the rise of the big-data era, there is growing interest in developing stochastic cubic regularized Newton (SCRN) methods that do not require exact gradient and Hessian evaluations. In this paper, we propose faster SCRN methods that incorporate gradient estimation with small, controlled errors and Hessian estimation with momentum-based variance reduction. These methods are particularly effective for problems where the gradient can be estimated accurately and at low cost, whereas accurate estimation of the Hessian is expensive. Under mild assumptions, we establish the iteration complexity of our SCRN methods by analyzing the descent of a novel potential sequence. Finally, numerical experiments show that our SCRN methods can achieve comparable performance to deterministic CRN methods and vastly outperform ffrst-order methods in terms of both iteration counts and solution quality.