Improving Convergence Guarantees of Random Subspace Second-order Algorithm for Nonconvex Optimization

TL;DR

This paper introduces RSHTR, a novel random subspace trust region method that guarantees improved convergence rates for nonconvex optimization, supported by theoretical analysis and real-world experiments.

Contribution

The paper proposes RSHTR, the first random subspace method with optimal convergence guarantees for both first- and second-order stationary points in nonconvex optimization.

Findings

RSHTR achieves $ ext{O}( extstyle{rac{1}{ ext{ε}^{3/2}}})$ iteration complexity.

RSHTR converges locally at a linear rate.

RSHTR exhibits quadratic convergence under rank-deficient conditions.

Abstract

In recent years, random subspace methods have been actively studied for large-dimensional nonconvex problems. Recent subspace methods have improved theoretical guarantees such as iteration complexity and local convergence rate while reducing computational costs by deriving descent directions in randomly selected low-dimensional subspaces. This paper proposes the Random Subspace Homogenized Trust Region (RSHTR) method with the best theoretical guarantees among random subspace algorithms for nonconvex optimization. RSHTR achieves an $\varepsilon$-approximate first-order stationary point in $O(\varepsilon^{-3/2})$ iterations, converging locally at a linear rate. Furthermore, under rank-deficient conditions, RSHTR satisfies $\varepsilon$-approximate second-order necessary conditions in $O(\varepsilon^{-3/2})$ iterations and exhibits a local quadratic convergence. Experiments on real-world datasets verify the benefits of RSHTR.