Faster Riemannian Newton-type Optimization by Subsampling and Cubic Regularization

TL;DR

This paper introduces a second-order Riemannian optimization algorithm that combines subsampling and cubic regularization to improve convergence speed and scalability in large-scale non-convex constrained problems.

Contribution

It proposes a novel Riemannian trust-region method with subsampling and cubic regularization, enhancing efficiency and stability over existing algorithms.

Findings

Improved computational speed over state-of-the-art methods.

Enhanced convergence stability and rate.

Effective in large-scale machine learning tasks.

Abstract

This work is on constrained large-scale non-convex optimization where the constraint set implies a manifold structure. Solving such problems is important in a multitude of fundamental machine learning tasks. Recent advances on Riemannian optimization have enabled the convenient recovery of solutions by adapting unconstrained optimization algorithms over manifolds. However, it remains challenging to scale up and meanwhile maintain stable convergence rates and handle saddle points. We propose a new second-order Riemannian optimization algorithm, aiming at improving convergence rate and reducing computational cost. It enhances the Riemannian trust-region algorithm that explores curvature information to escape saddle points through a mixture of subsampling and cubic regularization techniques. We conduct rigorous analysis to study the convergence behavior of the proposed algorithm. We also perform extensive experiments to evaluate it based on two general machine learning tasks using multiple datasets. The proposed algorithm exhibits improved computational speed and convergence behavior compared to a large set of state-of-the-art Riemannian optimization algorithms.