Random Subspace Cubic-Regularization Methods, with Applications to Low-Rank Functions

TL;DR

This paper introduces random subspace variants of the cubic regularization algorithm, which efficiently optimize low-rank functions by reducing computational complexity while maintaining optimal convergence rates.

Contribution

The authors develop and analyze subspace methods for cubic regularization that adaptively handle low-rank functions, improving scalability and efficiency.

Findings

Maintains optimal convergence rates in reduced subspaces.

Improves scalability for low-rank function optimization.

Automatically adapts to the true rank without prior knowledge.

Abstract

We propose and analyze random subspace variants of the second-order Adaptive Regularization using Cubics (ARC) algorithm. These methods iteratively restrict the search space to some random subspace of the parameters, constructing and minimizing a local model only within this subspace. Thus, our variants only require access to (small-dimensional) projections of first- and second-order problem derivatives and calculate a reduced step inexpensively. Under suitable assumptions, the ensuing methods maintain the optimal first-order, and second-order, global rates of convergence of (full-dimensional) cubic regularization, while showing improved scalability both theoretically and numerically, particularly when applied to low-rank functions. When applied to the latter, our adaptive variant naturally adapts the subspace size to the true rank of the function, without knowing it a priori.