Scalable Second-Order Optimization Algorithms for Minimizing Low-rank Functions

TL;DR

This paper introduces a scalable second-order optimization algorithm that adaptively chooses subspace sizes based on the function's rank, improving efficiency for low-rank problems while maintaining optimal convergence.

Contribution

It proposes a random-subspace cubic regularization method that adaptively adjusts subspace size, enhancing scalability for low-rank functions without prior rank knowledge.

Findings

Maintains optimal convergence rate of cubic regularization.

Improves scalability both theoretically and numerically.

Automatically adapts to the true rank of the function.

Abstract

We present a random-subspace variant of cubic regularization algorithm that chooses the size of the subspace adaptively, based on the rank of the projected second derivative matrix. Iteratively, our variant only requires access to (small-dimensional) projections of first- and second-order problem derivatives and calculates a reduced step inexpensively. The ensuing method maintains the optimal global rate of convergence of (full-dimensional) cubic regularization, while showing improved scalability both theoretically and numerically, particularly when applied to low-rank functions. When applied to the latter, our algorithm naturally adapts the subspace size to the true rank of the function, without knowing it a priori.