Randomized Subspace Nesterov Accelerated Gradient

TL;DR

This paper introduces randomized-subspace Nesterov accelerated gradient methods that leverage low-dimensional projected-gradient information to improve optimization efficiency under matrix smoothness assumptions.

Contribution

It develops novel accelerated gradient methods for subspace sketches, providing theoretical guarantees and a unified framework for comparing sketch families.

Findings

Establishes accelerated oracle-complexity guarantees for the methods.

Shows how matrix smoothness and sketch distribution affect complexity.

Provides a basis for when subspace acceleration outperforms full-dimensional methods.

Abstract

Randomized-subspace methods reduce the cost of first-order optimization by using only low-dimensional projected-gradient information, a feature that is attractive in forward-mode automatic differentiation and communication-limited settings. While Nesterov acceleration is well understood for full-gradient and coordinate-based methods, obtaining accelerated methods for general subspace sketches that use only projected-gradient information and can improve over full-dimensional Nesterov acceleration in oracle complexity is technically nontrivial. We develop randomized-subspace Nesterov accelerated gradient methods for smooth convex and smooth strongly convex optimization under matrix smoothness and generic sketch moment assumptions. The key technical ingredient is a three-sequence formulation tailored to matrix smoothness, which recovers the corresponding classical Nesterov methods in the full-dimensional case. The resulting theory establishes accelerated oracle-complexity guarantees and makes explicit how matrix smoothness and the sketch distribution enter the complexity. It also provides a unified basis for comparing sketch families and identifying when randomized-subspace acceleration improves over full-dimensional Nesterov acceleration in oracle complexity.