Randomized Fast Subspace Descent Methods

TL;DR

The paper introduces Randomized Fast Subspace Descent (RFASD), an efficient optimization method that combines space decomposition and randomized subspace selection to improve convergence rates for convex problems.

Contribution

It develops RFASD, a novel method that accelerates convergence by leveraging stable space decompositions and probabilistic subspace selection, outperforming existing block coordinate methods.

Findings

RFASD converges sublinearly for convex functions.

RFASD converges linearly for strongly convex functions.

RFASD outperforms randomized block coordinate descent when the condition number is small.

Abstract

Randomized Fast Subspace Descent (RFASD) Methods are developed and analyzed for smooth and non-constraint convex optimization problems. The efficiency of the method relies on a space decomposition which is stable in $A$-norm, and meanwhile, the condition number $\kappa_A$ measured in $A$-norm is small. At each iteration, the subspace is chosen randomly either uniformly or by a probability proportional to the local Lipschitz constants. Then in each chosen subspace, a preconditioned gradient descent method is applied. RFASD converges sublinearly for convex functions and linearly for strongly convex functions. Comparing with the randomized block coordinate descent methods, the convergence of RFASD is faster provided $\kappa_A$ is small and the subspace decomposition is $A$-stable. This improvement is supported by considering a multilevel space decomposition for Nesterov's `worst' problem.