Randomized subspace gradient method for constrained optimization

TL;DR

This paper introduces randomized subspace gradient methods for high-dimensional constrained optimization, improving efficiency and solution quality by leveraging random projections and longer step sizes.

Contribution

It develops novel algorithms that project gradients onto random subspaces, enabling better performance in constrained optimization with difficult gradient computations.

Findings

Algorithms can take larger steps than deterministic methods.

Methods are more efficient when gradients are hard to compute.

Numerical results show improved solutions and robustness.

Abstract

We propose randomized subspace gradient methods for high-dimensional constrained optimization. While there have been similarly purposed studies on unconstrained optimization problems, there have been few on constrained optimization problems due to the difficulty of handling constraints. Our algorithms project gradient vectors onto a subspace that is a random projection of the subspace spanned by the gradients of active constraints. We determine the worst-case iteration complexity under linear and nonlinear settings and theoretically confirm that our algorithms can take a larger step size than their deterministic version. From the advantages of taking longer step and randomized subspace gradients, we show that our algorithms are especially efficient in view of time complexity when gradients cannot be obtained easily. Numerical experiments show that they tend to find better solutions because of the randomness of their subspace selection. Furthermore, they performs well in cases where gradients could not be obtained directly, and instead, gradients are obtained using directional derivatives.