Stochastic dual coordinate descent with adaptive heavy ball momentum for linearly constrained convex optimization

TL;DR

This paper introduces an adaptive stochastic dual coordinate descent algorithm with heavy ball momentum for linearly constrained convex optimization, improving scalability and convergence in large-scale problems.

Contribution

It develops a novel adaptive momentum method that learns parameters during iterations, unifying and extending several existing algorithms for constrained convex optimization.

Findings

The method converges linearly in expectation for strongly convex functions.

Numerical experiments confirm the theoretical convergence and effectiveness.

The framework encompasses well-known algorithms like randomized Kaczmarz and conjugate gradient.

Abstract

The problem of finding a solution to the linear system $Ax = b$ with certain minimization properties arises in numerous scientific and engineering areas. In the era of big data, the stochastic optimization algorithms become increasingly significant due to their scalability for problems of unprecedented size. This paper focuses on the problem of minimizing a strongly convex function subject to linear constraints. We consider the dual formulation of this problem and adopt the stochastic coordinate descent to solve it. The proposed algorithmic framework, called adaptive stochastic dual coordinate descent, utilizes sampling matrices sampled from user-defined distributions to extract gradient information. Moreover, it employs Polyak's heavy ball momentum acceleration with adaptive parameters learned through iterations, overcoming the limitation of the heavy ball momentum method that it requires prior knowledge of certain parameters, such as the singular values of a matrix. With these extensions, the framework is able to recover many well-known methods in the context, including the randomized sparse Kaczmarz method, the randomized regularized Kaczmarz method, the linearized Bregman iteration, and a variant of the conjugate gradient (CG) method. Additionally, we introduce an equivalent formulation that, in certain cases, substantially reduces the need for full-dimensional vector operations introduced by the momentum term. We prove that, with strongly admissible objective function, the proposed method converges linearly in expectation. Numerical experiments are provided to confirm our results.