On the adaptive deterministic block coordinate descent methods with momentum for solving large linear least-squares problems

TL;DR

This paper introduces adaptive deterministic block coordinate descent methods with momentum for large linear least-squares problems, improving efficiency and convergence without pre-partitioning or pseudoinverse computations, and extends to highly overdetermined cases with count sketch technology.

Contribution

The paper proposes novel adaptive block coordinate descent algorithms with momentum that eliminate the need for pre-partitioning and pseudoinverse calculations, and integrates count sketch for overdetermined problems.

Findings

The methods converge linearly for full-rank matrices.

They outperform recent methods in CPU time and iteration count.

Count sketch integration enhances performance for overdetermined systems.

Abstract

In this work, we first present an adaptive deterministic block coordinate descent method with momentum (mADBCD) to solve the linear least-squares problem, which is based on Polyak's heavy ball method and a new column selection criterion for a set of block-controlled indices defined by the Euclidean norm of the residual vector of the normal equation. The mADBCD method eliminates the need for pre-partitioning the column indexes of the coefficient matrix, and it also obviates the need to compute the Moore-Penrose pseudoinverse of a column sub-matrix at each iteration. Moreover, we demonstrate the adaptability and flexibility in the automatic selection and updating of the block control index set. When the coefficient matrix has full rank, the theoretical analysis of the mADBCD method indicates that it linearly converges towards the unique solution of the linear least-squares problem. Furthermore, by effectively integrating count sketch technology with the mADBCD method, we also propose a novel count sketch adaptive block coordinate descent method with momentum (CS-mADBCD) for solving highly overdetermined linear least-squares problems and analysis its convergence. Finally, numerical experiments illustrate the advantages of the proposed two methods in terms of both CPU times and iteration counts compared to recent block coordinate descent methods.