On a Randomized Multi-Block ADMM for Solving Selected Machine Learning Problems

TL;DR

This paper introduces a randomized multi-block ADMM algorithm tailored for large-scale convex quadratic optimization problems in machine learning, demonstrating superior efficiency and solution quality compared to existing methods.

Contribution

The paper extends the RAC-MBADMM algorithm to key machine learning problems and proves its linear convergence under standard data assumptions.

Findings

RAC-MBADMM outperforms traditional optimization algorithms in solution time and quality.

The method matches or exceeds the performance of specialized solvers like Glmnet and LIBSVM.

Numerical tests on synthetic and large-scale benchmark problems validate the effectiveness of RAC-MBADMM.

Abstract

The Alternating Direction Method of Multipliers (ADMM) has now days gained tremendous attentions for solving large-scale machine learning and signal processing problems due to the relative simplicity. However, the two-block structure of the classical ADMM still limits the size of the real problems being solved. When one forces a more-than-two-block structure by variable-splitting, the convergence speed slows down greatly as observed in practice. Recently, a randomly assembled cyclic multi-block ADMM (RAC-MBADMM) was developed by the authors for solving general convex and nonconvex quadratic optimization problems where the number of blocks can go greater than two so that each sub-problem has a smaller size and can be solved much more efficiently. In this paper, we apply this method to solving few selected machine learning problems related to convex quadratic optimization, such as Linear Regression, LASSO, Elastic-Net, and SVM. We prove that the algorithm would converge in expectation linearly under the standard statistical data assumptions. We use our general-purpose solver to conduct multiple numerical tests, solving both synthetic and large-scale bench-mark problems. Our results show that RAC-MBADMM could significantly outperform, in both solution time and quality, other optimization algorithms/codes for solving these machine learning problems, and match up the performance of the best tailored methods such as Glmnet or LIBSVM. In certain problem regions RAC-MBADMM even achieves a superior performance than that of the tailored methods.