A Primer on Coordinate Descent Algorithms

TL;DR

This paper introduces coordinate descent algorithms, highlighting their effectiveness for large-scale optimization problems in machine learning and data science, with practical guidance for implementation.

Contribution

It provides a comprehensive overview of coordinate descent algorithms, including theory and practical examples, tailored for practitioners in data science and engineering.

Findings

Effective for large-scale optimization tasks

Suitable for parallel and distributed computing

Applicable across machine learning, image processing, and statistics

Abstract

This monograph presents a class of algorithms called coordinate descent algorithms for mathematicians, statisticians, and engineers outside the field of optimization. This particular class of algorithms has recently gained popularity due to their effectiveness in solving large-scale optimization problems in machine learning, compressed sensing, image processing, and computational statistics. Coordinate descent algorithms solve optimization problems by successively minimizing along each coordinate or coordinate hyperplane, which is ideal for parallelized and distributed computing. Avoiding detailed technicalities and proofs, this monograph gives relevant theory and examples for practitioners to effectively apply coordinate descent to modern problems in data science and engineering.