Coordinate Descent Algorithm for Least Absolute Deviations Regression

TL;DR

This paper introduces a coordinate descent algorithm for LAD regression that is scalable, stable, and easy to implement, effectively handling high-dimensional data and surpassing traditional methods in efficiency.

Contribution

The paper presents a novel coordinate descent method for LAD regression that avoids matrix inversion and efficiently handles high-dimensional data with provable convergence.

Findings

Matches accuracy of linear-programming LAD solvers

Offers improved scalability and stability in high-dimensional settings

Effective even when predictors outnumber observations

Abstract

Least Absolute Deviations (LAD) regression provides a robust alternative to ordinary least squares by minimizing the sum of absolute residuals. However, its widespread use has been limited by the computational cost of existing solvers, particularly simplex-based methods in high-dimensional settings. We propose a coordinate descent algorithm for LAD regression that avoids matrix inversion, naturally accommodates the non-differentiability of the objective function, and remains well-defined even when the number of predictors exceeds the number of observations. The key observation is that each coordinate update reduces to a one-dimensional minimization admitting a closed-form solution given by a median or weighted median. The resulting algorithm has per-iteration complexity $O(p\,n \log n)$ and is provably convergent due to the convexity of the LAD objective and the exactness of each coordinate update. Experiments on synthetic and real datasets show that the method matches the accuracy of linear-programming-based LAD solvers while offering improved scalability and stability in high-dimensional regimes, including cases where $p \ge n$. The method is easy to implement, requires no specialized optimization software, and provides a practical tool for robust linear models.