Online and Offline Robust Multivariate Linear Regression

TL;DR

This paper introduces robust online and offline algorithms for multivariate Gaussian linear regression, demonstrating improved robustness and computational efficiency over classical methods, with theoretical guarantees and practical implementation.

Contribution

It proposes novel robust stochastic gradient and fix-point algorithms for multivariate regression, with theoretical asymptotic normality and practical robustness improvements.

Findings

Robust algorithms outperform classical least squares in noisy data.

Online algorithms are computationally efficient and scalable.

The methods are implemented in an accessible R package.

Abstract

We consider the robust estimation of the parameters of multivariate Gaussian linear regression models. To this aim we consider robust version of the usual (Mahalanobis) least-square criterion, with or without Ridge regularization. We introduce two methods each considered contrast: (i) online stochastic gradient descent algorithms and their averaged versions and (ii) offline fix-point algorithms. Under weak assumptions, we prove the asymptotic normality of the resulting estimates. Because the variance matrix of the noise is usually unknown, we propose to plug a robust estimate of it in the Mahalanobis-based stochastic gradient descent algorithms. We show, on synthetic data, the dramatic gain in terms of robustness of the proposed estimates as compared to the classical least-square ones. Well also show the computational efficiency of the online versions of the proposed algorithms. All the proposed algorithms are implemented in the R package RobRegression available on CRAN.