Robust Linear Predictions: Analyses of Uniform Concentration, Fast Rates and Model Misspecification

TL;DR

This paper develops a unified robust framework for linear prediction in Hilbert spaces using Median of Means, achieving near-optimal error rates that are robust to outliers and model misspecification.

Contribution

It introduces a comprehensive robust estimation method that does not assume data distribution or support compactness, providing error bounds and conditions for fast rates.

Findings

Achieves error rate of O(max{|O|^{1/2} n^{-1/2}, |I|^{1/2} n^{-1}) + ε

Error rates are slightly slower than classical but robust to outliers

Fast rates are attainable under additional assumptions

Abstract

The problem of linear predictions has been extensively studied for the past century under pretty generalized frameworks. Recent advances in the robust statistics literature allow us to analyze robust versions of classical linear models through the prism of Median of Means (MoM). Combining these approaches in a piecemeal way might lead to ad-hoc procedures, and the restricted theoretical conclusions that underpin each individual contribution may no longer be valid. To meet these challenges coherently, in this study, we offer a unified robust framework that includes a broad variety of linear prediction problems on a Hilbert space, coupled with a generic class of loss functions. Notably, we do not require any assumptions on the distribution of the outlying data points ($\mathcal{O}$) nor the compactness of the support of the inlying ones ($\mathcal{I}$). Under mild conditions on the dual norm, we show that for misspecification level $\epsilon$, these estimators achieve an error rate of $O(\max\left\{|\mathcal{O}|^{1/2}n^{-1/2}, |\mathcal{I}|^{1/2}n^{-1} \right\}+\epsilon)$, matching the best-known rates in literature. This rate is slightly slower than the classical rates of $O(n^{-1/2})$, indicating that we need to pay a price in terms of error rates to obtain robust estimates. Additionally, we show that this rate can be improved to achieve so-called "fast rates" under additional assumptions.