Robust High-Dimensional Linear Regression

TL;DR

This paper introduces a robust high-dimensional linear regression method that relaxes traditional assumptions, leveraging low-rank matrix approximation and principal component regression to improve robustness and performance.

Contribution

It presents an integrated approach combining robust low-rank matrix approximation and principal component regression, relaxing prior assumptions for better robustness.

Findings

Outperforms state-of-the-art methods in prediction error

Achieves faster running times

Provides strong theoretical performance guarantees

Abstract

The effectiveness of supervised learning techniques has made them ubiquitous in research and practice. In high-dimensional settings, supervised learning commonly relies on dimensionality reduction to improve performance and identify the most important factors in predicting outcomes. However, the economic importance of learning has made it a natural target for adversarial manipulation of training data, which we term poisoning attacks. Prior approaches to dealing with robust supervised learning rely on strong assumptions about the nature of the feature matrix, such as feature independence and sub-Gaussian noise with low variance. We propose an integrated method for robust regression that relaxes these assumptions, assuming only that the feature matrix can be well approximated by a low-rank matrix. Our techniques integrate improved robust low-rank matrix approximation and robust principle component regression, and yield strong performance guarantees. Moreover, we experimentally show that our methods significantly outperform state of the art both in running time and prediction error.