Corruption-tolerant Algorithms for Generalized Linear Models

TL;DR

This paper introduces SVAM, a robust algorithm for learning generalized linear models that withstands adversarial label corruption, extending beyond least squares to logistic and gamma regressions with strong theoretical and empirical results.

Contribution

SVAM is a novel variance reduction framework that generalizes robust learning to multiple GLM tasks with provable guarantees and improved empirical performance.

Findings

SVAM achieves superior model recovery guarantees under adversarial corruption.

SVAM outperforms existing methods in robust regression and classification tasks.

The framework is applicable to various GLMs, including logistic and gamma regressions.

Abstract

This paper presents SVAM (Sequential Variance-Altered MLE), a unified framework for learning generalized linear models under adversarial label corruption in training data. SVAM extends to tasks such as least squares regression, logistic regression, and gamma regression, whereas many existing works on learning with label corruptions focus only on least squares regression. SVAM is based on a novel variance reduction technique that may be of independent interest and works by iteratively solving weighted MLEs over variance-altered versions of the GLM objective. SVAM offers provable model recovery guarantees superior to the state-of-the-art for robust regression even when a constant fraction of training labels are adversarially corrupted. SVAM also empirically outperforms several existing problem-specific techniques for robust regression and classification. Code for SVAM is available at https://github.com/purushottamkar/svam/