Smoothly Giving up: Robustness for Simple Models

TL;DR

This paper introduces a margin-based alpha-loss that smoothly transitions between convex and non-convex losses, enabling simple models like logistic regression and boosting to be more robust against label noise, with demonstrated effectiveness on real datasets.

Contribution

It proposes a novel alpha-loss function that adaptively balances convexity and non-convexity to improve robustness of simple models against noisy labels.

Findings

Enhanced robustness of logistic regression and boosting to label noise.

Effective performance on COVID-19 survey and Long-Servedio datasets.

Smooth transition between convex and non-convex losses improves training outcomes.

Abstract

There is a growing need for models that are interpretable and have reduced energy and computational cost (e.g., in health care analytics and federated learning). Examples of algorithms to train such models include logistic regression and boosting. However, one challenge facing these algorithms is that they provably suffer from label noise; this has been attributed to the joint interaction between oft-used convex loss functions and simpler hypothesis classes, resulting in too much emphasis being placed on outliers. In this work, we use the margin-based $\alpha$-loss, which continuously tunes between canonical convex and quasi-convex losses, to robustly train simple models. We show that the $\alpha$ hyperparameter smoothly introduces non-convexity and offers the benefit of "giving up" on noisy training examples. We also provide results on the Long-Servedio dataset for boosting and a COVID-19 survey dataset for logistic regression, highlighting the efficacy of our approach across multiple relevant domains.