Learning with Bad Training Data via Iterative Trimmed Loss Minimization

TL;DR

This paper introduces an iterative trimmed loss minimization framework that effectively learns from corrupted training data by focusing on samples with the lowest current loss, demonstrating strong theoretical and empirical results across various settings.

Contribution

It proposes a novel iterative method for robust learning from corrupted data and provides theoretical guarantees of convergence, outperforming existing approaches in multiple scenarios.

Findings

Recovers ground truth with linear convergence in generalized linear models.

Achieves state-of-the-art results on label noise without prior clean data.

Effective against adversarial training data, including backdoor attacks.

Abstract

In this paper, we study a simple and generic framework to tackle the problem of learning model parameters when a fraction of the training samples are corrupted. We first make a simple observation: in a variety of such settings, the evolution of training accuracy (as a function of training epochs) is different for clean and bad samples. Based on this we propose to iteratively minimize the trimmed loss, by alternating between (a) selecting samples with lowest current loss, and (b) retraining a model on only these samples. We prove that this process recovers the ground truth (with linear convergence rate) in generalized linear models with standard statistical assumptions. Experimentally, we demonstrate its effectiveness in three settings: (a) deep image classifiers with errors only in labels, (b) generative adversarial networks with bad training images, and (c) deep image classifiers with adversarial (image, label) pairs (i.e., backdoor attacks). For the well-studied setting of random label noise, our algorithm achieves state-of-the-art performance without having access to any a-priori guaranteed clean samples.