IMAE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matters

TL;DR

This paper analyzes the limitations of MAE in noise-robust learning, revealing its unequal treatment of examples and proposing IMAE, which adjusts gradient magnitude variance to improve robustness and fitting ability.

Contribution

The paper uncovers MAE's underfitting and unequal example treatment issues and introduces IMAE, a simple variance adjustment method that enhances noise robustness and model fitting.

Findings

MAE is theoretically noise-robust but underfits in practice.

MAE emphasizes uncertain examples, not equal treatment.

IMAE improves fitting ability while maintaining noise robustness.

Abstract

In this work, we study robust deep learning against abnormal training data from the perspective of example weighting built in empirical loss functions, i.e., gradient magnitude with respect to logits, an angle that is not thoroughly studied so far. Consequently, we have two key findings: (1) Mean Absolute Error (MAE) Does Not Treat Examples Equally. We present new observations and insightful analysis about MAE, which is theoretically proved to be noise-robust. First, we reveal its underfitting problem in practice. Second, we analyse that MAE's noise-robustness is from emphasising on uncertain examples instead of treating training samples equally, as claimed in prior work. (2) The Variance of Gradient Magnitude Matters. We propose an effective and simple solution to enhance MAE's fitting ability while preserving its noise-robustness. Without changing MAE's overall weighting scheme, i.e., what examples get higher weights, we simply change its weighting variance non-linearly so that the impact ratio between two examples are adjusted. Our solution is termed Improved MAE (IMAE). We prove IMAE's effectiveness using extensive experiments: image classification under clean labels, synthetic label noise, and real-world unknown noise.