Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization

TL;DR

This paper introduces an adaptive regularization method for deep learning on heteroskedastic and imbalanced datasets, improving robustness by differentially regularizing uncertain and low-density regions, validated on benchmarks including WebVision.

Contribution

It proposes a novel data-dependent regularization technique that adaptively adjusts regularization strength based on data uncertainty and density, addressing heteroskedasticity and imbalance simultaneously.

Findings

Significant performance improvement over existing methods.

Theoretical support for adaptive regularization in uncertain regions.

Effective on real-world heteroskedastic and imbalanced datasets.

Abstract

Real-world large-scale datasets are heteroskedastic and imbalanced -- labels have varying levels of uncertainty and label distributions are long-tailed. Heteroskedasticity and imbalance challenge deep learning algorithms due to the difficulty of distinguishing among mislabeled, ambiguous, and rare examples. Addressing heteroskedasticity and imbalance simultaneously is under-explored. We propose a data-dependent regularization technique for heteroskedastic datasets that regularizes different regions of the input space differently. Inspired by the theoretical derivation of the optimal regularization strength in a one-dimensional nonparametric classification setting, our approach adaptively regularizes the data points in higher-uncertainty, lower-density regions more heavily. We test our method on several benchmark tasks, including a real-world heteroskedastic and imbalanced dataset, WebVision. Our experiments corroborate our theory and demonstrate a significant improvement over other methods in noise-robust deep learning.