Robust Sampling in Deep Learning

TL;DR

This paper introduces a novel regularization technique for deep learning based on distributional robust optimization, which emphasizes poorly performing samples during training to enhance convergence and accuracy.

Contribution

It proposes a new regularization method that adjusts sample contributions based on their accuracy, improving training efficiency and model performance.

Findings

Can accelerate convergence in certain scenarios

Increases model accuracy by focusing on worst-performing samples

Effective in reducing overfitting through sample re-weighting

Abstract

Deep learning requires regularization mechanisms to reduce overfitting and improve generalization. We address this problem by a new regularization method based on distributional robust optimization. The key idea is to modify the contribution from each sample for tightening the empirical risk bound. During the stochastic training, the selection of samples is done according to their accuracy in such a way that the worst performed samples are the ones that contribute the most in the optimization. We study different scenarios and show the ones where it can make the convergence faster or increase the accuracy.