Reweighting Augmented Samples by Minimizing the Maximal Expected Loss

TL;DR

This paper introduces a novel reweighting strategy for augmented samples in deep learning, focusing on minimizing the worst-case expected loss to improve model generalization across various data augmentation techniques.

Contribution

It proposes a new loss minimization approach inspired by adversarial training that assigns higher weights to harder augmented samples, enhancing generalization.

Findings

Improves generalization on natural language understanding tasks.

Enhances performance in image classification with common augmentations.

Applicable on top of existing data augmentation methods.

Abstract

Data augmentation is an effective technique to improve the generalization of deep neural networks. However, previous data augmentation methods usually treat the augmented samples equally without considering their individual impacts on the model. To address this, for the augmented samples from the same training example, we propose to assign different weights to them. We construct the maximal expected loss which is the supremum over any reweighted loss on augmented samples. Inspired by adversarial training, we minimize this maximal expected loss (MMEL) and obtain a simple and interpretable closed-form solution: more attention should be paid to augmented samples with large loss values (i.e., harder examples). Minimizing this maximal expected loss enables the model to perform well under any reweighting strategy. The proposed method can generally be applied on top of any data augmentation methods. Experiments are conducted on both natural language understanding tasks with token-level data augmentation, and image classification tasks with commonly-used image augmentation techniques like random crop and horizontal flip. Empirical results show that the proposed method improves the generalization performance of the model.