A Mathematics Framework of Artificial Shifted Population Risk and Its Further Understanding Related to Consistency Regularization

TL;DR

This paper develops a mathematical framework linking data augmentation and consistency regularization, revealing the impact of shifted population risk on training and proposing methods to improve neural network generalization and stability.

Contribution

It introduces a comprehensive mathematical framework for data augmentation, clarifies the relationship with consistency regularization, and proposes mitigation strategies for training issues.

Findings

The shifted population risk can be decomposed into original risk and a gap term.

The gap term negatively affects early training stages.

Proposed methods improve generalization and convergence stability.

Abstract

Data augmentation is an important technique in training deep neural networks as it enhances their ability to generalize and remain robust. While data augmentation is commonly used to expand the sample size and act as a consistency regularization term, there is a lack of research on the relationship between them. To address this gap, this paper introduces a more comprehensive mathematical framework for data augmentation. Through this framework, we establish that the expected risk of the shifted population is the sum of the original population risk and a gap term, which can be interpreted as a consistency regularization term. The paper also provides a theoretical understanding of this gap, highlighting its negative effects on the early stages of training. We also propose a method to mitigate these effects. To validate our approach, we conducted experiments using same data augmentation techniques and computing resources under several scenarios, including standard training, out-of-distribution, and imbalanced classification. The results demonstrate that our methods surpass compared methods under all scenarios in terms of generalization ability and convergence stability. We provide our code implementation at the following link: https://github.com/ydlsfhll/ASPR.