A Group-Theoretic Framework for Data Augmentation

TL;DR

This paper introduces a mathematical framework for data augmentation, showing it as an averaging over group orbits that reduces variance and improves model training, applicable to various models and symmetry problems.

Contribution

It develops a formal group-theoretic framework for understanding data augmentation, linking it to variance reduction and broadening its theoretical foundation.

Findings

Data augmentation corresponds to averaging over group orbits.

It reduces variance in empirical risk minimization.

Applicable to neural networks and symmetry-based problems.

Abstract

Data augmentation is a widely used trick when training deep neural networks: in addition to the original data, properly transformed data are also added to the training set. However, to the best of our knowledge, a clear mathematical framework to explain the performance benefits of data augmentation is not available. In this paper, we develop such a theoretical framework. We show data augmentation is equivalent to an averaging operation over the orbits of a certain group that keeps the data distribution approximately invariant. We prove that it leads to variance reduction. We study empirical risk minimization, and the examples of exponential families, linear regression, and certain two-layer neural networks. We also discuss how data augmentation could be used in problems with symmetry where other approaches are prevalent, such as in cryo-electron microscopy (cryo-EM).