First-Order Manifold Data Augmentation for Regression Learning

TL;DR

FOMA introduces a novel domain-independent data augmentation technique for regression tasks, sampling from tangent planes to improve neural network generalization and robustness, outperforming mixup baselines.

Contribution

The paper proposes FOMA, a new data-driven augmentation method for regression that samples from tangent planes, enhancing generalization and robustness across neural architectures.

Findings

FOMA improves in-distribution generalization.

FOMA enhances out-of-distribution robustness.

FOMA outperforms mixup baselines.

Abstract

Data augmentation (DA) methods tailored to specific domains generate synthetic samples by applying transformations that are appropriate for the characteristics of the underlying data domain, such as rotations on images and time warping on time series data. In contrast, domain-independent approaches, e.g. mixup, are applicable to various data modalities, and as such they are general and versatile. While regularizing classification tasks via DA is a well-explored research topic, the effect of DA on regression problems received less attention. To bridge this gap, we study the problem of domain-independent augmentation for regression, and we introduce FOMA: a new data-driven domain-independent data augmentation method. Essentially, our approach samples new examples from the tangent planes of the train distribution. Augmenting data in this way aligns with the network tendency towards capturing the dominant features of its input signals. We evaluate FOMA on in-distribution generalization and out-of-distribution robustness benchmarks, and we show that it improves the generalization of several neural architectures. We also find that strong baselines based on mixup are less effective in comparison to our approach. Our code is publicly available athttps://github.com/azencot-group/FOMA.