Curvature Enhanced Data Augmentation for Regression

TL;DR

This paper introduces CEMS, a curvature-enhanced data augmentation method for regression that leverages second-order manifold information to generate synthetic data, improving model generalization with minimal computational cost.

Contribution

It presents a novel second-order manifold sampling technique for data augmentation in regression, extending previous first-order approaches with theoretical and practical advancements.

Findings

CEMS outperforms state-of-the-art methods in various datasets.

CEMS improves generalization in both in-distribution and out-of-distribution scenarios.

CEMS requires minimal additional computational resources.

Abstract

Deep learning models with a large number of parameters, often referred to as over-parameterized models, have achieved exceptional performance across various tasks. Despite concerns about overfitting, these models frequently generalize well to unseen data, thanks to effective regularization techniques, with data augmentation being among the most widely used. While data augmentation has shown great success in classification tasks using label-preserving transformations, its application in regression problems has received less attention. Recently, a novel \emph{manifold learning} approach for generating synthetic data was proposed, utilizing a first-order approximation of the data manifold. Building on this foundation, we present a theoretical framework and practical tools for approximating and sampling general data manifolds. Furthermore, we introduce the Curvature-Enhanced Manifold Sampling (CEMS) method for regression tasks. CEMS leverages a second-order representation of the data manifold to enable efficient sampling and reconstruction of new data points. Extensive evaluations across multiple datasets and comparisons with state-of-the-art methods demonstrate that CEMS delivers superior performance in both in-distribution and out-of-distribution scenarios, while introducing only minimal computational overhead. Code is available at https://github.com/azencot-group/CEMS.