Regularizing Neural Networks with Meta-Learning Generative Models

TL;DR

This paper introduces Meta Generative Regularization (MGR), a novel method that uses meta-learning to select synthetic samples for regularization, improving the effectiveness of generative data augmentation in small dataset scenarios.

Contribution

The paper proposes MGR, a new strategy that dynamically selects synthetic samples for regularization via meta-learning, reducing degradation and enhancing performance in data augmentation.

Findings

MGR prevents performance degradation in generative data augmentation.

MGR outperforms baseline methods on six datasets.

MGR is especially effective with small datasets.

Abstract

This paper investigates methods for improving generative data augmentation for deep learning. Generative data augmentation leverages the synthetic samples produced by generative models as an additional dataset for classification with small dataset settings. A key challenge of generative data augmentation is that the synthetic data contain uninformative samples that degrade accuracy. This is because the synthetic samples do not perfectly represent class categories in real data and uniform sampling does not necessarily provide useful samples for tasks. In this paper, we present a novel strategy for generative data augmentation called meta generative regularization (MGR). To avoid the degradation of generative data augmentation, MGR utilizes synthetic samples in the regularization term for feature extractors instead of in the loss function, e.g., cross-entropy. These synthetic samples are dynamically determined to minimize the validation losses through meta-learning. We observed that MGR can avoid the performance degradation of na\"ive generative data augmentation and boost the baselines. Experiments on six datasets showed that MGR is effective particularly when datasets are smaller and stably outperforms baselines.