ScoreMix: A Scalable Augmentation Strategy for Training GANs with Limited Data

TL;DR

ScoreMix is a scalable data augmentation method that improves GAN training on limited data by generating diverse samples close to the data manifold through score-based optimization, leading to better synthesis quality.

Contribution

We introduce ScoreMix, a novel augmentation strategy that uses score-based sample optimization to enhance GAN training with limited data, without requiring hyperparameter tuning.

Findings

ScoreMix significantly reduces overfitting in GANs trained on limited data.

GANs with ScoreMix achieve higher quality and diversity in generated images.

The method is easy to integrate into existing GAN frameworks.

Abstract

Generative Adversarial Networks (GANs) typically suffer from overfitting when limited training data is available. To facilitate GAN training, current methods propose to use data-specific augmentation techniques. Despite the effectiveness, it is difficult for these methods to scale to practical applications. In this work, we present ScoreMix, a novel and scalable data augmentation approach for various image synthesis tasks. We first produce augmented samples using the convex combinations of the real samples. Then, we optimize the augmented samples by minimizing the norms of the data scores, i.e., the gradients of the log-density functions. This procedure enforces the augmented samples close to the data manifold. To estimate the scores, we train a deep estimation network with multi-scale score matching. For different image synthesis tasks, we train the score estimation network using different data. We do not require the tuning of the hyperparameters or modifications to the network architecture. The ScoreMix method effectively increases the diversity of data and reduces the overfitting problem. Moreover, it can be easily incorporated into existing GAN models with minor modifications. Experimental results on numerous tasks demonstrate that GAN models equipped with the ScoreMix method achieve significant improvements.