Small-GAN: Speeding Up GAN Training Using Core-sets

TL;DR

Small-GAN introduces a coreset-based batch compression technique that accelerates GAN training, reduces memory usage, and improves mode coverage and anomaly detection performance.

Contribution

The paper presents a novel coreset-selection method for creating effectively large batches from small samples, enhancing GAN training efficiency and effectiveness.

Findings

Reduces training time and memory consumption for GANs.

Decreases mode dropping in synthetic datasets.

Achieves state-of-the-art results in anomaly detection.

Abstract

Recent work by Brock et al. (2018) suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. Unfortunately, using large batches is slow and expensive on conventional hardware. Thus, it would be nice if we could generate batches that were effectively large though actually small. In this work, we propose a method to do this, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of 'real' images, we create a cached dataset of Inception activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected activations at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it allows GANs to reach a new state of the art in anomaly detection.