Parallelly Tempered Generative Adversarial Nets: Toward Stabilized Gradients

TL;DR

This paper introduces a novel parallel tempering approach for GAN training that stabilizes gradients and mitigates mode collapse by leveraging tempered distributions, improving data synthesis quality and enabling fair data generation.

Contribution

The work proposes a new GAN training framework using parallel tempering with tempered distributions, reducing gradient variance and enhancing training stability compared to existing methods.

Findings

Outperforms existing training strategies in image and tabular data synthesis

Theoretically reduces gradient variance through tempered distributions

Enables fair synthetic data generation for trustworthy AI

Abstract

A generative adversarial network (GAN) has been a representative backbone model in generative artificial intelligence (AI) because of its powerful performance in capturing intricate data-generating processes. However, the GAN training is well-known for its notorious training instability, usually characterized by the occurrence of mode collapse. Through the lens of gradients' variance, this work particularly analyzes the training instability and inefficiency in the presence of mode collapse by linking it to multimodality in the target distribution. To ease the raised training issues from severe multimodality, we introduce a novel GAN training framework that leverages a series of tempered distributions produced via convex interpolation. With our newly developed GAN objective function, the generator can learn all the tempered distributions simultaneously, conceptually resonating with the parallel tempering in statistics. Our simulation studies demonstrate the superiority of our approach over existing popular training strategies in both image and tabular data synthesis. We theoretically analyze that such significant improvement can arise from reducing the variance of gradient estimates by using the tempered distributions. Finally, we further develop a variant of the proposed framework aimed at generating fair synthetic data which is one of the growing interests in the field of trustworthy AI.