ViTGAN: Training GANs with Vision Transformers

TL;DR

This paper explores integrating Vision Transformers into GANs, introducing new regularization techniques to stabilize training and achieve competitive image generation performance on standard datasets.

Contribution

The paper presents ViTGAN, a novel approach combining ViTs with GANs, including new regularization methods and architectural insights for stable training and high-quality image synthesis.

Findings

ViTGAN achieves performance comparable to CNN-based GANs on CIFAR-10, CelebA, and LSUN datasets.

Novel regularization techniques improve training stability with ViT discriminators.

Architectural choices for ViT generators facilitate convergence during training.

Abstract

Recently, Vision Transformers (ViTs) have shown competitive performance on image recognition while requiring less vision-specific inductive biases. In this paper, we investigate if such performance can be extended to image generation. To this end, we integrate the ViT architecture into generative adversarial networks (GANs). For ViT discriminators, we observe that existing regularization methods for GANs interact poorly with self-attention, causing serious instability during training. To resolve this issue, we introduce several novel regularization techniques for training GANs with ViTs. For ViT generators, we examine architectural choices for latent and pixel mapping layers to facilitate convergence. Empirically, our approach, named ViTGAN, achieves comparable performance to the leading CNN-based GAN models on three datasets: CIFAR-10, CelebA, and LSUN bedroom.