Efficient generative adversarial networks using linear additive-attention Transformers

TL;DR

LadaGAN introduces a linear attention Transformer-based GAN architecture that significantly reduces computational costs while outperforming existing models on benchmarks, making high-quality image generation more accessible.

Contribution

The paper presents LadaGAN, a novel GAN architecture with linear additive-attention Transformers that improves efficiency and stability over traditional and Transformer-based GANs.

Findings

LadaGAN outperforms existing convolutional and Transformer GANs on benchmarks.

LadaGAN is significantly more computationally efficient.

LadaGAN achieves competitive results with less resources.

Abstract

Although the capacity of deep generative models for image generation, such as Diffusion Models (DMs) and Generative Adversarial Networks (GANs), has dramatically improved in recent years, much of their success can be attributed to computationally expensive architectures. This has limited their adoption and use to research laboratories and companies with large resources, while significantly raising the carbon footprint for training, fine-tuning, and inference. In this work, we present a novel GAN architecture which we call LadaGAN. This architecture is based on a linear attention Transformer block named Ladaformer. The main component of this block is a linear additive-attention mechanism that computes a single attention vector per head instead of the quadratic dot-product attention. We employ Ladaformer in both the generator and discriminator, which reduces the computational complexity and overcomes the training instabilities often associated with Transformer GANs. LadaGAN consistently outperforms existing convolutional and Transformer GANs on benchmark datasets at different resolutions while being significantly more efficient. Moreover, LadaGAN shows competitive performance compared to state-of-the-art multi-step generative models (e.g. DMs) using orders of magnitude less computational resources.