Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?

TL;DR

This study investigates replacing RNNs with Transformers in text-generating GANs, finding that direct replacement leads to performance issues and mode collapse, indicating the need for architecture adaptation.

Contribution

It provides empirical evidence that Transformers cannot be directly used as drop-in replacements for RNNs in text GANs without modifications.

Findings

Transformers underperform during pre-training when replacing LSTMs.

Mode collapse occurs during GAN tuning with Transformers.

Transformers require adaptation before replacing RNNs in text GANs.

Abstract

In this paper we address the problem of fine-tuned text generation with a limited computational budget. For that, we use a well-performing text generative adversarial network (GAN) architecture - Diversity-Promoting GAN (DPGAN), and attempted a drop-in replacement of the LSTM layer with a self-attention-based Transformer layer in order to leverage their efficiency. The resulting Self-Attention DPGAN (SADPGAN) was evaluated for performance, quality and diversity of generated text and stability. Computational experiments suggested that a transformer architecture is unable to drop-in replace the LSTM layer, under-performing during the pre-training phase and undergoing a complete mode collapse during the GAN tuning phase. Our results suggest that the transformer architecture need to be adapted before it can be used as a replacement for RNNs in text-generating GANs.