GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution

TL;DR

This paper explores the use of Gumbel-softmax distributions within GANs to effectively generate sequences of discrete elements, addressing the challenge of non-differentiability in discrete data generation.

Contribution

It introduces a method combining GANs with Gumbel-softmax to improve sequence generation of discrete data, which was previously difficult due to non-differentiability.

Findings

Gumbel-softmax enables differentiable sampling of discrete sequences.

Recurrent neural network-based GANs with Gumbel-softmax outperform traditional methods.

The approach effectively generates realistic discrete sequences.

Abstract

Generative Adversarial Networks (GAN) have limitations when the goal is to generate sequences of discrete elements. The reason for this is that samples from a distribution on discrete objects such as the multinomial are not differentiable with respect to the distribution parameters. This problem can be avoided by using the Gumbel-softmax distribution, which is a continuous approximation to a multinomial distribution parameterized in terms of the softmax function. In this work, we evaluate the performance of GANs based on recurrent neural networks with Gumbel-softmax output distributions in the task of generating sequences of discrete elements.