OptAGAN: Entropy-based finetuning on text VAE-GAN

TL;DR

This paper introduces OptAGAN, a novel approach combining VAE-GANs with entropy-based reinforcement learning to improve text generation quality and diversity, leveraging pre-trained models BERT and GPT-2.

Contribution

It presents a new method for fine-tuning text VAE-GANs with entropy-based RL, enhancing text quality and diversity beyond existing models.

Findings

Significant improvement in text quality over state-of-the-art methods.

Enhanced diversity in generated texts due to entropy-based rewards.

Effective modeling of high-level sentence features and low-level word generation.

Abstract

Transfer learning through large pre-trained models has changed the landscape of current applications in natural language processing (NLP). Recently Optimus, a variational autoencoder (VAE) which combines two pre-trained models, BERT and GPT-2, has been released, and its combination with generative adversial networks (GANs) has been shown to produce novel, yet very human-looking text. The Optimus and GANs combination avoids the troublesome application of GANs to the discrete domain of text, and prevents the exposure bias of standard maximum likelihood methods. We combine the training of GANs in the latent space, with the finetuning of the decoder of Optimus for single word generation. This approach lets us model both the high-level features of the sentences, and the low-level word-by-word generation. We finetune using reinforcement learning (RL) by exploiting the structure of GPT-2 and by adding entropy-based intrinsically motivated rewards to balance between quality and diversity. We benchmark the results of the VAE-GAN model, and show the improvements brought by our RL finetuning on three widely used datasets for text generation, with results that greatly surpass the current state-of-the-art for the quality of the generated texts.