Modern Methods for Text Generation

TL;DR

This paper reviews modern Transformer-based models like BERT and GPT-2, highlighting their architecture and comparing their effectiveness in text generation tasks such as translation and summarization.

Contribution

It provides an analysis and comparison of BERT and GPT-2, emphasizing their performance in text generation and understanding of sequential data.

Findings

Transformers improve understanding of sequential data

BERT and GPT-2 excel in text classification and translation

Comparison shows differences in output quality for text generation

Abstract

Synthetic text generation is challenging and has limited success. Recently, a new architecture, called Transformers, allow machine learning models to understand better sequential data, such as translation or summarization. BERT and GPT-2, using Transformers in their cores, have shown a great performance in tasks such as text classification, translation and NLI tasks. In this article, we analyse both algorithms and compare their output quality in text generation tasks.