Text-to-Text Pre-Training for Data-to-Text Tasks

TL;DR

This paper demonstrates that text-to-text pre-training with T5 significantly improves data-to-text generation, outperforming other models and enhancing generalization, especially on out-of-domain data.

Contribution

The study introduces T5-based pre-training for data-to-text tasks, showing it surpasses previous architectures and language models in performance and generalization.

Findings

T5 pre-training outperforms pipelined neural architectures.

T5 achieves better out-of-domain generalization.

Pre-training enhances transfer learning for data-to-text tasks.

Abstract

We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5, enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternative language model based pre-training techniques such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-of-domain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.