Table-to-Text Generation with Pretrained Diffusion Models

TL;DR

This paper systematically explores the application of pretrained diffusion models to the table-to-text generation task, analyzing various training aspects, sampling strategies, and comparison with auto-regressive models, demonstrating diffusion models' promising balance of quality and diversity.

Contribution

It introduces the adaptation of diffusion models to table-to-text generation and provides an in-depth analysis of training, sampling, and aggregation methods, highlighting their effectiveness.

Findings

Diffusion models achieve comparable results to auto-regressive models in table-to-text generation.

Using MBR aggregation with strict length constraints yields high-quality outputs.

Fast samplers like DPM-Solver++ can accelerate generation with some trade-offs.

Abstract

Diffusion models have demonstrated significant potential in achieving state-of-the-art performance across various text generation tasks. In this systematic study, we investigate their application to the table-to-text problem by adapting the diffusion model to the task and conducting an in-depth analysis. Our experiments cover multiple aspects of diffusion models training. We explore sampling strategy influence by inducing recent diffusion model accelerator DPM-Solver++ into our core model. We have tested different prediction aggregation methods, like ROVER and Minimum Bayes-Risk (MBR). Our studies cover the impact of the pre-training phase in diffusion models and the generation length constraints influence. We also have compared diffusion model generation with auto-regressive text-to-text models with different temperature settings for diversity evaluation. Our key observation is that diffusion models demonstrate the balance between quality and diversity while auto-regressive text-to-text models are not successful at handling both at the same time. Furthermore, we found out that to achieve the highest quality possible, it is preferable to use a regular sampler with the strictest length constraint to create multiple samples, and then use MBR to aggregate the predictions. However, if you are prepared to give up high level of diversity and to accelerate the process, you can also utilize a fast sampler DPM-Solver++. Our findings reveal that diffusion models achieve comparable results in the table-to-text domain, highlighting their viability in the table-to-text challenge as a promising research direction.