DU-VLG: Unifying Vision-and-Language Generation via Dual Sequence-to-Sequence Pre-training

TL;DR

DU-VLG introduces a unified sequence-to-sequence framework for vision-and-language generation, leveraging dual pre-training tasks and a commitment loss to improve image captioning and text-to-image synthesis.

Contribution

It proposes a novel dual pre-training approach and a commitment loss to unify vision-and-language generation tasks within a sequence generation framework.

Findings

Outperforms previous state-of-the-art on three vision-language tasks.

Achieves higher quality image captioning and text-to-image generation.

Human evaluation confirms more relevant and faithful outputs.

Abstract

Due to the limitations of the model structure and pre-training objectives, existing vision-and-language generation models cannot utilize pair-wise images and text through bi-directional generation. In this paper, we propose DU-VLG, a framework which unifies vision-and-language generation as sequence generation problems. DU-VLG is trained with novel dual pre-training tasks: multi-modal denoising autoencoder tasks and modality translation tasks. To bridge the gap between image understanding and generation, we further design a novel commitment loss. We compare pre-training objectives on image captioning and text-to-image generation datasets. Results show that DU-VLG yields better performance than variants trained with uni-directional generation objectives or the variant without the commitment loss. We also obtain higher scores compared to previous state-of-the-art systems on three vision-and-language generation tasks. In addition, human judges further confirm that our model generates real and relevant images as well as faithful and informative captions.