An End-to-End Model for Photo-Sharing Multi-modal Dialogue Generation

TL;DR

This paper introduces the first end-to-end model for photo-sharing multi-modal dialogue generation, integrating visual perception and image generation within a large language model to improve performance over pipeline approaches.

Contribution

The paper presents a novel end-to-end architecture that combines an image perceptron and image generator with a large language model, enabling gradient flow and improved multi-modal dialogue generation.

Findings

Achieves state-of-the-art results on PhotoChat and DialogCC datasets.

Outperforms pipeline models in text and image generation metrics.

Validates the effectiveness of end-to-end training for multi-modal dialogue tasks.

Abstract

Photo-Sharing Multi-modal dialogue generation requires a dialogue agent not only to generate text responses but also to share photos at the proper moment. Using image text caption as the bridge, a pipeline model integrates an image caption model, a text generation model, and an image generation model to handle this complex multi-modal task. However, representing the images with text captions may loss important visual details and information and cause error propagation in the complex dialogue system. Besides, the pipeline model isolates the three models separately because discrete image text captions hinder end-to-end gradient propagation. We propose the first end-to-end model for photo-sharing multi-modal dialogue generation, which integrates an image perceptron and an image generator with a large language model. The large language model employs the Q-Former to perceive visual images in the input end. For image generation in the output end, we propose a dynamic vocabulary transformation matrix and use straight-through and gumbel-softmax techniques to align the large language model and stable diffusion model and achieve end-to-end gradient propagation. We perform experiments on PhotoChat and DialogCC datasets to evaluate our end-to-end model. Compared with pipeline models, the end-to-end model gains state-of-the-art performances on various metrics of text and image generation. More analysis experiments also verify the effectiveness of the end-to-end model for photo-sharing multi-modal dialogue generation.