Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

TL;DR

This paper introduces a universal multimodal transformer that effectively integrates video and audio data for scene-aware dialogue generation, achieving top performance in the AVSD task.

Contribution

It presents a novel multimodal transformer with multi-task learning for joint representation and response generation in video-audio dialogue tasks.

Findings

Achieved the best performance in DSTC8 AVSD challenge.

Extended pre-trained language models to multimodal dialogue generation.

Demonstrated effectiveness of multi-task learning for joint modality representations.

Abstract

Audio-Visual Scene-Aware Dialog (AVSD) is a task to generate responses when chatting about a given video, which is organized as a track of the 8th Dialog System Technology Challenge (DSTC8). To solve the task, we propose a universal multimodal transformer and introduce the multi-task learning method to learn joint representations among different modalities as well as generate informative and fluent responses. Our method extends the natural language generation pre-trained model to multimodal dialogue generation task. Our system achieves the best performance in both objective and subjective evaluations in the challenge.