End-to-End Multimodal Representation Learning for Video Dialog

TL;DR

This paper introduces an end-to-end multimodal learning framework combining 3D-CNN and transformers to better utilize visual features in video dialog tasks, leading to improved performance.

Contribution

It proposes a novel visual encoder that integrates 3D-CNN and transformer models for enhanced video feature extraction in dialog systems.

Findings

Significant performance improvements on AVSD benchmark.

Better utilization of visual cues over previous models.

Enhanced robustness of semantic video representations.

Abstract

Video-based dialog task is a challenging multimodal learning task that has received increasing attention over the past few years with state-of-the-art obtaining new performance records. This progress is largely powered by the adaptation of the more powerful transformer-based language encoders. Despite this progress, existing approaches do not effectively utilize visual features to help solve tasks. Recent studies show that state-of-the-art models are biased toward textual information rather than visual cues. In order to better leverage the available visual information, this study proposes a new framework that combines 3D-CNN network and transformer-based networks into a single visual encoder to extract more robust semantic representations from videos. The visual encoder is jointly trained end-to-end with other input modalities such as text and audio. Experiments on the AVSD task show significant improvement over baselines in both generative and retrieval tasks.