Modality Shifting Attention Network for Multi-modal Video Question Answering

TL;DR

This paper introduces MSAN, a novel network for multimodal video question answering that dynamically shifts focus between modalities for localization and answer prediction, achieving state-of-the-art results.

Contribution

The paper proposes MSAN, which decomposes MVQA into localization and answer prediction, with a modality shifting mechanism that improves performance.

Findings

MSAN achieves 71.13% accuracy on TVQA dataset.

MSAN outperforms previous state-of-the-art methods.

Extensive ablation studies validate the effectiveness of MSAN components.

Abstract

This paper considers a network referred to as Modality Shifting Attention Network (MSAN) for Multimodal Video Question Answering (MVQA) task. MSAN decomposes the task into two sub-tasks: (1) localization of temporal moment relevant to the question, and (2) accurate prediction of the answer based on the localized moment. The modality required for temporal localization may be different from that for answer prediction, and this ability to shift modality is essential for performing the task. To this end, MSAN is based on (1) the moment proposal network (MPN) that attempts to locate the most appropriate temporal moment from each of the modalities, and also on (2) the heterogeneous reasoning network (HRN) that predicts the answer using an attention mechanism on both modalities. MSAN is able to place importance weight on the two modalities for each sub-task using a component referred to as Modality Importance Modulation (MIM). Experimental results show that MSAN outperforms previous state-of-the-art by achieving 71.13\% test accuracy on TVQA benchmark dataset. Extensive ablation studies and qualitative analysis are conducted to validate various components of the network.