Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model

TL;DR

This paper introduces a multimodal machine comprehension model that fuses audio and textual data, enabling it to perform well on various MC tasks and outperform unimodal models through novel attention and knowledge distillation techniques.

Contribution

It proposes the DIIA model for effective audio-text fusion and the MKD module for unimodal prediction, advancing multimodal MC capabilities.

Findings

DIIA improves accuracy by up to 21.08%.

MKD enables the model to outperform unimodal models by up to 18.87%.

The model handles multiple MC tasks with a single architecture.

Abstract

While Machine Comprehension (MC) has attracted extensive research interests in recent years, existing approaches mainly belong to the category of Machine Reading Comprehension task which mines textual inputs (paragraphs and questions) to predict the answers (choices or text spans). However, there are a lot of MC tasks that accept audio input in addition to the textual input, e.g. English listening comprehension test. In this paper, we target the problem of Audio-Oriented Multimodal Machine Comprehension, and its goal is to answer questions based on the given audio and textual information. To solve this problem, we propose a Dynamic Inter- and Intra-modality Attention (DIIA) model to effectively fuse the two modalities (audio and textual). DIIA can work as an independent component and thus be easily integrated into existing MC models. Moreover, we further develop a Multimodal Knowledge Distillation (MKD) module to enable our multimodal MC model to accurately predict the answers based only on either the text or the audio. As a result, the proposed approach can handle various tasks including: Audio-Oriented Multimodal Machine Comprehension, Machine Reading Comprehension and Machine Listening Comprehension, in a single model, making fair comparisons possible between our model and the existing unimodal MC models. Experimental results and analysis prove the effectiveness of the proposed approaches. First, the proposed DIIA boosts the baseline models by up to 21.08% in terms of accuracy; Second, under the unimodal scenarios, the MKD module allows our multimodal MC model to significantly outperform the unimodal models by up to 18.87%, which are trained and tested with only audio or textual data.