Temporal Reasoning via Audio Question Answering

TL;DR

This paper introduces the DAQA dataset for audio question answering to evaluate and improve models' temporal reasoning abilities, proposing MALiMo as a novel model that outperforms existing methods.

Contribution

The paper presents the DAQA dataset for probing temporal reasoning in audio QA and introduces MALiMo, a new model that enhances temporal reasoning performance.

Findings

State-of-the-art models perform poorly on temporal reasoning questions.

MALiMo significantly improves temporal reasoning capabilities.

DAQA fosters research in audio question answering and temporal reasoning.

Abstract

Multimodal question answering tasks can be used as proxy tasks to study systems that can perceive and reason about the world. Answering questions about different types of input modalities stresses different aspects of reasoning such as visual reasoning, reading comprehension, story understanding, or navigation. In this paper, we use the task of Audio Question Answering (AQA) to study the temporal reasoning abilities of machine learning models. To this end, we introduce the Diagnostic Audio Question Answering (DAQA) dataset comprising audio sequences of natural sound events and programmatically generated questions and answers that probe various aspects of temporal reasoning. We adapt several recent state-of-the-art methods for visual question answering to the AQA task, and use DAQA to demonstrate that they perform poorly on questions that require in-depth temporal reasoning. Finally, we propose a new model, Multiple Auxiliary Controllers for Linear Modulation (MALiMo) that extends the recent Feature-wise Linear Modulation (FiLM) model and significantly improves its temporal reasoning capabilities. We envisage DAQA to foster research on AQA and temporal reasoning and MALiMo a step towards models for AQA.