Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios

TL;DR

This paper introduces a novel target-aware joint spatio-temporal reasoning network for audio-visual question answering, effectively integrating spatial and temporal grounding with a focus on query-relevant cues and cross-modal synchronization.

Contribution

It proposes a unified target-aware spatial grounding module and a single-stream joint temporal grounding module with cross-modal synchrony loss, advancing AVQA performance.

Findings

Outperforms existing state-of-the-art methods

Effective spatial and temporal grounding in AVQA

Improved cross-modal synchronization

Abstract

Audio-visual question answering (AVQA) is a challenging task that requires multistep spatio-temporal reasoning over multimodal contexts. Recent works rely on elaborate target-agnostic parsing of audio-visual scenes for spatial grounding while mistreating audio and video as separate entities for temporal grounding. This paper proposes a new target-aware joint spatio-temporal grounding network for AVQA. It consists of two key components: the target-aware spatial grounding module (TSG) and the single-stream joint audio-visual temporal grounding module (JTG). The TSG can focus on audio-visual cues relevant to the query subject by utilizing explicit semantics from the question. Unlike previous two-stream temporal grounding modules that required an additional audio-visual fusion module, JTG incorporates audio-visual fusion and question-aware temporal grounding into one module with a simpler single-stream architecture. The temporal synchronization between audio and video in the JTG is facilitated by our proposed cross-modal synchrony loss (CSL). Extensive experiments verified the effectiveness of our proposed method over existing state-of-the-art methods.