R-AVST: Empowering Video-LLMs with Fine-Grained Spatio-Temporal Reasoning in Complex Audio-Visual Scenarios

TL;DR

This paper introduces R-AVST, a comprehensive dataset with fine-grained spatio-temporal annotations for complex video audio-visual reasoning, and proposes AVST-Zero, a reinforcement learning model that advances reasoning capabilities in real-world scenarios.

Contribution

The paper presents the first dataset for real-world audio-visual spatio-temporal reasoning and a reinforcement learning model that directly optimizes reasoning behavior without intermediate supervision.

Findings

R-AVST contains over 5K videos with 27K objects across 100 event types.

AVST-Zero achieves competitive performance on reasoning tasks.

Extensive experiments validate the dataset's effectiveness and the model's reasoning capabilities.

Abstract

Recently, rapid advancements have been made in multimodal large language models (MLLMs), especially in video understanding tasks. However, current research focuses on simple video scenarios, failing to reflect the complex and diverse nature of real-world audio-visual events in videos. To bridge this gap, we firstly introduce R-AVST, a dataset for audio-visual reasoning featuring fine-grained spatio-temporal annotations. In constructing this, we design a pipeline consisting of LLM-based key object extraction, automatic spatial annotation and manual quality inspection, resulting in over 5K untrimmed videos with 27K objects across 100 types of audio-visual events. Building on this dataset, we define three core tasks for spatio-temporal reasoning in audio-visual scenes and generate more than 8K high-quality, evenly distributed question-answer pairs to effectively benchmark model performance. To further enhance reasoning, we propose AVST-Zero, a reinforcement learning-based model that avoids intermediate supervision, directly optimizing behavior via carefully designed multi-dimensional rewards. Extensive experiments validate the effectiveness of our R-AVST in advancing audio-visual spatio-temporal reasoning, upon which AVST-Zero demonstrates competitive performance compared to existing models. To the best of our knowledge, R-AVST is the first dataset designed for real-world audio-visual spatio-temporal reasoning, and AVST-Zero offers a novel perspective for tackling future challenges in this domain.