SAVEn-Vid: Synergistic Audio-Visual Integration for Enhanced Understanding in Long Video Context

TL;DR

This paper introduces SAVEn-Vid, a large audio-visual dataset and a time-aware AV-LLM model, SAVEnVideo, to improve understanding of long videos by effectively integrating audio-visual information, outperforming existing models.

Contribution

The paper presents the first long audio-visual dataset SAVEn-Vid, a novel AV-LLM SAVEnVideo, and a benchmark AVBench for evaluating audio-visual comprehension in long videos.

Findings

SAVEnVideo outperforms existing Video-LLMs by 3.61% on Video-MME.

SAVEnVideo surpasses leading audio-visual LLM by 1.29% on Music-AVQA.

SAVEn-Vid enables better integration of audio-visual info in long video understanding.

Abstract

Endeavors have been made to explore Large Language Models for video analysis (Video-LLMs), particularly in understanding and interpreting long videos. However, existing Video-LLMs still face challenges in effectively integrating the rich and diverse audio-visual information inherent in long videos, which is crucial for comprehensive understanding. This raises the question: how can we leverage embedded audio-visual information to enhance long video understanding? Therefore, (i) we introduce SAVEn-Vid, the first-ever long audio-visual video dataset comprising over 58k audio-visual instructions. (ii) From the model perspective, we propose a time-aware Audio-Visual Large Language Model (AV-LLM), SAVEnVideo, fine-tuned on SAVEn-Vid. (iii) Besides, we present AVBench, a benchmark containing 2,500 QAs designed to evaluate models on enhanced audio-visual comprehension tasks within long video, challenging their ability to handle intricate audio-visual interactions. Experiments on AVBench reveal the limitations of current AV-LLMs. Experiments also demonstrate that SAVEnVideo outperforms the best Video-LLM by 3.61% on the zero-shot long video task (Video-MME) and surpasses the leading audio-visual LLM by 1.29% on the zero-shot audio-visual task (Music-AVQA). Consequently, at the 7B parameter scale, SAVEnVideo can achieve state-of-the-art performance. Our dataset and code will be released at https://ljungang.github.io/SAVEn-Vid/ upon acceptance.