Audio-Visual LLM for Video Understanding

TL;DR

This paper introduces Audio-Visual LLM, a multimodal large language model that integrates visual and auditory data for comprehensive video understanding, demonstrating strong zero-shot performance on various tasks.

Contribution

The paper presents a novel modality-augmented training approach and a high-quality video instruction dataset, enabling end-to-end joint training for multimodal video understanding.

Findings

Achieves 53.7% accuracy on MSRVTT-QA, surpassing previous models.

Demonstrates strong zero-shot performance across multiple video understanding tasks.

Performs competitively on audio-specific tasks like AudioCaps.

Abstract

This paper presents Audio-Visual LLM, a Multimodal Large Language Model that takes both visual and auditory inputs for holistic video understanding. A key design is the modality-augmented training, which involves the integration of modality-specific tokens engineered to activate the appropriate visual and/or auditory encoder selectively. This mechanism is pivotal in enabling end-to-end joint training with video data at different modalities, including visual-only, audio-only, and audio-visual formats. Moreover, we introduce a high-quality video instruction dataset, derived from GPT-4. This dataset allows Audio-Visual LLM to adeptly process a variety of task-oriented video instructions, ranging from multi-turn conversations and audio-visual narratives to complex reasoning tasks. Extensive experiments demonstrate that Audio-Visual LLM impressively achieves strong zero-shot results across a range of video understanding tasks. For example, Audio-Visual LLM achieves an accuracy of 53.7% on MSRVTT-QA, outperforming non-LLM-based InterVideo by 6.6% and LLM-based Valley by 4.4%, respectively. Additionally, our Audio-Visual LLM also achieves competitive performance on audio tasks (e.g., AudioCaps).