JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation

TL;DR

JointAVBench is a comprehensive benchmark designed to evaluate multi-modal reasoning in Omni-Large Language Models across audio-visual dependencies, diverse audio types, and scene spans, revealing current models' limitations.

Contribution

The paper introduces JointAVBench, a novel automated benchmark for joint audio-visual reasoning, covering multiple dimensions and scene spans, with a pipeline for question synthesis.

Findings

Best Omni-LLMs achieve only 65.3% accuracy, indicating significant room for improvement.

Omni-LLMs outperform uni-modal baselines but struggle with cross-scene reasoning.

The benchmark reveals current models' limitations in multi-modal understanding.

Abstract

Understanding videos inherently requires reasoning over both visual and auditory information. To properly evaluate Omni-Large Language Models (Omni-LLMs), which are capable of processing multi-modal information including vision and audio, an effective benchmark must comprehensively cover three key aspects: (1) multi-modal dependency (i.e., questions that cannot be answered using vision or audio alone), (2) diverse audio information types (e.g., speech, sound events), and (3) varying scene spans. However, existing datasets fall short in one or more of these dimensions, limiting strict and comprehensive evaluation. To address this gap, we introduce JointAVBench, a novel benchmark with strict audio-video correlation, spanning five cognitive dimensions, four audio information types (speech, sound events, music, vocal traits), and three scene spans (single-, cross-, and full-scene). Given the high cost of manual annotation, we propose an automated pipeline that leverages state-of-the-art vision-LLMs, audio-LLMs, and general-purpose LLMs to synthesize questions and answers that strictly require joint audio-visual understanding. We evaluate leading vision-only, audio-only, and Omni-LLMs on our dataset. Results show that even the best-performing Omni-LLM achieves an average accuracy of only 65.3\%, outperforming uni-modal baselines but revealing substantial room for improvement, especially in cross-scene reasoning.