AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning

TL;DR

AURA is a new benchmark and metric for evaluating the reasoning process of audio-visual models, emphasizing logical coherence and factual grounding beyond mere answer accuracy.

Contribution

We introduce AURA, a comprehensive AV reasoning benchmark with a novel metric AuraScore to evaluate reasoning fidelity and identify reasoning gaps in current models.

Findings

High accuracy models often lack reasoning fidelity

Models show significant gaps in factual consistency and logical inference

AURA reveals the need for improved reasoning capabilities in AV models

Abstract

Current audio-visual (AV) benchmarks focus on final answer accuracy, overlooking the underlying reasoning process. This makes it difficult to distinguish genuine comprehension from correct answers derived through flawed reasoning or hallucinations. To address this, we introduce AURA (Audio-visual Understanding and Reasoning Assessment), a benchmark for evaluating the cross-modal reasoning capabilities of Audio-Visual Large Language Models (AV-LLMs) and Omni-modal Language Models (OLMs). AURA includes questions across six challenging cognitive domains, such as causality, timbre and pitch, tempo and AV synchronization, unanswerability, implicit distractions, and skill profiling, explicitly designed to be unanswerable from a single modality. This forces models to construct a valid logical path grounded in both audio and video, setting AURA apart from AV datasets that allow uni-modal shortcuts. To assess reasoning traces, we propose a novel metric, AuraScore, which addresses the lack of robust tools for evaluating reasoning fidelity. It decomposes reasoning into two aspects: (i) Factual Consistency - whether reasoning is grounded in perceptual evidence, and (ii) Core Inference - the logical validity of each reasoning step. Evaluations of SOTA models on AURA reveal a critical reasoning gap: although models achieve high accuracy (up to 92% on some tasks), their Factual Consistency and Core Inference scores fall below 45%. This discrepancy highlights that models often arrive at correct answers through flawed logic, underscoring the need for our benchmark and paving the way for more robust multimodal evaluation.