VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categories

TL;DR

VidAudio-Bench is a comprehensive benchmark for evaluating video-to-audio and video-text-to-audio generation across four audio categories, introducing new metrics and revealing current model limitations.

Contribution

It introduces a multi-task benchmark with extensive evaluation metrics and validation, addressing the lack of fine-grained assessment in V2A and VT2A systems.

Findings

Current models perform poorly in speech and singing generation.

Visual conditioning improves video-audio alignment but may reduce audio category accuracy.

The benchmark provides insights into the trade-offs in multimodal audio generation.

Abstract

Video-to-Audio (V2A) generation is essential for immersive multimedia experiences, yet its evaluation remains underexplored. Existing benchmarks typically assess diverse audio types under a unified protocol, overlooking the fine-grained requirements of distinct audio categories. To address this gap, we propose VidAudio-Bench, a multi-task benchmark for V2A evaluation with four key features: (1) Broad Coverage: It encompasses four representative audio categories - sound effects, music, speech, and singing - under both V2A and Video-Text-to-Audio (VT2A) settings. (2) Extensive Evaluation: It comprises 1,634 video-text pairs and benchmarks 11 state-of-the-art generation models. (3) Comprehensive Metrics: It introduces 13 task-specific, reference-free metrics to systematically assess audio quality, video-audio consistency, and text-audio consistency. (4) Human Alignment: It validates all metrics through subjective studies, demonstrating strong consistency with human preferences. Experimental results reveal that current V2A models perform poorly in speech and singing compared to sound effects. Our VT2A results further highlight a fundamental tension between instruction following and visually grounded generation: stronger visual conditioning improves video-audio alignment, but often at the cost of generating the intended audio category. These findings establish VidAudio-Bench as a comprehensive and scalable framework for diagnosing V2A systems and provide new insights into multimodal audio generation.