VABench: A Comprehensive Benchmark for Audio-Video Generation

TL;DR

VABench is a new comprehensive benchmark framework designed to evaluate the quality and synchronization of audio-video generation across multiple task types and content categories.

Contribution

It introduces a multi-dimensional evaluation framework covering 15 metrics and 7 content categories for assessing synchronized audio-video generation models.

Findings

Systematic analysis and visualization of evaluation results.

Establishes a new standard for assessing audio-video generation models.

Addresses the gap in existing benchmarks for synchronized audio-video outputs.

Abstract

Recent advances in video generation have been remarkable, enabling models to produce visually compelling videos with synchronized audio. While existing video generation benchmarks provide comprehensive metrics for visual quality, they lack convincing evaluations for audio-video generation, especially for models aiming to generate synchronized audio-video outputs. To address this gap, we introduce VABench, a comprehensive and multi-dimensional benchmark framework designed to systematically evaluate the capabilities of synchronous audio-video generation. VABench encompasses three primary task types: text-to-audio-video (T2AV), image-to-audio-video (I2AV), and stereo audio-video generation. It further establishes two major evaluation modules covering 15 dimensions. These dimensions specifically assess pairwise similarities (text-video, text-audio, video-audio), audio-video synchronization, lip-speech consistency, and carefully curated audio and video question-answering (QA) pairs, among others. Furthermore, VABench covers seven major content categories: animals, human sounds, music, environmental sounds, synchronous physical sounds, complex scenes, and virtual worlds. We provide a systematic analysis and visualization of the evaluation results, aiming to establish a new standard for assessing video generation models with synchronous audio capabilities and to promote the comprehensive advancement of the field.