AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation

TL;DR

AVGen-Bench introduces a comprehensive, task-driven benchmark for evaluating multi-granular aspects of text-to-audio-video generation, addressing the limitations of existing coarse evaluation methods.

Contribution

It presents a new benchmark with high-quality prompts and a multi-granular evaluation framework combining specialist models and MLLMs for detailed assessment.

Findings

Significant gap between aesthetic quality and semantic reliability.

Persistent failures in text rendering, speech coherence, and physical reasoning.

Universal breakdown in musical pitch control.

Abstract

Text-to-Audio-Video (T2AV) generation is rapidly becoming a core interface for media creation, yet its evaluation remains fragmented. Existing benchmarks largely assess audio and video in isolation or rely on coarse embedding similarity, failing to capture the fine-grained joint correctness required by realistic prompts. We introduce AVGen-Bench, a task-driven benchmark for T2AV generation featuring high-quality prompts across 11 real-world categories. To support comprehensive assessment, we propose a multi-granular evaluation framework that combines lightweight specialist models with Multimodal Large Language Models (MLLMs), enabling evaluation from perceptual quality to fine-grained semantic controllability. Our evaluation reveals a pronounced gap between strong audio-visual aesthetics and weak semantic reliability, including persistent failures in text rendering, speech coherence, physical reasoning, and a universal breakdown in musical pitch control. Code and benchmark resources are available at http://aka.ms/avgenbench.