AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM

TL;DR

This paper introduces AIGV-Assessor, a new model for evaluating the perceptual quality of AI-generated videos, supported by a large dataset and demonstrating superior performance over existing methods.

Contribution

The paper presents AIGVQA-DB, a large-scale dataset of annotated AI-generated videos, and introduces AIGV-Assessor, a novel VQA model leveraging spatiotemporal features and LMMs for accurate quality assessment.

Findings

AIGV-Assessor outperforms existing evaluation methods in multiple perceptual quality metrics.

The dataset AIGVQA-DB contains 36,576 videos with 370k expert ratings.

AIGV-Assessor accurately predicts video quality scores and preferences.

Abstract

The rapid advancement of large multimodal models (LMMs) has led to the rapid expansion of artificial intelligence generated videos (AIGVs), which highlights the pressing need for effective video quality assessment (VQA) models designed specifically for AIGVs. Current VQA models generally fall short in accurately assessing the perceptual quality of AIGVs due to the presence of unique distortions, such as unrealistic objects, unnatural movements, or inconsistent visual elements. To address this challenge, we first present AIGVQA-DB, a large-scale dataset comprising 36,576 AIGVs generated by 15 advanced text-to-video models using 1,048 diverse prompts. With these AIGVs, a systematic annotation pipeline including scoring and ranking processes is devised, which collects 370k expert ratings to date. Based on AIGVQA-DB, we further introduce AIGV-Assessor, a novel VQA model that leverages spatiotemporal features and LMM frameworks to capture the intricate quality attributes of AIGVs, thereby accurately predicting precise video quality scores and video pair preferences. Through comprehensive experiments on both AIGVQA-DB and existing AIGV databases, AIGV-Assessor demonstrates state-of-the-art performance, significantly surpassing existing scoring or evaluation methods in terms of multiple perceptual quality dimensions.