GAIA: Rethinking Action Quality Assessment for AI-Generated Videos

TL;DR

This paper introduces GAIA, a large-scale dataset for assessing action quality in AI-generated videos, revealing current evaluation methods' limitations and emphasizing the need for improved AQA techniques.

Contribution

The paper constructs GAIA, a comprehensive dataset for action quality assessment in AI-generated videos, and benchmarks existing evaluation methods, highlighting their shortcomings.

Findings

Traditional AQA methods perform poorly on AIGV evaluation.

Current metrics show a significant gap compared to human perception.

GAIA enables better understanding and development of AQA models for AIGVs.

Abstract

Assessing action quality is both imperative and challenging due to its significant impact on the quality of AI-generated videos, further complicated by the inherently ambiguous nature of actions within AI-generated video (AIGV). Current action quality assessment (AQA) algorithms predominantly focus on actions from real specific scenarios and are pre-trained with normative action features, thus rendering them inapplicable in AIGVs. To address these problems, we construct GAIA, a Generic AI-generated Action dataset, by conducting a large-scale subjective evaluation from a novel causal reasoning-based perspective, resulting in 971,244 ratings among 9,180 video-action pairs. Based on GAIA, we evaluate a suite of popular text-to-video (T2V) models on their ability to generate visually rational actions, revealing their pros and cons on different categories of actions. We also extend GAIA as a testbed to benchmark the AQA capacity of existing automatic evaluation methods. Results show that traditional AQA methods, action-related metrics in recent T2V benchmarks, and mainstream video quality methods perform poorly with an average SRCC of 0.454, 0.191, and 0.519, respectively, indicating a sizable gap between current models and human action perception patterns in AIGVs. Our findings underscore the significance of action quality as a unique perspective for studying AIGVs and can catalyze progress towards methods with enhanced capacities for AQA in AIGVs.