VideoAesBench: Benchmarking the Video Aesthetics Perception Capabilities of Large Multimodal Models

TL;DR

VideoAesBench is a comprehensive benchmark designed to evaluate large multimodal models' understanding of video aesthetic quality across diverse content, question formats, and aesthetic dimensions, revealing current models' limited capabilities.

Contribution

The paper introduces VideoAesBench, a novel, diverse, and multi-faceted benchmark for assessing LMMs' video aesthetic perception, filling a gap in evaluation tools.

Findings

Current LMMs show limited video aesthetic perception abilities.

Performance of models remains incomplete and imprecise.

VideoAesBench provides a new platform for explainable video aesthetics assessment.

Abstract

Large multimodal models (LMMs) have demonstrated outstanding capabilities in various visual perception tasks, which has in turn made the evaluation of LMMs significant. However, the capability of video aesthetic quality assessment, which is a fundamental ability for human, remains underexplored for LMMs. To address this, we introduce VideoAesBench, a comprehensive benchmark for evaluating LMMs' understanding of video aesthetic quality. VideoAesBench has several significant characteristics: (1) Diverse content including 1,804 videos from multiple video sources including user-generated (UGC), AI-generated (AIGC), compressed, robotic-generated (RGC), and game videos. (2) Multiple question formats containing traditional single-choice questions, multi-choice questions, True or False questions, and a novel open-ended questions for video aesthetics description. (3) Holistic video aesthetics dimensions including visual form related questions from 5 aspects, visual style related questions from 4 aspects, and visual affectiveness questions from 3 aspects. Based on VideoAesBench, we benchmark 23 open-source and commercial large multimodal models. Our findings show that current LMMs only contain basic video aesthetics perception ability, their performance remains incomplete and imprecise. We hope our VideoAesBench can be served as a strong testbed and offer insights for explainable video aesthetics assessment. The data will be released on https://github.com/michaelliyunhao/VideoAesBench