AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception

TL;DR

AesBench is a comprehensive benchmark designed to evaluate multimodal large language models' capabilities in image aesthetics perception, revealing current models' limited abilities and guiding future improvements.

Contribution

This work introduces AesBench, including a high-quality expert-labeled database and a set of criteria for assessing aesthetic perception in MLLMs, filling a critical evaluation gap.

Findings

Current MLLMs show only basic aesthetic perception abilities.

Significant gap exists between MLLMs and human aesthetic judgment.

The benchmark facilitates future research in MLLM aesthetic perception.

Abstract

With collective endeavors, multimodal large language models (MLLMs) are undergoing a flourishing development. However, their performances on image aesthetics perception remain indeterminate, which is highly desired in real-world applications. An obvious obstacle lies in the absence of a specific benchmark to evaluate the effectiveness of MLLMs on aesthetic perception. This blind groping may impede the further development of more advanced MLLMs with aesthetic perception capacity. To address this dilemma, we propose AesBench, an expert benchmark aiming to comprehensively evaluate the aesthetic perception capacities of MLLMs through elaborate design across dual facets. (1) We construct an Expert-labeled Aesthetics Perception Database (EAPD), which features diversified image contents and high-quality annotations provided by professional aesthetic experts. (2) We propose a set of integrative criteria to measure the aesthetic perception abilities of MLLMs from four perspectives, including Perception (AesP), Empathy (AesE), Assessment (AesA) and Interpretation (AesI). Extensive experimental results underscore that the current MLLMs only possess rudimentary aesthetic perception ability, and there is still a significant gap between MLLMs and humans. We hope this work can inspire the community to engage in deeper explorations on the aesthetic potentials of MLLMs. Source data will be available at https://github.com/yipoh/AesBench.