AesTest: Measuring Aesthetic Intelligence from Perception to Production

TL;DR

AesTest is a comprehensive benchmark designed to evaluate multimodal large language models' abilities in aesthetic perception and production across diverse tasks, sources, and aesthetic query types, addressing current limitations in scope and diversity.

Contribution

The paper introduces AesTest, a novel benchmark with diverse tasks and data sources to systematically evaluate aesthetic intelligence in multimodal models.

Findings

Instruction-tuned IAA MLLMs show significant challenges on AesTest.

The benchmark covers perception, appreciation, creation, and photography tasks.

AesTest will be publicly released for future research.

Abstract

Perceiving and producing aesthetic judgments is a fundamental yet underexplored capability for multimodal large language models (MLLMs). However, existing benchmarks for image aesthetic assessment (IAA) are narrow in perception scope or lack the diversity needed to evaluate systematic aesthetic production. To address this gap, we introduce AesTest, a comprehensive benchmark for multimodal aesthetic perception and production, distinguished by the following features: 1) It consists of curated multiple-choice questions spanning ten tasks, covering perception, appreciation, creation, and photography. These tasks are grounded in psychological theories of generative learning. 2) It integrates data from diverse sources, including professional editing workflows, photographic composition tutorials, and crowdsourced preferences. It ensures coverage of both expert-level principles and real-world variation. 3) It supports various aesthetic query types, such as attribute-based analysis, emotional resonance, compositional choice, and stylistic reasoning. We evaluate both instruction-tuned IAA MLLMs and general MLLMs on AesTest, revealing significant challenges in building aesthetic intelligence. We will publicly release AesTest to support future research in this area.