Enhancing Image Aesthetics with Dual-Conditioned Diffusion Models Guided by Multimodal Perception

TL;DR

This paper introduces DIAE, a diffusion-based model that enhances image aesthetics by integrating multimodal perception and weak supervision, effectively addressing challenges in aesthetic editing and data scarcity.

Contribution

It proposes a novel dual-supervised diffusion model with multimodal aesthetic perception and a new dataset to improve aesthetic enhancement of images.

Findings

DIAE outperforms baseline models in aesthetic score metrics.

The model maintains high content consistency in edited images.

Experimental results validate the effectiveness of multimodal guidance.

Abstract

Image aesthetic enhancement aims to perceive aesthetic deficiencies in images and perform corresponding editing operations, which is highly challenging and requires the model to possess creativity and aesthetic perception capabilities. Although recent advancements in image editing models have significantly enhanced their controllability and flexibility, they struggle with enhancing image aesthetic. The primary challenges are twofold: first, following editing instructions with aesthetic perception is difficult, and second, there is a scarcity of "perfectly-paired" images that have consistent content but distinct aesthetic qualities. In this paper, we propose Dual-supervised Image Aesthetic Enhancement (DIAE), a diffusion-based generative model with multimodal aesthetic perception. First, DIAE incorporates Multimodal Aesthetic Perception (MAP) to convert the ambiguous aesthetic instruction into explicit guidance by (i) employing detailed, standardized aesthetic instructions across multiple aesthetic attributes, and (ii) utilizing multimodal control signals derived from text-image pairs that maintain consistency within the same aesthetic attribute. Second, to mitigate the lack of "perfectly-paired" images, we collect "imperfectly-paired" dataset called IIAEData, consisting of images with varying aesthetic qualities while sharing identical semantics. To better leverage the weak matching characteristics of IIAEData during training, a dual-branch supervision framework is also introduced for weakly supervised image aesthetic enhancement. Experimental results demonstrate that DIAE outperforms the baselines and obtains superior image aesthetic scores and image content consistency scores.