ArtAug: Enhancing Text-to-Image Generation through Synthesis-Understanding Interaction

TL;DR

ArtAug introduces a novel interaction-based method that leverages understanding models to iteratively enhance text-to-image synthesis, producing more aesthetically pleasing images without extra computational costs.

Contribution

This paper presents the first approach to improve text-to-image models through model interactions with understanding models, enabling aesthetic enhancements without additional computational overhead.

Findings

ArtAug improves image aesthetics in text-to-image synthesis.

The method enhances models without extra computational costs.

Evaluation metrics confirm consistent improvements.

Abstract

The emergence of diffusion models has significantly advanced image synthesis. The recent studies of model interaction and self-corrective reasoning approach in large language models offer new insights for enhancing text-to-image models. Inspired by these studies, we propose a novel method called ArtAug for enhancing text-to-image models in this paper. To the best of our knowledge, ArtAug is the first one that improves image synthesis models via model interactions with understanding models. In the interactions, we leverage human preferences implicitly learned by image understanding models to provide fine-grained suggestions for image synthesis models. The interactions can modify the image content to make it aesthetically pleasing, such as adjusting exposure, changing shooting angles, and adding atmospheric effects. The enhancements brought by the interaction are iteratively fused into the synthesis model itself through an additional enhancement module. This enables the synthesis model to directly produce aesthetically pleasing images without any extra computational cost. In the experiments, we train the ArtAug enhancement module on existing text-to-image models. Various evaluation metrics consistently demonstrate that ArtAug enhances the generative capabilities of text-to-image models without incurring additional computational costs. The source code and models will be released publicly.