Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering

TL;DR

This paper introduces a prompt learning framework that automatically optimizes textual descriptions to enhance the accuracy of text-to-image diffusion synthesis, especially for complex texts, reducing manual effort.

Contribution

It proposes a novel prompt learning approach utilizing quality and semantic guidance from pre-trained diffusion models to improve image-text alignment.

Findings

Enhanced image quality and accuracy with learned prompts

Reduced manual prompt engineering effort

Validated effectiveness through extensive experiments

Abstract

The text-to-image synthesis by diffusion models has recently shown remarkable performance in generating high-quality images. Although performs well for simple texts, the models may get confused when faced with complex texts that contain multiple objects or spatial relationships. To get the desired images, a feasible way is to manually adjust the textual descriptions, i.e., narrating the texts or adding some words, which is labor-consuming. In this paper, we propose a framework to learn the proper textual descriptions for diffusion models through prompt learning. By utilizing the quality guidance and the semantic guidance derived from the pre-trained diffusion model, our method can effectively learn the prompts to improve the matches between the input text and the generated images. Extensive experiments and analyses have validated the effectiveness of the proposed method.