Generating Intermediate Representations for Compositional Text-To-Image Generation

TL;DR

This paper introduces a two-stage, compositional diffusion approach for text-to-image generation that produces intermediate representations to better capture spatial details, leading to improved image quality.

Contribution

It presents a novel two-stage diffusion-based method that generates intermediate representations to enhance spatial accuracy in text-to-image synthesis.

Findings

Improved FID score over baseline

Comparable CLIP score to baseline

Enhanced spatial detail in generated images

Abstract

Text-to-image diffusion models have demonstrated an impressive ability to produce high-quality outputs. However, they often struggle to accurately follow fine-grained spatial information in an input text. To this end, we propose a compositional approach for text-to-image generation based on two stages. In the first stage, we design a diffusion-based generative model to produce one or more aligned intermediate representations (such as depth or segmentation maps) conditioned on text. In the second stage, we map these representations, together with the text, to the final output image using a separate diffusion-based generative model. Our findings indicate that such compositional approach can improve image generation, resulting in a notable improvement in FID score and a comparable CLIP score, when compared to the standard non-compositional baseline.