Counting Guidance for High Fidelity Text-to-Image Synthesis

TL;DR

This paper introduces a counting-guidance method that improves diffusion-based text-to-image models by accurately generating the specified number of objects in images, enhancing fidelity and object count accuracy.

Contribution

It proposes a novel counting network and attention map guidance to refine diffusion models for precise object count generation from text prompts.

Findings

Significant improvement in object count accuracy in generated images.

Enhanced image fidelity with respect to specified object numbers.

Effective integration of counting and attention mechanisms in diffusion models.

Abstract

Recently, there have been significant improvements in the quality and performance of text-to-image generation, largely due to the impressive results attained by diffusion models. However, text-to-image diffusion models sometimes struggle to create high-fidelity content for the given input prompt. One specific issue is their difficulty in generating the precise number of objects specified in the text prompt. For example, when provided with the prompt "five apples and ten lemons on a table," images generated by diffusion models often contain an incorrect number of objects. In this paper, we present a method to improve diffusion models so that they accurately produce the correct object count based on the input prompt. We adopt a counting network that performs reference-less class-agnostic counting for any given image. We calculate the gradients of the counting network and refine the predicted noise for each step. To address the presence of multiple types of objects in the prompt, we utilize novel attention map guidance to obtain high-quality masks for each object. Finally, we guide the denoising process using the calculated gradients for each object. Through extensive experiments and evaluation, we demonstrate that the proposed method significantly enhances the fidelity of diffusion models with respect to object count. Code is available at https://github.com/furiosa-ai/counting-guidance.