Detection-Driven Object Count Optimization for Text-to-Image Diffusion Models

TL;DR

This paper introduces a novel framework for controlling object counts in text-to-image diffusion models by leveraging pre-trained counting and detection models, improving robustness, efficiency, and accuracy without extensive retraining.

Contribution

The authors propose a detection-driven optimization method that enhances object count control in diffusion models, addressing limitations of prior supervised and iterative approaches.

Findings

Improved object counting accuracy across diverse categories.

Enhanced robustness to viewpoint and proportion variations.

Reduced computational cost through token reuse.

Abstract

Accurately controlling object count in text-to-image generation remains a key challenge. Supervised methods often fail, as training data rarely covers all count variations. Methods that manipulate the denoising process to add or remove objects can help; however, they still require labeled data, limit robustness and image quality, and rely on a slow, iterative process. Pre-trained differentiable counting models that rely on soft object density summation exist and could steer generation, but employing them presents three main challenges: (i) they are pre-trained on clean images, making them less effective during denoising steps that operate on noisy inputs; (ii) they are not robust to viewpoint changes; and (iii) optimization is computationally expensive, requiring repeated model evaluations per image. We propose a new framework that uses pre-trained object counting techniques and object detectors to guide generation. First, we optimize a counting token using an outer-loop loss computed on fully generated images. Second, we introduce a detection-driven scaling term that corrects errors caused by viewpoint and proportion shifts, among other factors, without requiring backpropagation through the detection model. Third, we show that the optimized parameters can be reused for new prompts, removing the need for repeated optimization. Our method provides efficiency through token reuse, flexibility via compatibility with various detectors, and accuracy with improved counting across diverse object categories.