CountSteer: Steering Attention for Object Counting in Diffusion Models

TL;DR

CountSteer is a training-free technique that guides diffusion models to generate images with accurate object counts by manipulating internal attention signals, enhancing numerical control without affecting image quality.

Contribution

We introduce CountSteer, a novel method that leverages internal attention signals to improve object counting accuracy in diffusion models without additional training.

Findings

Object count accuracy improved by about 4%.

No compromise on visual quality.

Model's internal signals are correlated with counting correctness.

Abstract

Text-to-image diffusion models generate realistic and coherent images but often fail to follow numerical instructions in text, revealing a gap between language and visual representation. Interestingly, we found that these models are not entirely blind to numbers-they are implicitly aware of their own counting accuracy, as their internal signals shift in consistent ways depending on whether the output meets the specified count. This observation suggests that the model already encodes a latent notion of numerical correctness, which can be harnessed to guide generation more precisely. Building on this intuition, we introduce CountSteer, a training-free method that improves generation of specified object counts by steering the model's cross-attention hidden states during inference. In our experiments, CountSteer improved object-count accuracy by about 4% without compromising visual quality, demonstrating a simple yet effective step toward more controllable and semantically reliable text-to-image generation.