T2ICount: Enhancing Cross-modal Understanding for Zero-Shot Counting

TL;DR

T2ICount introduces a diffusion-based framework with hierarchical semantic correction and regional coherence loss to improve zero-shot object counting by enhancing text-image alignment and sensitivity.

Contribution

The paper proposes T2ICount, a novel diffusion-based approach with modules that refine cross-modal understanding and introduces a challenging dataset subset for better evaluation.

Findings

Achieves superior performance on multiple benchmarks.

Effectively refines text-image alignment through hierarchical correction.

Demonstrates improved sensitivity to text prompts in counting tasks.

Abstract

Zero-shot object counting aims to count instances of arbitrary object categories specified by text descriptions. Existing methods typically rely on vision-language models like CLIP, but often exhibit limited sensitivity to text prompts. We present T2ICount, a diffusion-based framework that leverages rich prior knowledge and fine-grained visual understanding from pretrained diffusion models. While one-step denoising ensures efficiency, it leads to weakened text sensitivity. To address this challenge, we propose a Hierarchical Semantic Correction Module that progressively refines text-image feature alignment, and a Representational Regional Coherence Loss that provides reliable supervision signals by leveraging the cross-attention maps extracted from the denosing U-Net. Furthermore, we observe that current benchmarks mainly focus on majority objects in images, potentially masking models' text sensitivity. To address this, we contribute a challenging re-annotated subset of FSC147 for better evaluation of text-guided counting ability. Extensive experiments demonstrate that our method achieves superior performance across different benchmarks. Code is available at https://github.com/cha15yq/T2ICount.