CountGD++: Generalized Prompting for Open-World Counting

TL;DR

CountGD++ introduces a flexible, multi-modal counting framework that allows specifying what not to count, automates annotation with pseudo-exemplars, and enhances open-world counting accuracy and generalization.

Contribution

The paper presents CountGD++, a novel counting model that extends prompt capabilities with negative descriptions, pseudo-exemplar automation, and external image inputs, improving open-world counting.

Findings

Significant accuracy improvements across multiple datasets

Enhanced generalization to diverse counting scenarios

Automated annotation reduces manual effort

Abstract

The flexibility and accuracy of methods for automatically counting objects in images and videos are limited by the way the object can be specified. While existing methods allow users to describe the target object with text and visual examples, the visual examples must be manually annotated inside the image, and there is no way to specify what not to count. To address these gaps, we introduce novel capabilities that expand how the target object can be specified. Specifically, we extend the prompt to enable what not to count to be described with text and/or visual examples, introduce the concept of `pseudo-exemplars' that automate the annotation of visual examples at inference, and extend counting models to accept visual examples from both natural and synthetic external images. We also use our new counting model, CountGD++, as a vision expert agent for an LLM. Together, these contributions expand the prompt flexibility of multi-modal open-world counting and lead to significant improvements in accuracy, efficiency, and generalization across multiple datasets. Code is available at https://github.com/niki-amini-naieni/CountGDPlusPlus.