Enhancing Zero-shot Counting via Language-guided Exemplar Learning

TL;DR

This paper introduces ExpressCount, a novel zero-shot counting method leveraging language-guided exemplar learning with large language models and a dual-branch visual pipeline, achieving state-of-the-art results.

Contribution

The paper presents a new framework combining language-oriented exemplar perception and a visual counting pipeline, advancing zero-shot counting capabilities.

Findings

Achieves state-of-the-art zero-shot counting performance.

Demonstrates accuracy comparable to partial category-specific models.

Introduces FSC-147-Express dataset with linguistic annotations.

Abstract

Recently, Class-Agnostic Counting (CAC) problem has garnered increasing attention owing to its intriguing generality and superior efficiency compared to Category-Specific Counting (CSC). This paper proposes a novel ExpressCount to enhance zero-shot object counting by delving deeply into language-guided exemplar learning. Specifically, the ExpressCount is comprised of an innovative Language-oriented Exemplar Perceptron and a downstream visual Zero-shot Counting pipeline. Thereinto, the perceptron hammers at exploiting accurate exemplar cues from collaborative language-vision signals by inheriting rich semantic priors from the prevailing pre-trained Large Language Models (LLMs), whereas the counting pipeline excels in mining fine-grained features through dual-branch and cross-attention schemes, contributing to the high-quality similarity learning. Apart from building a bridge between the LLM in vogue and the visual counting tasks, expression-guided exemplar estimation significantly advances zero-shot learning capabilities for counting instances with arbitrary classes. Moreover, devising a FSC-147-Express with annotations of meticulous linguistic expressions pioneers a new venue for developing and validating language-based counting models. Extensive experiments demonstrate the state-of-the-art performance of our ExpressCount, even showcasing the accuracy on par with partial CSC models.