Bootstrapping MLLM for Weakly-Supervised Class-Agnostic Object Counting

TL;DR

This paper introduces WS-COC, a novel weakly-supervised, multi-label learning framework for class-agnostic object counting that leverages dialogue tuning, compare-and-rank optimization, and multi-scale counting to outperform fully-supervised methods.

Contribution

The paper presents the first MLLM-driven weakly-supervised framework for class-agnostic object counting, introducing three strategies to improve counting without detailed annotations.

Findings

WS-COC matches or surpasses state-of-the-art fully-supervised methods.

Significantly reduces annotation costs in object counting tasks.

Effective in dense scenes with improved local and global count aggregation.

Abstract

Object counting is a fundamental task in computer vision, with broad applicability in many real-world scenarios. Fully-supervised counting methods require costly point-level annotations per object. Few weakly-supervised methods leverage only image-level object counts as supervision and achieve fairly promising results. They are, however, often limited to counting a single category, e.g. person. In this paper, we propose WS-COC, the first MLLM-driven weakly-supervised framework for class-agnostic object counting. Instead of directly fine-tuning MLLMs to predict object counts, which can be challenging due to the modality gap, we incorporate three simple yet effective strategies to bootstrap the counting paradigm in both training and testing: First, a divide-and-discern dialogue tuning strategy is proposed to guide the MLLM to determine whether the object count falls within a specific range and progressively break down the range through multi-round dialogue. Second, a compare-and-rank count optimization strategy is introduced to train the MLLM to optimize the relative ranking of multiple images according to their object counts. Third, a global-and-local counting enhancement strategy aggregates and fuses local and global count predictions to improve counting performance in dense scenes. Extensive experiments on FSC-147, CARPK, PUCPR+, and ShanghaiTech show that WS-COC matches or even surpasses many state-of-art fully-supervised methods while significantly reducing annotation costs. Code is available at https://github.com/viscom-tongji/WS-COC.