Multi-modal Crowd Counting via a Broker Modality

TL;DR

This paper introduces a novel triple-modal learning framework for multi-modal crowd counting, utilizing an auxiliary broker modality to improve fusion of visual and thermal/depth images, addressing cross-modal fusion challenges.

Contribution

The paper proposes a new broker modality and a fusion method using lightweight diffusion models, significantly enhancing multi-modal crowd counting performance.

Findings

Achieves promising results on popular datasets

Adds only 4 million parameters to existing models

Effectively addresses ghosting effects in cross-modal fusion

Abstract

Multi-modal crowd counting involves estimating crowd density from both visual and thermal/depth images. This task is challenging due to the significant gap between these distinct modalities. In this paper, we propose a novel approach by introducing an auxiliary broker modality and on this basis frame the task as a triple-modal learning problem. We devise a fusion-based method to generate this broker modality, leveraging a non-diffusion, lightweight counterpart of modern denoising diffusion-based fusion models. Additionally, we identify and address the ghosting effect caused by direct cross-modal image fusion in multi-modal crowd counting. Through extensive experimental evaluations on popular multi-modal crowd-counting datasets, we demonstrate the effectiveness of our method, which introduces only 4 million additional parameters, yet achieves promising results. The code is available at https://github.com/HenryCilence/Broker-Modality-Crowd-Counting.