Dense Point-to-Mask Optimization with Reinforced Point Selection for Crowd Instance Segmentation

TL;DR

This paper introduces a novel dense point-to-mask optimization method combined with reinforced point selection to improve crowd instance segmentation and counting accuracy, leveraging limited point annotations.

Contribution

It proposes Dense Point-to-Mask Optimization with NNEC constraints and a Reinforced Point Selection framework trained with GRPO, achieving state-of-the-art results.

Findings

Achieved top performance on multiple crowd datasets.

Enhanced counting accuracy using mask-supervised loss functions.

Demonstrated the effectiveness of the proposed methods in dense crowd scenarios.

Abstract

Crowd instance segmentation is a crucial task with a wide range of applications, including surveillance and transportation. Currently, point labels are common in crowd datasets, while region labels (e.g., boxes) are rare and inaccurate. The masks obtained through segmentation help to improve the accuracy of region labels and resolve the correspondence between individual location coordinates and crowd density maps. However, directly applying currently popular large foundation models such as SAM does not yield ideal results in dense crowds. To this end, we first propose Dense Point-to-Mask Optimization (DPMO), which integrates SAM with the Nearest Neighbor Exclusive Circle (NNEC) constraint to generate dense instance segmentation from point annotations. With DPMO and manual correction, we obtain mask annotations from the existing point annotations for traditional crowd datasets. Then, to predict instance segmentation in dense crowds, we propose a Reinforced Point Selection (RPS) framework trained with Group Relative Policy Optimization (GRPO), which selects the best predicted point from a sampling of the initial point prediction. Through extensive experiments, we achieve state-of-the-art crowd instance segmentation performance on ShanghaiTech, UCF-QNRF, JHU-CROWD++, and NWPU-Crowd datasets. Furthermore, we design new loss functions supervised by masks that boost counting performance across different models, demonstrating the significant role of mask annotations in enhancing counting accuracy.