Detection, Pose Estimation and Segmentation for Multiple Bodies: Closing the Virtuous Circle

TL;DR

This paper introduces BBox-Mask-Pose, a novel iterative framework that jointly improves detection, segmentation, and pose estimation for multiple humans, especially in crowded scenes, achieving state-of-the-art results.

Contribution

The paper proposes a closed-loop, multi-model approach that enforces mutual consistency among bounding boxes, masks, and poses, advancing multi-human analysis in complex scenes.

Findings

Achieves SOTA on OCHuman for detection, segmentation, and pose estimation.

Improves detection accuracy by 39% in overlapping scenes.

Outperforms existing top-down methods on COCO pose estimation.

Abstract

Human pose estimation methods work well on isolated people but struggle with multiple-bodies-in-proximity scenarios. Previous work has addressed this problem by conditioning pose estimation by detected bounding boxes or keypoints, but overlooked instance masks. We propose to iteratively enforce mutual consistency of bounding boxes, instance masks, and poses. The introduced BBox-Mask-Pose (BMP) method uses three specialized models that improve each other's output in a closed loop. All models are adapted for mutual conditioning, which improves robustness in multi-body scenes. MaskPose, a new mask-conditioned pose estimation model, is the best among top-down approaches on OCHuman. BBox-Mask-Pose pushes SOTA on OCHuman dataset in all three tasks - detection, instance segmentation, and pose estimation. It also achieves SOTA performance on COCO pose estimation. The method is especially good in scenes with large instances overlap, where it improves detection by 39% over the baseline detector. With small specialized models and faster runtime, BMP is an effective alternative to large human-centered foundational models. Code and models are available on https://MiraPurkrabek.github.io/BBox-Mask-Pose.