Proactive Multi-Camera Collaboration For 3D Human Pose Estimation

TL;DR

This paper introduces a multi-agent reinforcement learning approach with a novel reward mechanism for proactive multi-camera collaboration, significantly improving 3D human pose estimation in dynamic, occlusion-prone environments.

Contribution

It proposes a new Collaborative Triangulation Contribution Reward (CTCR) and joint training with environment dynamics learning to enhance multi-camera coordination and pose estimation accuracy.

Findings

Outperforms fixed and active camera baselines in diverse scenarios

Improves convergence and credit assignment in multi-agent settings

Demonstrates robustness across different numbers of cameras and humans

Abstract

This paper presents a multi-agent reinforcement learning (MARL) scheme for proactive Multi-Camera Collaboration in 3D Human Pose Estimation in dynamic human crowds. Traditional fixed-viewpoint multi-camera solutions for human motion capture (MoCap) are limited in capture space and susceptible to dynamic occlusions. Active camera approaches proactively control camera poses to find optimal viewpoints for 3D reconstruction. However, current methods still face challenges with credit assignment and environment dynamics. To address these issues, our proposed method introduces a novel Collaborative Triangulation Contribution Reward (CTCR) that improves convergence and alleviates multi-agent credit assignment issues resulting from using 3D reconstruction accuracy as the shared reward. Additionally, we jointly train our model with multiple world dynamics learning tasks to better capture environment dynamics and encourage anticipatory behaviors for occlusion avoidance. We evaluate our proposed method in four photo-realistic UE4 environments to ensure validity and generalizability. Empirical results show that our method outperforms fixed and active baselines in various scenarios with different numbers of cameras and humans.