Towards Balanced Multi-Modal Learning in 3D Human Pose Estimation

TL;DR

This paper introduces a balanced multi-modal learning approach for 3D human pose estimation that effectively combines RGB, LiDAR, mmWave, and WiFi data, addressing modality imbalance with a novel contribution assessment and regulation strategy.

Contribution

It presents a new method using Shapley value-based contribution assessment and a learning regulation strategy to balance modalities in 3D pose estimation.

Findings

Outperforms existing methods on the MM-Fi dataset

Effectively detects and mitigates modality imbalance

Enhances pose estimation accuracy in complex scenarios

Abstract

3D human pose estimation (3D HPE) has emerged as a prominent research topic, particularly in the realm of RGB-based methods. However, the use of RGB images is often limited by issues such as occlusion and privacy constraints. Consequently, multi-modal sensing, which leverages non-intrusive sensors, is gaining increasing attention. Nevertheless, multi-modal 3D HPE still faces challenges, including modality imbalance. In this work, we introduce a novel balanced multi-modal learning method for 3D HPE, which harnesses the power of RGB, LiDAR, mmWave, and WiFi. Specifically, we propose a Shapley value-based contribution algorithm to assess the contribution of each modality and detect modality imbalance. To address this imbalance, we design a modality learning regulation strategy that decelerates the learning process during the early stages of training. We conduct extensive experiments on the widely adopted multi-modal dataset, MM-Fi, demonstrating the superiority of our approach in enhancing 3D pose estimation under complex conditions. Our source code is available at https://github.com/MICLAB-BUPT/AWC.