HuPR: A Benchmark for Human Pose Estimation Using Millimeter Wave Radar

TL;DR

This paper presents HuPR, a new benchmark dataset for human pose estimation using millimeter wave radar, along with a cross-modality training framework that improves radar-based pose estimation performance.

Contribution

Introduction of HuPR, a novel radar-based human pose estimation benchmark with synchronized vision data, and a new training framework leveraging monocular images for improved radar pose estimation.

Findings

Radar-based pose estimation outperforms traditional methods.

Cross-modality training improves accuracy with radar data.

Proposed methods are robust in low-light and privacy-sensitive scenarios.

Abstract

This paper introduces a novel human pose estimation benchmark, Human Pose with Millimeter Wave Radar (HuPR), that includes synchronized vision and radio signal components. This dataset is created using cross-calibrated mmWave radar sensors and a monocular RGB camera for cross-modality training of radar-based human pose estimation. There are two advantages of using mmWave radar to perform human pose estimation. First, it is robust to dark and low-light conditions. Second, it is not visually perceivable by humans and thus, can be widely applied to applications with privacy concerns, e.g., surveillance systems in patient rooms. In addition to the benchmark, we propose a cross-modality training framework that leverages the ground-truth 2D keypoints representing human body joints for training, which are systematically generated from the pre-trained 2D pose estimation network based on a monocular camera input image, avoiding laborious manual label annotation efforts. The framework consists of a new radar pre-processing method that better extracts the velocity information from radar data, Cross- and Self-Attention Module (CSAM), to fuse multi-scale radar features, and Pose Refinement Graph Convolutional Networks (PRGCN), to refine the predicted keypoint confidence heatmaps. Our intensive experiments on the HuPR benchmark show that the proposed scheme achieves better human pose estimation performance with only radar data, as compared to traditional pre-processing solutions and previous radio-frequency-based methods.