mmBody Benchmark: 3D Body Reconstruction Dataset and Analysis for Millimeter Wave Radar

TL;DR

This paper introduces a large-scale dataset and analysis for 3D human body reconstruction using millimeter wave radar, comparing its performance with RGB and depth cameras across various scenes and weather conditions.

Contribution

It provides the first large-scale, synchronized dataset combining mmWave radar with RGB(D) images for 3D body reconstruction and evaluates the reconstruction accuracy across different sensors and environments.

Findings

mmWave radar achieves better accuracy than RGB but worse than depth cameras

Radar reconstruction is moderately affected by adverse weather

Combining sensors can improve 3D reconstruction robustness

Abstract

Millimeter Wave (mmWave) Radar is gaining popularity as it can work in adverse environments like smoke, rain, snow, poor lighting, etc. Prior work has explored the possibility of reconstructing 3D skeletons or meshes from the noisy and sparse mmWave Radar signals. However, it is unclear how accurately we can reconstruct the 3D body from the mmWave signals across scenes and how it performs compared with cameras, which are important aspects needed to be considered when either using mmWave radars alone or combining them with cameras. To answer these questions, an automatic 3D body annotation system is first designed and built up with multiple sensors to collect a large-scale dataset. The dataset consists of synchronized and calibrated mmWave radar point clouds and RGB(D) images in different scenes and skeleton/mesh annotations for humans in the scenes. With this dataset, we train state-of-the-art methods with inputs from different sensors and test them in various scenarios. The results demonstrate that 1) despite the noise and sparsity of the generated point clouds, the mmWave radar can achieve better reconstruction accuracy than the RGB camera but worse than the depth camera; 2) the reconstruction from the mmWave radar is affected by adverse weather conditions moderately while the RGB(D) camera is severely affected. Further, analysis of the dataset and the results shadow insights on improving the reconstruction from the mmWave radar and the combination of signals from different sensors.