M4Human: A Large-Scale Multimodal mmWave Radar Benchmark for Human Mesh Reconstruction

TL;DR

M4Human is the largest multimodal radar dataset for human mesh reconstruction, combining high-resolution mmWave radar, RGB, and depth data with extensive annotations to advance RF-based human modeling research.

Contribution

It introduces M4Human, a large-scale, multimodal benchmark dataset with high-quality annotations, enabling new research in radar-based human mesh reconstruction.

Findings

Benchmark results highlight the importance of multimodal data for human modeling.

Challenges remain in modeling fast, unconstrained motions.

The dataset supports research across raw radar, point clouds, and multimodal fusion.

Abstract

Human mesh reconstruction (HMR) provides direct insights into body-environment interaction, which enables various immersive applications. While existing large-scale HMR datasets rely heavily on line-of-sight RGB input, vision-based sensing is limited by occlusion, lighting variation, and privacy concerns. To overcome these limitations, recent efforts have explored radio-frequency (RF) mmWave radar for privacy-preserving indoor human sensing. However, current radar datasets are constrained by sparse skeleton labels, limited scale, and simple in-place actions. To advance the HMR research community, we introduce M4Human, the current largest-scale (661K-frame) ($9\times$ prior largest) multimodal benchmark, featuring high-resolution mmWave radar, RGB, and depth data. M4Human provides both raw radar tensors (RT) and processed radar point clouds (RPC) to enable research across different levels of RF signal granularity. M4Human includes high-quality motion capture (MoCap) annotations with 3D meshes and global trajectories, and spans 20 subjects and 50 diverse actions, including in-place, sit-in-place, and free-space sports or rehabilitation movements. We establish benchmarks on both RT and RPC modalities, as well as multimodal fusion with RGB-D modalities. Extensive results highlight the significance of M4Human for radar-based human modeling while revealing persistent challenges under fast, unconstrained motion. The dataset and code will be released after the paper publication.