mmPred: Radar-based Human Motion Prediction in the Dark

TL;DR

This paper introduces mmPred, a novel radar-based human motion prediction framework using diffusion models, which is robust to lighting and privacy concerns, and outperforms existing RGB-D methods.

Contribution

The work pioneers the use of radar for human motion prediction with a diffusion-based approach, addressing radar-specific noise and artifacts with dual-domain representations and a transformer backbone.

Findings

Achieves 8.6% improvement on mmBody dataset

Achieves 22% improvement on mm-Fi dataset

Outperforms existing methods significantly

Abstract

Existing Human Motion Prediction (HMP) methods based on RGB-D cameras are sensitive to lighting conditions and raise privacy concerns, limiting their real-world applications such as firefighting and healthcare. Motivated by the robustness and privacy-preserving nature of millimeter-wave (mmWave) radar, this work introduces radar as a novel sensing modality for HMP, for the first time. Nevertheless, radar signals often suffer from specular reflections and multipath effects, resulting in noisy and temporally inconsistent measurements, such as body-part miss-detection. To address these radar-specific artifacts, we propose mmPred, the first diffusion-based framework tailored for radar-based HMP. mmPred introduces a dual-domain historical motion representation to guide the generation process, combining a Time-domain Pose Refinement (TPR) branch for learning fine-grained details and a Frequency-domain Dominant Motion (FDM) branch for capturing global motion trends and suppressing frame-level inconsistency. Furthermore, we design a Global Skeleton-relational Transformer (GST) as the diffusion backbone to model global inter-joint cooperation, enabling corrupted joints to dynamically aggregate information from others. Extensive experiments show that mmPred achieves state-of-the-art performance, outperforming existing methods by 8.6% on mmBody and 22% on mm-Fi.