MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video

TL;DR

MAEPose introduces a self-supervised, spatiotemporal learning method for human pose estimation directly from mmWave radar videos, outperforming existing approaches and reducing system complexity.

Contribution

It presents MAEPose, a novel masked autoencoding framework that learns from unlabelled radar videos and improves pose estimation accuracy without relying on intermediate representations.

Findings

Outperforms state-of-the-art by up to 22.1% in MPJPE

Maintains robustness under zero-shot bystander interference with only 6.5% error increase

Leverages Range-Doppler videos for better performance and lower computational cost

Abstract

Millimetre-wave (mmWave) radar offers a more privacy-preserving alternative to RGB-based human pose estimation. However, existing methods typically rely on pre-extracted intermediate representations such as sparse point clouds or spectrogram images, where the rich spatiotemporal information naturally present in radar video streams is discarded for model learning, while such signal processing adds system complexity. In addition, existing solutions are mainly conducted in an end-to-end supervised manner without leveraging unlabelled raw video streams to learn generalized representations. In this study, we present MAEPose, a masked autoencoding-based human pose estimation approach that operates directly on mmWave spectrogram videos. MAEPose learns spatiotemporal motion-aware generalized representations from unlabelled radar video, and leverages its heatmap decoder for multi-frame pose estimation predictions. We evaluate it across three datasets based on leave-one-person-out cross-validation with rigorous statistical testing. MAEPose consistently outperforms state-of-the-art baselines by up to 22.1% in MPJPE p<0.05, and maintains robust accuracy under zero-shot bystander interference with only a 6.5% error increase. Ablation studies confirm that both the pre-training and the heatmap decoder contribute substantially, while modality analysis indicates that leveraging Range-Doppler video as input achieves better pose estimation performance than Range-Azimuth or their fusion, with lower computational cost.