mmEgoHand: Egocentric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMU
Yizhe Lv, Tingting Zhang, Zhijian Wang, Yunpeng Song, Han Ding, Jinsong Han, Fei Wang

TL;DR
mmEgoHand is a head-mounted system combining mmWave radar and IMUs with a Transformer model to accurately estimate 3D hand poses and recognize gestures in egocentric scenarios, enabling advanced human-computer interaction.
Contribution
The paper introduces mmEgoHand, a novel head-mounted egocentric system that fuses mmWave radar and IMUs with a Transformer architecture for robust hand pose estimation and gesture recognition.
Findings
Achieves 90.8% gesture recognition accuracy across various postures.
Outperforms existing methods significantly in egocentric hand pose tasks.
Demonstrates effective sensor fusion and viewpoint compensation in dynamic scenarios.
Abstract
Recent advancements in millimeter-wave (mmWave) radar have demonstrated its potential for human action recognition and pose estimation, offering privacy-preserving advantages over conventional cameras while maintaining occlusion robustness, with promising applications in human-computer interaction and wellness care. However, existing mmWave systems typically employ fixed-position configurations, restricting user mobility to predefined zones and limiting practical deployment scenarios. We introduce mmEgoHand, a head-mounted egocentric system for hand pose estimation to support applications such as gesture recognition, VR interaction, skill digitization and assessment, and robotic teleoperation. mmEgoHand synergistically integrates mmWave radar with inertial measurement units (IMUs) to enable dynamic perception. The IMUs actively compensate for radar interference induced by head…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHand Gesture Recognition Systems · Facial Nerve Paralysis Treatment and Research
MethodsAttention Is All You Need · Adam · Softmax · Absolute Position Encodings · Residual Connection · Dropout · Byte Pair Encoding · Linear Layer · Multi-Head Attention · Position-Wise Feed-Forward Layer
