MM-Gesture: Towards Precise Micro-Gesture Recognition through Multimodal Fusion

TL;DR

MM-Gesture is a multimodal fusion framework that accurately recognizes micro-gestures by integrating multiple data sources and advanced neural architectures, achieving top performance in a challenging benchmark competition.

Contribution

This paper introduces a novel multimodal fusion approach with a modality-weighted ensemble strategy and transfer learning, significantly improving micro-gesture recognition accuracy.

Findings

Achieved 73.213% top-1 accuracy on iMiGUE benchmark.

Outperformed previous state-of-the-art methods in the MiGA Challenge.

Validated effectiveness through extensive ablation studies.

Abstract

In this paper, we present MM-Gesture, the solution developed by our team HFUT-VUT, which ranked 1st in the micro-gesture classification track of the 3rd MiGA Challenge at IJCAI 2025, achieving superior performance compared to previous state-of-the-art methods. MM-Gesture is a multimodal fusion framework designed specifically for recognizing subtle and short-duration micro-gestures (MGs), integrating complementary cues from joint, limb, RGB video, Taylor-series video, optical-flow video, and depth video modalities. Utilizing PoseConv3D and Video Swin Transformer architectures with a novel modality-weighted ensemble strategy, our method further enhances RGB modality performance through transfer learning pre-trained on the larger MA-52 dataset. Extensive experiments on the iMiGUE benchmark, including ablation studies across different modalities, validate the effectiveness of our proposed approach, achieving a top-1 accuracy of 73.213%. Code is available at: https://github.com/momiji-bit/MM-Gesture.