B-MoE: A Body-Part-Aware Mixture-of-Experts "All Parts Matter" Approach to Micro-Action Recognition

TL;DR

B-MoE is a novel body-part-aware mixture-of-experts framework that improves micro-action recognition by modeling structured human motion and capturing subtle, short-duration actions.

Contribution

The paper introduces B-MoE, a body-part-aware Mixture-of-Experts model with a dual-stream encoder and cross-attention routing for enhanced micro-action recognition.

Findings

Achieves state-of-the-art results on MA-52, SocialGesture, and MPII-GroupInteraction benchmarks.

Improves recognition of ambiguous, underrepresented, and low-amplitude micro-actions.

Effectively models structured human motion and inter-region relationships.

Abstract

Micro-actions, fleeting and low-amplitude motions, such as glances, nods, or minor posture shifts, carry rich social meaning but remain difficult for current action recognition models to recognize due to their subtlety, short duration, and high inter-class ambiguity. In this paper, we introduce B-MoE, a Body-part-aware Mixture-of-Experts framework designed to explicitly model the structured nature of human motion. In B-MoE, each expert specializes in a distinct body region (head, body, upper limbs, lower limbs), and is based on the lightweight Macro-Micro Motion Encoder (M3E) that captures long-range contextual structure and fine-grained local motion. A cross-attention routing mechanism learns inter-region relationships and dynamically selects the most informative regions for each micro-action. B-MoE uses a dual-stream encoder that fuses these region-specific semantic cues with global motion features to jointly capture spatially localized cues and temporally subtle variations that characterize micro-actions. Experiments on three challenging benchmarks (MA-52, SocialGesture, and MPII-GroupInteraction) show consistent state-of-theart gains, with improvements in ambiguous, underrepresented, and low amplitude classes.