DF-Mamba: Deformable State Space Modeling for 3D Hand Pose Estimation in Interactions

TL;DR

This paper introduces DF-Mamba, a deformable state space model that enhances 3D hand pose estimation by capturing global context and handling occlusions more effectively than traditional CNN-based methods.

Contribution

The paper proposes DF-Mamba, a novel deformable state space framework that improves feature extraction for 3D hand pose estimation, outperforming existing backbones across multiple datasets.

Findings

DF-Mamba achieves state-of-the-art accuracy on five diverse datasets.

It significantly outperforms existing backbones like VMamba and Spatial-Mamba.

The method maintains comparable inference speed to ResNet-50.

Abstract

Modeling daily hand interactions often struggles with severe occlusions, such as when two hands overlap, which highlights the need for robust feature learning in 3D hand pose estimation (HPE). To handle such occluded hand images, it is vital to effectively learn the relationship between local image features (e.g., for occluded joints) and global context (e.g., cues from inter-joints, inter-hands, or the scene). However, most current 3D HPE methods still rely on ResNet for feature extraction, and such CNN's inductive bias may not be optimal for 3D HPE due to its limited capability to model the global context. To address this limitation, we propose an effective and efficient framework for visual feature extraction in 3D HPE using recent state space modeling (i.e., Mamba), dubbed Deformable Mamba (DF-Mamba). DF-Mamba is designed to capture global context cues beyond standard convolution through Mamba's selective state modeling and the proposed deformable state scanning. Specifically, for local features after convolution, our deformable scanning aggregates these features within an image while selectively preserving useful cues that represent the global context. This approach significantly improves the accuracy of structured 3D HPE, with comparable inference speed to ResNet-50. Our experiments involve extensive evaluations on five divergent datasets including single-hand and two-hand scenarios, hand-only and hand-object interactions, as well as RGB and depth-based estimation. DF-Mamba outperforms the latest image backbones, including VMamba and Spatial-Mamba, on all datasets and achieves state-of-the-art performance.