Occlusion-Aware 3D Hand-Object Pose Estimation with Masked AutoEncoders

TL;DR

This paper introduces HOMAE, a novel occlusion-aware method for 3D hand-object pose estimation using masked autoencoders, which effectively handles occlusions by learning context-aware features and combining implicit and explicit geometric representations.

Contribution

The paper proposes a target-focused masking strategy and a fusion of signed distance fields with point clouds to improve occlusion handling in hand-object pose estimation.

Findings

Achieves state-of-the-art results on DexYCB and HO3Dv2 benchmarks.

Effectively models occluded regions by combining global context and local geometry.

Demonstrates robustness in challenging occlusion scenarios.

Abstract

Hand-object pose estimation from monocular RGB images remains a significant challenge mainly due to the severe occlusions inherent in hand-object interactions. Existing methods do not sufficiently explore global structural perception and reasoning, which limits their effectiveness in handling occluded hand-object interactions. To address this challenge, we propose an occlusion-aware hand-object pose estimation method based on masked autoencoders, termed as HOMAE. Specifically, we propose a target-focused masking strategy that imposes structured occlusion on regions of hand-object interaction, encouraging the model to learn context-aware features and reason about the occluded structures. We further integrate multi-scale features extracted from the decoder to predict a signed distance field (SDF), capturing both global context and fine-grained geometry. To enhance geometric perception, we combine the implicit SDF with an explicit point cloud derived from the SDF, leveraging the complementary strengths of both representations. This fusion enables more robust handling of occluded regions by combining the global context from the SDF with the precise local geometry provided by the point cloud. Extensive experiments on challenging DexYCB and HO3Dv2 benchmarks demonstrate that HOMAE achieves state-of-the-art performance in hand-object pose estimation. We will release our code and model.