Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion

TL;DR

This paper introduces a joint hand-object 3D reconstruction method from a single RGB image that leverages cross-branch feature fusion and depth estimation to improve accuracy, outperforming existing methods.

Contribution

It proposes a novel joint feature fusion architecture with LSTM units and auxiliary depth estimation for enhanced hand-object 3D reconstruction.

Findings

LSTM-based feature fusion outperforms other architectures.

Auxiliary depth estimation improves reconstruction accuracy.

The method significantly surpasses existing approaches on public datasets.

Abstract

Accurate 3D reconstruction of the hand and object shape from a hand-object image is important for understanding human-object interaction as well as human daily activities. Different from bare hand pose estimation, hand-object interaction poses a strong constraint on both the hand and its manipulated object, which suggests that hand configuration may be crucial contextual information for the object, and vice versa. However, current approaches address this task by training a two-branch network to reconstruct the hand and object separately with little communication between the two branches. In this work, we propose to consider hand and object jointly in feature space and explore the reciprocity of the two branches. We extensively investigate cross-branch feature fusion architectures with MLP or LSTM units. Among the investigated architectures, a variant with LSTM units that enhances object feature with hand feature shows the best performance gain. Moreover, we employ an auxiliary depth estimation module to augment the input RGB image with the estimated depth map, which further improves the reconstruction accuracy. Experiments conducted on public datasets demonstrate that our approach significantly outperforms existing approaches in terms of the reconstruction accuracy of objects.