HOReeNet: 3D-aware Hand-Object Grasping Reenactment

TL;DR

HOReeNet is a novel, fully differentiable framework for manipulating images involving hands and objects, enabling transfer and realistic reenactment of hand-object interactions with state-of-the-art results.

Contribution

We introduce HOReeNet, the first fully differentiable model for 3D-aware hand-object manipulation and reenactment in images, integrating 3D reconstruction, contact reasoning, and image refinement.

Findings

Achieved state-of-the-art performance on hand-object interaction datasets.

Outperformed conventional image translation and reenactment algorithms.

Demonstrated effective 3D to 2D projection and manipulation capabilities.

Abstract

We present HOReeNet, which tackles the novel task of manipulating images involving hands, objects, and their interactions. Especially, we are interested in transferring objects of source images to target images and manipulating 3D hand postures to tightly grasp the transferred objects. Furthermore, the manipulation needs to be reflected in the 2D image space. In our reenactment scenario involving hand-object interactions, 3D reconstruction becomes essential as 3D contact reasoning between hands and objects is required to achieve a tight grasp. At the same time, to obtain high-quality 2D images from 3D space, well-designed 3D-to-2D projection and image refinement are required. Our HOReeNet is the first fully differentiable framework proposed for such a task. On hand-object interaction datasets, we compared our HOReeNet to the conventional image translation algorithms and reenactment algorithm. We demonstrated that our approach could achieved the state-of-the-art on the proposed task.