ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping

TL;DR

ManiVideo is a new method that generates realistic, temporally consistent videos of hand-object manipulation by learning 3D occlusion relationships and leveraging large-scale 3D object datasets for better generalization.

Contribution

The paper introduces the MLO representation and integrates it into a UNet architecture, enabling improved 3D consistency and generalizable grasping in manipulation videos.

Findings

Outperforms existing state-of-the-art methods.

Produces videos with plausible hand-object interactions.

Demonstrates effective generalization to various objects.

Abstract

In this paper, we introduce ManiVideo, a novel method for generating consistent and temporally coherent bimanual hand-object manipulation videos from given motion sequences of hands and objects. The core idea of ManiVideo is the construction of a multi-layer occlusion (MLO) representation that learns 3D occlusion relationships from occlusion-free normal maps and occlusion confidence maps. By embedding the MLO structure into the UNet in two forms, the model enhances the 3D consistency of dexterous hand-object manipulation. To further achieve the generalizable grasping of objects, we integrate Objaverse, a large-scale 3D object dataset, to address the scarcity of video data, thereby facilitating the learning of extensive object consistency. Additionally, we propose an innovative training strategy that effectively integrates multiple datasets, supporting downstream tasks such as human-centric hand-object manipulation video generation. Through extensive experiments, we demonstrate that our approach not only achieves video generation with plausible hand-object interaction and generalizable objects, but also outperforms existing SOTA methods.