NL2Contact: Natural Language Guided 3D Hand-Object Contact Modeling with Diffusion Model

TL;DR

NL2Contact introduces a controllable 3D hand-object contact modeling approach guided by natural language, enabling realistic contact generation and grasp optimization, supported by a new contact description dataset.

Contribution

The paper presents NL2Contact, a diffusion model-based system for controllable contact modeling from language, and introduces ContactDescribe, a novel dataset with diverse contact descriptions.

Findings

Successfully generates realistic hand-object contacts from text.

Enables grasp pose optimization based on natural language descriptions.

Demonstrates improved grasp diversity and accuracy.

Abstract

Modeling the physical contacts between the hand and object is standard for refining inaccurate hand poses and generating novel human grasp in 3D hand-object reconstruction. However, existing methods rely on geometric constraints that cannot be specified or controlled. This paper introduces a novel task of controllable 3D hand-object contact modeling with natural language descriptions. Challenges include i) the complexity of cross-modal modeling from language to contact, and ii) a lack of descriptive text for contact patterns. To address these issues, we propose NL2Contact, a model that generates controllable contacts by leveraging staged diffusion models. Given a language description of the hand and contact, NL2Contact generates realistic and faithful 3D hand-object contacts. To train the model, we build \textit{ContactDescribe}, the first dataset with hand-centered contact descriptions. It contains multi-level and diverse descriptions generated by large language models based on carefully designed prompts (e.g., grasp action, grasp type, contact location, free finger status). We show applications of our model to grasp pose optimization and novel human grasp generation, both based on a textual contact description.