FunHOI: Annotation-Free 3D Hand-Object Interaction Generation via Functional Text Guidanc

TL;DR

This paper introduces FunHOI, a novel framework that generates functional 3D hand-object interactions from text without needing annotated 3D data, improving the realism and utility of AI-driven interaction modeling.

Contribution

The paper presents FGS-Net, a two-stage, annotation-free method for synthesizing functional 3D hand-object interactions guided solely by text descriptions.

Findings

Achieves high-quality, realistic HOI generation without 3D annotations.

Outperforms existing methods in generating functionally accurate grasps.

Demonstrates effectiveness across diverse hand-object interaction scenarios.

Abstract

Hand-object interaction(HOI) is the fundamental link between human and environment, yet its dexterous and complex pose significantly challenges for gesture control. Despite significant advances in AI and robotics, enabling machines to understand and simulate hand-object interactions, capturing the semantics of functional grasping tasks remains a considerable challenge. While previous work can generate stable and correct 3D grasps, they are still far from achieving functional grasps due to unconsidered grasp semantics. To address this challenge, we propose an innovative two-stage framework, Functional Grasp Synthesis Net (FGS-Net), for generating 3D HOI driven by functional text. This framework consists of a text-guided 3D model generator, Functional Grasp Generator (FGG), and a pose optimization strategy, Functional Grasp Refiner (FGR). FGG generates 3D models of hands and objects based on text input, while FGR fine-tunes the poses using Object Pose Approximator and energy functions to ensure the relative position between the hand and object aligns with human intent and remains physically plausible. Extensive experiments demonstrate that our approach achieves precise and high-quality HOI generation without requiring additional 3D annotation data.