TIGeR: Text-Instructed Generation and Refinement for Template-Free Hand-Object Interaction

TL;DR

TIGeR introduces a text-driven framework for 3D object shape refinement and pose estimation in hand-object interactions, reducing manual effort and improving adaptability to occlusions.

Contribution

The paper presents a novel two-stage text-instructed generation and refinement framework that leverages text priors and vision guidance for template-free 3D object reconstruction.

Findings

Achieves competitive Chamfer distances on Dex-YCB and Obman datasets.

Demonstrates robustness to occlusion in object reconstruction.

Maintains compatibility with heterogeneous prior sources.

Abstract

Pre-defined 3D object templates are widely used in 3D reconstruction of hand-object interactions. However, they often require substantial manual efforts to capture or source, and inherently restrict the adaptability of models to unconstrained interaction scenarios, e.g., heavily-occluded objects. To overcome this bottleneck, we propose a new Text-Instructed Generation and Refinement (TIGeR) framework, harnessing the power of intuitive text-driven priors to steer the object shape refinement and pose estimation. We use a two-stage framework: a text-instructed prior generation and vision-guided refinement. As the name implies, we first leverage off-the-shelf models to generate shape priors according to the text description without tedious 3D crafting. Considering the geometric gap between the synthesized prototype and the real object interacted with the hand, we further calibrate the synthesized prototype via 2D-3D collaborative attention. TIGeR achieves competitive performance, i.e., 1.979 and 5.468 object Chamfer distance on the widely-used Dex-YCB and Obman datasets, respectively, surpassing existing template-free methods. Notably, the proposed framework shows robustness to occlusion, while maintaining compatibility with heterogeneous prior sources, e.g., retrieved hand-crafted prototypes, in practical deployment scenarios.