HOICraft: In-Situ VLM-based Authoring Tool for Part-Level Hand-Object Interaction Design in VR

TL;DR

HOICraft is an innovative in-situ VR authoring tool that leverages vision-language models to facilitate part-level hand-object interaction design, reducing manual effort and iteration time for VR designers.

Contribution

This paper introduces HOICraft, a novel VLM-based tool enabling in-situ, part-level HOI design in VR, with automated recommendations and mapping to streamline the process.

Findings

Reduced iteration time in HOI design

Effective in-context learning for HOI mapping

High usability scores from designers

Abstract

Hand-Object Interaction (HOI) is a key interaction component in Virtual Reality (VR). However, designing HOI still requires manual efforts to decide how object should be selected and manipulated, while also considering user abilities, which leads to time-consuming refinements. We present HOICraft, a VLM-based in-situ HOI authoring tool that enables part-level interaction design in VR. Here, HOICraft assists designers by recommending interactable elements from 3D objects, customizing HOI design properties, and mapping hand movement with virtual object behavior. We conducted a formative study with three expert VR designers to identify five representative HOI designs to support diverse user experiences. Building upon preference data from 20 participants, we develop an HOI mapping module with in-context learning. In a user study with 12 VR interaction designers, HOI mapping from HOICraft significantly reduced trial-and-error iterations compared to manual authoring. Finally, we assessed the usability of HOICraft, demonstrating its effectiveness for HOI design in VR.