Open-Vocabulary Functional 3D Human-Scene Interaction Generation

TL;DR

This paper introduces FunHSI, a framework that generates functionally correct 3D human-scene interactions from open-vocabulary prompts by reasoning about object functionality and contact, improving plausibility and diversity.

Contribution

FunHSI is a training-free, functionality-driven method that explicitly models object functions and contact reasoning to produce realistic 3D human-scene interactions from open prompts.

Findings

Generates more plausible 3D human-scene interactions than existing methods.

Supports fine-grained functional interactions like adjusting room temperature.

Works across diverse indoor and outdoor scenes.

Abstract

Generating 3D humans that functionally interact with 3D scenes remains an open problem with applications in embodied AI, robotics, and interactive content creation. The key challenge involves reasoning about both the semantics of functional elements in 3D scenes and the 3D human poses required to achieve functionality-aware interaction. Unfortunately, existing methods typically lack explicit reasoning over object functionality and the corresponding human-scene contact, resulting in implausible or functionally incorrect interactions. In this work, we propose FunHSI, a training-free, functionality-driven framework that enables functionally correct human-scene interactions from open-vocabulary task prompts. Given a task prompt, FunHSI performs functionality-aware contact reasoning to identify functional scene elements, reconstruct their 3D geometry, and model high-level interactions via a contact graph. We then leverage vision-language models to synthesize a human performing the task in the image and estimate proposed 3D body and hand poses. Finally, the proposed 3D body configuration is refined via stage-wise optimization to ensure physical plausibility and functional correctness. In contrast to existing methods, FunHSI not only synthesizes more plausible general 3D interactions, such as "sitting on a sofa'', while supporting fine-grained functional human-scene interactions, e.g., "increasing the room temperature''. Extensive experiments demonstrate that FunHSI consistently generates functionally correct and physically plausible human-scene interactions across diverse indoor and outdoor scenes.