Can You Move These Over There? An LLM-based VR Mover for Supporting Object Manipulation

TL;DR

This paper introduces VR Mover, an LLM-powered system that enables natural language and gesture-based object manipulation in VR, improving usability and reducing fatigue.

Contribution

It presents a novel LLM-based interface for VR object manipulation that supports natural speech and gestures, enhancing user experience and interaction efficiency.

Findings

Enhanced user usability and experience with VR Mover

Reduced workload and arm fatigue during manipulation tasks

Users prefer natural language and gesture interactions for broad movements

Abstract

In our daily lives, we can naturally convey instructions for the spatial manipulation of objects using words and gestures. Transposing this form of interaction into virtual reality (VR) object manipulation can be beneficial. We propose VR Mover, an LLM-empowered solution that can understand and interpret the user's vocal instruction to support object manipulation. By simply pointing and speaking, the LLM can manipulate objects without structured input. Our user study demonstrates that VR Mover enhances user usability, overall experience and performance on multi-object manipulation, while also reducing workload and arm fatigue. Users prefer the proposed natural interface for broad movements and may complementarily switch to gizmos or virtual hands for finer adjustments. These findings are believed to contribute to design implications for future LLM-based object manipulation interfaces, highlighting the potential for more intuitive and efficient user interactions in VR environments.