Voice2Action: Language Models as Agent for Efficient Real-Time Interaction in Virtual Reality

TL;DR

Voice2Action introduces a hierarchical framework that enables efficient, accurate, real-time interaction in virtual reality by analyzing voice commands and environment feedback, addressing challenges in deploying language model agents in complex 3D environments.

Contribution

It presents a novel hierarchical analysis framework for voice commands in VR, improving efficiency and accuracy of LLM-based agents in complex 3D environments.

Findings

Outperforms non-optimized approaches in efficiency and accuracy

Effective in real-time urban VR environment with synthetic data

Reduces errors through environment feedback integration

Abstract

Large Language Models (LLMs) are trained and aligned to follow natural language instructions with only a handful of examples, and they are prompted as task-driven autonomous agents to adapt to various sources of execution environments. However, deploying agent LLMs in virtual reality (VR) has been challenging due to the lack of efficiency in online interactions and the complex manipulation categories in 3D environments. In this work, we propose Voice2Action, a framework that hierarchically analyzes customized voice signals and textual commands through action and entity extraction and divides the execution tasks into canonical interaction subsets in real-time with error prevention from environment feedback. Experiment results in an urban engineering VR environment with synthetic instruction data show that Voice2Action can perform more efficiently and accurately than approaches without optimizations.