Hands-Free VR
Jorge Askur Vazquez Fernandez, Jae Joong Lee, Santiago Andr\'es, Serrano Vacca, Alejandra Magana, Radim Pesam, Bedrich Benes, Voicu Popescu

TL;DR
This paper presents Hands-Free VR, a voice-controlled VR interface using deep learning for speech recognition and large language models for command mapping, demonstrating high robustness, efficiency, and user preference in a controlled study.
Contribution
It introduces a novel voice-based natural-language interface for VR that is robust to accents and language diversity, with validated efficiency and user satisfaction improvements.
Findings
96.71% command transcription accuracy
97.83% correct command mapping
Significant efficiency improvements over traditional interfaces
Abstract
The paper introduces Hands-Free VR, a voice-based natural-language interface for VR. The user gives a command using their voice, the speech audio data is converted to text using a speech-to-text deep learning model that is fine-tuned for robustness to word phonetic similarity and to spoken English accents, and the text is mapped to an executable VR command using a large language model that is robust to natural language diversity. Hands-Free VR was evaluated in a controlled within-subjects study (N = 22) that asked participants to find specific objects and to place them in various configurations. In the control condition participants used a conventional VR user interface to grab, carry, and position the objects using the handheld controllers. In the experimental condition participants used Hands-Free VR. The results confirm that: (1) Hands-Free VR is robust to spoken English accents, as…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsVirtual Reality Applications and Impacts
