Text2VR: Automated instruction Generation in Virtual Reality using Large language Models for Assembly Task

TL;DR

This paper introduces Text2VR, a system that uses large language models to automatically generate immersive VR training instructions from text, reducing development time and enhancing scalability for industrial applications.

Contribution

It presents a novel framework combining LLMs and VR to automate instructional content creation, improving efficiency and adaptability in VR training environments.

Findings

Automated instruction generation improves training scalability.

System reduces development overhead for VR training content.

Enhanced visual cues and animations aid learning effectiveness.

Abstract

Virtual Reality (VR) has emerged as a powerful tool for workforce training, offering immersive, interactive, and risk-free environments that enhance skill acquisition, decision-making, and confidence. Despite its advantages, developing VR applications for training remains a significant challenge due to the time, expertise, and resources required to create accurate and engaging instructional content. To address these limitations, this paper proposes a novel approach that leverages Large Language Models (LLMs) to automate the generation of virtual instructions from textual input. The system comprises two core components: an LLM module that extracts task-relevant information from the text, and an intelligent module that transforms this information into animated demonstrations and visual cues within a VR environment. The intelligent module receives input from the LLM module and interprets the extracted information. Based on this, an instruction generator creates training content using relevant data from a database. The instruction generator generates the instruction by changing the color of virtual objects and creating animations to illustrate tasks. This approach enhances training effectiveness and reduces development overhead, making VR-based training more scalable and adaptable to evolving industrial needs.