AI-Powered Immersive Assistance for Interactive Task Execution in Industrial Environments
Tomislav Duricic, Peter M\"ullner, Nicole Weidinger, Neven ElSayed,, Dominik Kowald, Eduardo Veas

TL;DR
This paper presents an AI-powered immersive assistance system using VR and multimodal AI models to guide users through complex industrial tasks, aiming to improve safety and efficiency.
Contribution
It introduces a novel VR-based digital twin with integrated AI assistance leveraging large language and speech-to-text models for industrial task support.
Findings
Demonstrated effective step-by-step guidance in a simulated industrial environment
Reduced cognitive load for users performing complex tasks
Potential to enhance safety and productivity in industrial settings
Abstract
Many industrial sectors rely on well-trained employees that are able to operate complex machinery. In this work, we demonstrate an AI-powered immersive assistance system that supports users in performing complex tasks in industrial environments. Specifically, our system leverages a VR environment that resembles a juice mixer setup. This digital twin of a physical setup simulates complex industrial machinery used to mix preparations or liquids (e.g., similar to the pharmaceutical industry) and includes various containers, sensors, pumps, and flow controllers. This setup demonstrates our system's capabilities in a controlled environment while acting as a proof-of-concept for broader industrial applications. The core components of our multimodal AI assistant are a large language model and a speech-to-text model that process a video and audio recording of an expert performing the task in a…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAugmented Reality Applications
