Towards Deeper Understanding of Natural User Interactions in Virtual Reality Based Assembly Tasks

TL;DR

This study investigates natural multimodal interactions in VR assembly tasks, analyzing user behaviors and strategies through a Wizard-of-Oz experiment to inform better interface design.

Contribution

It provides new insights into natural interaction strategies in VR assembly tasks and offers a dataset to support future interface development.

Findings

Distinct behaviors in instructive vs. collaborative scenarios

Use of spatial language varies with task clarity

Collected data reveals natural interaction patterns

Abstract

We explore natural user interactions using a virtual reality simulation of a robot arm for assembly tasks. Using a Wizard-of-Oz study, participants completed collaborative LEGO and instructive PCB assembly tasks, with the robot responding under experimenter control. We collected voice, hand tracking, and gaze data from users. Statistical analyses revealed that instructive and collaborative scenarios elicit distinct behaviors and adopted strategies, particularly as tasks progress. Users tended to use put-that-there language in spatially ambiguous contexts and more descriptive instructions in spatially clear ones. Our contributions include the identification of natural interaction strategies through analyses of collected data, as well as the supporting dataset, to guide the understanding and design of natural multimodal user interfaces for instructive interaction with systems in virtual reality.