A Virtual Reality Simulation Pipeline for Online Mental Workload Modeling

TL;DR

This paper presents a novel VR simulation pipeline that integrates multimodal biosignal data for real-time mental workload modeling, enhancing human-robot interaction and adaptive system responses in high-stress scenarios.

Contribution

It introduces a VR system inspired by NASA's MATB-II that synchronously collects performance and biosignal data for online mental workload estimation.

Findings

Developed a VR pipeline capable of real-time biosignal collection and processing.

Facilitated integration of psychophysiological data into MW models via ROS.

Enabled adaptive responses in VR environments based on operator mental workload.

Abstract

Seamless human robot interaction (HRI) and cooperative human-robot (HR) teaming critically rely upon accurate and timely human mental workload (MW) models. Cognitive Load Theory (CLT) suggests representative physical environments produce representative mental processes; physical environment fidelity corresponds with improved modeling accuracy. Virtual Reality (VR) systems provide immersive environments capable of replicating complicated scenarios, particularly those associated with high-risk, high-stress scenarios. Passive biosignal modeling shows promise as a noninvasive method of MW modeling. However, VR systems rarely include multimodal psychophysiological feedback or capitalize on biosignal data for online MW modeling. Here, we develop a novel VR simulation pipeline, inspired by the NASA Multi-Attribute Task Battery II (MATB-II) task architecture, capable of synchronous collection of objective performance, subjective performance, and passive human biosignals in a simulated hazardous exploration environment. Our system design extracts and publishes biofeatures through the Robot Operating System (ROS), facilitating real time psychophysiology-based MW model integration into complete end-to-end systems. A VR simulation pipeline capable of evaluating MWs online could be foundational for advancing HR systems and VR experiences by enabling these systems to adaptively alter their behaviors in response to operator MW.