A Multimodal Virtual Reality Data Acquisition Platform and Dataset to Assess Systemic Human Cognitive States

TL;DR

This paper introduces a virtual reality platform and dataset for studying human cognitive states using multiple sensors during tasks like driving.

Contribution

A new multimodal VR platform and dataset for capturing synchronized cognitive state data in real-time during complex tasks.

Findings

The platform enables real-time recording of fNIRS, EEG, and other physiological signals during interactive tasks.

The dataset includes data from 80 subjects performing driving and secondary tasks like braking and dialogue.

The framework supports developing models to predict cognitive states like workload and distraction.

Abstract

In recent years, there has been an increasing interest in human-machine teaming for search and rescue operations, deep space missions, and agricultural tasks, among others. To be effective teammates, artificial agents should be able to detect and be responsive to systemic human cognitive states such as workload, sense of urgency, mind wandering, interference, and distraction. Here, we introduce an experimental paradigm and a comprehensive multimodal dataset that provides the necessary data for analyzing the relationships among multiple systematic human cognitive states, enables the development of robust prediction models of these states, and details the framework for developing new experiments. The introduced experimental setup allows for the synchronized real-time recording of multiple data streams from various sensing devices including fNIRS, EEG, pupillometry, respiration, electrodermal activity, and plethysmography, and is applicable in many other interactive task settings where human performance needs to be monitored. The dataset was acquired from 80 subjects performing a driving task and several secondary tasks including car braking events, dialogue communications, and tactile stimulations.

Figures