Multi-Modal Measurements of Mental Load

TL;DR

This paper explores how various physiological signals like pupil size, blinking, and heart metrics relate to mental load, aiming to develop real-time estimation methods through an experiment involving language comprehension tasks.

Contribution

It introduces an experimental approach to analyze multiple physiological measures for assessing mental load during cognitive tasks.

Findings

Physiological measures correlate with task difficulty

Response times vary with mental load levels

Preliminary analysis shows potential for real-time mental load estimation

Abstract

This position paper describes an experiment conducted to understand the relationships between different physiological measures including pupil Diameter, Blinking Rate, Heart Rate, and Heart Rate Variability in order to develop an estimation of users' mental load in real-time (see Sidebar 1). Our experiment involved performing a task to spot a correct or an incorrect word or sentence with different difficulties in order to induce mental load. We briefly present the analysis of task performance and response time for the items of the experiment task.

Full text

Cognitive load, Physiological Sensor, Human-Computer Interaction, Human-Robot Interaction

idth=.5]flow.png Experimental Protocol

1. Introduction

There is an ongoing interest in creating robots that can adapt their behaviour according to users traits in the field of Human-Robot/Computer Interaction (HCI/HRI) (Ahmad et al., 2017). User’s Cognitive Load or Operator’s Mental Load are among important attributes during the HCI. It has been found that a higher level of cognitive or mental load of the users may have an adverse effect on their task performance during HCI scenario (Reeves et al., 2004).

The amount of effort placed on a user’s working memory during a task is referred to as the cognitive load. According to the Cognitive Load Theory (Sweller, 1994), there are three different types of load produced during problem-solving or learning: 1) intrinsic load, 2) the extraneous load and 3) the germane load.

1. (1)

Intrinsic load depends on the complexity of the structure of the material and its association with the learner. 2. (2)

The extraneous load is caused by the way this material is presented to the learner. 3. (3)

The germane load is a result of the learner’s ability to assimilate the material.

On the other hand, cognitive load is also defined as a construct that can be measured in three dimensions: 1) mental load, 2) mental effort and 3) performance (Sweller et al., 1998). In other words, past research indicates that cognitive load and mental load are related to each other in terms of the working memory. Therefore, we will be using them interchangeably. It is also interesting to note that the amount of workload varies between individuals (Baddeley, 1992). This suggests that the amount of working memory is different for each individual. Thus, it highlights the significance of measuring the amount of the mental load in real-time in order to adapt the Interface during the HRI or HCI. Consequently, we are also involved in creating the technology that can adapt to the user’s mental load in real-time. More specifically, we want to personalise the behaviour of the robot or adjust the information on the user interface to adjust humans mental overload.

It is currently an open challenge to measure the individual’s mental load in a non-intrusive and robust manner in real-time (Paas et al., 2003; Chen et al., 2012). Existing literature shows that cognitive load can be measured in three different ways: through subjective rating questionnaires (NASA Task Load Index111 NASA Task Load Index - https://humansystems.arc.nasa.gov/groups/TLX/ downloads/TLXScale.pdf), through physiological sensors, or through performance-based objective measures (Mathematical Equations). However, subjective rating and performance-based objective measures are limited and cannot be utilised in real-time during HCI or HRI to adjust the interface or the behaviour of the robot. On the contrary, existing literature has highlighted different physiological measures that are continuous and can be used to estimate mental load in real-time (Paas et al., 2003). These measures include Pupil Diameter (PD), Blinking Rate (BR), Heart Rate (HR), Heart Rate Variability (HRV), Electroencephalography (EEG) and Galvanic Skin Response (GSR). Each of these methods has been shown to measure mental load during various tasks; however, each of them has its limitations. For instance: Eye activity (PD, BR) may not be suitable for tasks requiring continuous reading. Similarly, the sensitivity of pupil changes in cognitive load diminishes with age (Van Gerven et al., 2004). Additionally, the HRV may depend on the physical fitness of the individuals (Luque-Casado et al., 2013). It is, therefore, significant to understand the relation between these physiological behaviours under different tasks or circumstances with each other and also onto each other. As our long-term goal is to estimate the user’s mental load in real-time to personalise interfaces or robot’s behaviour, therefore, we believe that understanding the relationship between different physiological behaviour is vital and will help us in achieving our long-term goal of creating a model that takes various physiological behaviours as input to estimate mental overload or cognitive load in an efficient manner.

To achieve our long-term goal, we are currently collecting data from various physiological sensors to understand the relationship between aforementioned physiological behaviours under different task during HCI and HRI respectively (Yaseen and Lohan, 2018; Sabyruly et al., 2015a; Minadakis and Lohan, 2018). Building on the prior work, this position paper describes another series of one of our experiments conducted to understand the relationship between different physiological behaviours during a game-based task. We intend to collect data based on various physiological behaviours (HR, HRV, BR, and PD) during different tasks and later want to create datasets based on these behaviours. Our long-term goal is to use these datasets for a linear mixed-effects regression model to estimate mental or cognitive load in real-time.

2. Related Work

In the past, empirical findings have highlighted a relationship between mental load or cognitive load and various physiological behaviour (PD, HRV, HR, BR, GSR and EEG). It has been shown that the HRV reduces with an increase in the amount of cognitive load during a range of computerised tasks (Mukherjee et al., 2011). Similarly, another study observed an increase in HR with an increase in the difficulty level of the task or in other words in a situation demanding higher cognitive load (Cranford et al., 2014). In addition, we find a body of literature that suggests that pupil diameter increases with an increase in the number of mental load (Reilly et al., 2018; Sabyruly et al., 2015b). Moreover, research indicates that the average number of blinks per minute in normal healthy adults varies according to the task. A study conducted with 150 adults showed that individuals on average blink four to five times per minute during a reading activity. Similarly, the rate of blinking per minute is higher during resting (17) and during conversation (26) (Bentivoglio et al., 1997). Later, it has been shown that the blinking rate is minimised in the case of higher mental overload (Holland and Tarlow, 1972).

In summary, we find enough empirical evidence from past research implying that the change in the physiological behaviour can be attributed to cognitive load or mental load and it also refers to various levels of mental processing. It is also important to note that the data collected on physiological behaviours through the various state of the art sensors is not only continuous but is also a robust and accurate representation of the particular behaviour (HRV, 2018; Tommaso, 2018). It has been highlighted that users are required to wear cumbersome equipment; however, with the advancement of design and technology, existing solutions have been devised to collect such data in non-invasive ways.

In relation to our long-term goal, we currently find empirical evidence that few of the physiological behaviours are correlated with each other. For instance, both pupil dilation and blink-eye rate were seen to be correlating with each other in a digit-sorting task (Siegle et al., 2008). However, to the best of our knowledge, the relationship between all these behaviours have not been explored and is necessary to efficiently create an estimation for cognitive load or mental load. As highlighted we need to understand the relationships between different behaviours because this can help in creating a linear mixed-effect regression model that may estimate mental load in real-time during HCI or HRI. Consequently, We, in this paper, presents intial finding of a study conducted in order to collect data on these behaviours in a synchronous way to help us analyze the relationships between different physiological behaviours and later create a dataset that can be used to estimate cognitive load in real-time.

3. Study

The purpose of the study was to understand the relationship between different physiological measures (HR, HRV, PD and BR) to use these variables to help estimate mental load in real-time.

3.1. Experiment Protocol

The experiment was conducted with 41 participants (demographics as shown in table 1) in seven different steps:

• •

Participant fills a Consent Form.

• •

Participant fills a General Questionnaire to report information on age, numbers of languages they speak and in case they have reading difficulties and a Physical Activity Questionnaire (Fogelholm et al., 2006) to control for any bias in the HR and HRV measurements.

• •

Participant wears Tobii Eye Tracker and performs a simple Calibration to get the values for their Pupil Diameter.

• •

Participant wears corSense Heart-Rate Device and performs the task.

• •

Participant plays spot the real or made-up word and sentence game.

• •

Mental load is measured from Tobii Glasses 2 (BR, PD), Heart Rate Monitor (HR, HRV), and Webcam facing the participant during the game task.

• •

Participant fills the NASA TLX Questionnaire.

idth=.5]setup.jpg Setup: Participant is performing the game based task

3.2. Task

The task given was a correct or an incorrect English word or sentence recognition task with different difficulties. The participants were presented with items from the following categories. All word items were 10 letters long. All sentence items consisted of 10 words each.

[TABLE]

Table 1: Task Item Overview

3.3. Initial Results

The following table shows the first analysis of task performance and response time for the items of the experiment task.

[TABLE]

Table 2: Initial results with participant’s demographics

We are currently creating the dataset that will be comprised of eye and front camera video and the pupil dilation data from the eye tracker, the data from the heart rate monitor, and the video from a webcam facing the participants. Due to in-completion of the dataset, we are not reporting the results in this paper but give an overview of some of its properties.

3.4. Conclusion and Future Work

In the paper, we presented the initial results of our study conducted to understand the relationship between various physiological behaviours (HR, HRV, BR, PD) during an HCI task. We are currently analysing the data on all these behaviours, and our long-term goal lies in the development of a system that could potentially provide a robust non-intrusive measurement of mental workload in real-time during HCI. Our work is focused on creating a dataset of these physiological behaviours and later using them for a regression-based model to estimate mental load during various HCI and HRI scenarios.

Acknowledgements.

We acknowledge funding and support from the EPSRC ORCA Hub (EP/R026173/1, 2017-2021) and consortium partners.