A scoping review of cognitive load assessment tools suitable for clinicians performing REBOA

TL;DR

This study reviews tools to measure cognitive load in clinicians performing REBOA and suggests the NASA-TLX and HRV as suitable options.

Contribution

The study proposes a REBOA-adapted version of the NASA-TLX for post-procedure cognitive load assessment.

Findings

The NASA-TLX was the most frequently used and highest-scoring subjective cognitive load tool for REBOA.

HRV was the most common objective measure and scored well for REBOA use.

A modified NASA-TLX tailored for REBOA is suggested for post-procedure assessment.

Abstract

The ability to measure a clinician’s cognitive load allows task adaptions to optimise performance. The aim of this study was to identify cognitive load tools suitable for use by pre-hospital clinicians performing REBOA and develop a bespoke pre-hospital REBOA cognitive load assessment tool.

A scoping review was conducted, following the PRISMA guidelines, to identify different cognitive load assessment tools in the literature from inception to January 2023. A qualitative narrative synthesis was used to compare tools based on their type, frequency of use, and context. Tools were assessed for contextual relevance and practical application to REBOA using defined criteria (CMTA-R score), created using domain experts.

Forty-nine articles were included for review, identifying 21 unique cognitive load tools: 10 subjective and 11 objective. The NASA-TLX was the most frequently used subjective tool (17 studies), scoring highest for potential REBOA use (CMTA-R 17). Heart Rate Variability (HRV) was the most common objective measure (14 studies), with a CMTA-R score of 13. A bespoke REBOA modification of the NASA-TLX, was suggested to quantify post-procedure cognitive load.

This scoping review identifies the NASA-TLX and HRV as potential tools for assessing cognitive load during prehospital REBOA. A bespoke REBOA-adapted NASA-TLX, could be used post-procedure, while intra-procedural HRV monitoring could provide real-time data. Future research should validate this approach in clinical settings.

The online version contains supplementary material available at 10.1186/s13049-025-01408-0.

Figures

PRISMA flow diagram

Full text

Introduction

The human brain is continually exposed to a myriad of sensory stimuli, processed for immediate action or conveyed into long-term memory. While long-term memory may be infinite, working memory is generally considered to have limited capacity [1, 2]. Cognitive load, sometimes referred to as bandwidth, describes the mental strain and effort required as working memory accumulates during a task. Cognitive overload may occur when the demands of a task outweigh an individual’s cognitive capacity [3]. Cognitive overload increases the risk of psychophysiological stress and medical error [4, 5]. Cognitive Load Theory divides cognitive capacity into three aspects: intrinsic load, related to the demands of the task itself; extrinsic load, related to the demands of the task environment; and germane load, the capacity required to construct long-term memory representations [6].

Resuscitative Endovascular Balloon Occlusion of the Aorta (REBOA) is used as a temporising measure to prevent exsanguination from non-compressible haemorrhage. For pre-hospital clinicians performing REBOA, the complex nature of the task, the challenging working environments and the high stakes decision making with limited information, generates significant cognitive load [7, 8]. Current REBOA research has predominantly focused on clinical application and survival rates. However, developing a deeper understanding of the cognitive demands of deploying REBOA could highlight areas that are mentally challenging for clinicians. Subsequently approaches to mitigate such challenges could then be developed.

Diverse strategies have been designed across many industries to quantify the amount of cognitive load a specific task requires. In medicine, measuring a clinician’s cognitive load enables tasks to be adapted to reduce intrinsic load, enhance environments to reduce extrinsic load and adapt training programmes to optimise germane load [3, 9]. To quantify a REBOA operator’s cognitive load, an appropriate tool must withstand both the variable contexts for REBOA and the dynamic course of a patient undergoing REBOA. This includes the extrinsic input from the varying pre-hospital environments alongside the intrinsic input of the procedure for differing patients with unique clinical pictures. This study aimed to conduct a scoping review of existing cognitive load tools to determine which tools would be suitable for pre-hospital clinicians performing REBOA. The findings will guide development of a pre-hospital REBOA cognitive load assessment tool.

Methods

A scoping review of the published literature was conducted to identify cognitive load assessment strategies used in multistep medical procedures. Subjective and objective cognitive load assessment tools were sought. The study followed the PRISMA guidelines [10, 11].

Search strategy

A literature search was undertaken for articles describing methods used to measure cognitive load of clinicians performing medical procedures. The following search terms were used: “methods”, “measuring”, “clinicians”, “cognitive load”, “haemorrhage control”, including variants for each field. PubMed, Embase, Scopus, Web of Science, Cochrane Library and IEEExplore databases were searched from inception to January 2023. The full search strategy can be viewed in Additional File 1 [see Additional File 1]. Peer-reviewed articles published in English were reviewed that described one or more cognitive load measurement tools used in simulated or real-world multistep procedures.

A procedure was defined as a multistep intervention requiring simultaneous decision-making and were performed prehospitally, in the Emergency Department, or in operating theatres. Systematic reviews and studies involving single-step or isolated decision-making tasks were excluded. Identified articles were screened for relevance by title then abstract. Those that remained underwent full manuscript review. Post review by a second reviewer, the literature that remained was read in-full for a second time. Data was extracted that included the study setting, the cognitive load measuring tool used, and the procedure performed.

Data analysis

A qualitative narrative synthesis was conducted to compare the different tools identified to measure cognitive load. Data extracted from the included studies incorporated the type of tool, frequency and context in which it was used. To assess the methodological quality of the identified studies, the Medical Education Research Study Quality Instrument (MERSQI) was used [12]. This is a validated six-domain assessment tool covering 10 items, generating a score between three and 18, with higher scores denoting more rigorous research methods.

Cognitive load measuring tool assessment for REBOA (CMTA-R)

The authors developed bespoke criteria to assess the contextual relevance and practical application of each tool. Operational factors impacting REBOA deployment were considered based on literature and the experiences of pre-hospital clinicians at London’s Air Ambulance [7, 13]. The Cognitive Load Measuring Tool Assessment for REBOA (CMTA-R) was used to score the applicability of subjective and objective cognitive load tools. The CMTA-R criteria can be found in Table 1. Assessment of subjective tools used criteria 1–14, whilst criteria 1,2,15–20 were applied to objective tools.

Table 1. Cognitive Load Measuring Tool Assessment for REBOA (CMTA-R) criteria for the assessment of subjective and objective tools Criteria

Score

Explanation

General Criteria 1 Simulated/Real-world Simulated1Has the tool shown application in simulation and/or real-world scenarios?Real-world22 Environment Theatre1What level of controlled environment was the tool used in?Emergency Department2Prehospital3 Subjective Tool-specific Criteria 3 Domains One0How many different cognitive load contributors were captured by the tool?Two1More than two24 Rating scale Broad1Were rating scales broad (fewer increments) or provided more accurate representation (more increments)?Focused25 Teamworking Yes1Did at least one tool domain encompass: teamworking?No06 Decision making Yes1Did at least one tool domain encompass: decision making?No07 Physical Exertion Yes1Did at least one tool domain encompass: physical exertion?No08 Multitasking Yes1Did at least one tool domain encompass: multitasking?No09 Situational awareness Yes1Did at least one tool domain encompass: situational awareness?No010 Frustration Yes1Did at least one tool domain encompass: clinician’s frustration?No011 Distraction Yes1Did at least one tool domain encompass: distractions from the task?No012 Time pressure Yes1Did at least one tool domain encompass: time pressure of the task?No013 Speed Yes1Did at least one tool domain encompass: the speed the task was completed?No014 Procedural result Yes1Did at least one tool domain encompass: the final result of the procedure?No0 Objective Tool-specific Criteria 15 Wireless capability Yes1Does the tool demonstrate wireless capability?No016 Restriction of equipment Non-restrictive1Is the equipment required for monitoring restrictive to the normal delivery of the task?Restrictive017 Distraction from primary task Not distracting1Does the method of cognitive load measure provide added distraction from the task?Distracting018 Learning curve Not required1Does the method of cognitive load measure require prior training before use?Required019 Exclusive measurement Yes1Are measurements exclusive to the participant and not affected by environmental factors?No020 Technological availability Commercial2Is the technology for measurement commercially available or bespoke for research purposes?Bespoke1

Results

The search identified 592 articles; an additional seven articles were identified from reference lists. From these, 362 were considered for screening, of which 49 met the inclusion criteria after full-text review. (Fig. 1) There were 21 different cognitive load assessment tools identified.Fig. 1PRISMA flow diagram

Study design

Of the 49 included articles, 32 (65%) described a single cognitive load measurement tool while 17 (35%) used two or more. Subjective measures were used in 30 (61%) articles, objective measures in 29 (59%), and both subjective and objective approaches were used in 10 (20%) articles [14–23]. Most articles, 27 (55%), measured cognitive load in real world scenarios. Surgical procedures were the most common focus (40 articles), with nine studies utilising non-surgical scenarios. No articles included procedures performed pre-hospital.

Subjective cognitive load tools

The review identified 10 distinct subjective cognitive load measuring tools. (Table 2) The National Aeronautics and Space Administration Task Load Index (NASA-TLX) was the most used subjective tool (17 articles) with an approximately even distribution between simulated (8/17) and real-world scenarios (9/17). Table 2. Cognitive load measuring tools captured from the included literature. A description of each tool is included, along with the number of studies and study environments they were applied in and their CMTA-R score Type of Measure

Specific Tool

Description of Tool

Number of studies

CMTA-R

Overall

Simulation

Real-world SubjectiveNASA-TLX6 domains scored from 0–100 and weighted 0–5 including: mental demand, physical demand, temporal demand, performance, effort, frustration178417SURG-TLX6 domains scored from 0–100 and weighted 0–5 including: mental demand, physical demand, temporal demand, task complexity, situational stress, distractions81715Likert Scales5–7-point single domain scale ranging from “not stressful” to “extremely stressful”3125SMEQSingle domain scored from 0–150 with 9 markers ranging from “not hard” to “extremely hard to do”3127HFEQ-CASS38 questions scored on a 5-point scale:Block 1 = 12 questions: mental workload, surgical results, situational awareness, speed, risk taking.Block 2 = 26 questions: surgical and ergonomic characteristic10112MRQ17 questions on 4-point scales assessing processing demand of: auditory, facial, manual, memory, spatial, tactile, visual and vocal10112SWAT3 domains scored from ‘low, moderate and high’, measuring time load, mental effort and emotional stress10111Leppink Questionnaire10 item questionnaire covering 3 domains of cognitive load: intrinsic load, extrinsic load, germane load1109STAI40 questions on 4-point scales across 2 categories: ‘state’ and ‘trait’, focus on mental state and personal outlook1019OTAS6-point scale across 5 domains: communication, coordination, cooperation, leadership, monitoring1018ObjectivePhysiological ParametersHRVStatistical analysis of beat-to-beat variation on ECG1441013Eye-trackingWearable monitor measuring pupillary dilation, tracking gaze duration and blink rate53211EEGElectrodes on the scalp measure the voltage of electrical brain activity under different conditions determining functional area activation3309GSRSensory placed on the fingertips measuring skin conductance reflecting sweat production3308fNIRNear-infrared spectroscopy for functional neuroimaging by estimating the haemodynamic activity of the brain1017EMGElectrodes positioned on the hands or arms delivering a small impulse to measure nerve conduction1106Oxygen SaturationsMeasure of peripheral SpO2% using a pulse oximeter1016Heat FluxThermal imaging of heat distribution within the face1105IOPUse of rebound tonometry as a measure of IOP in both eyes pre- and post-procedure1105Procedural ParametersError RateAnalysis of each procedural step, recording correct execution and number of errors1109Secondary Task AnalysisReaction TimeSecondary stimulus (auditory/tactile) delivered at varying intervals during the primary procedure, reaction times measured to identify and terminate the stimulus6606

First published in 1988, the NASA-TLX is often presented as a post-hoc questionnaire providing a weighted score across six domains, three measuring the burden of the task itself: Mental demand, Physical demand, Temporal demand; and three measuring the task environment: Performance, Effort and Frustration [24]. The NASA-TLX scored the highest for potential transferability to prehospital REBOA (CMTA-R = 17) (Table 2). SURG-TLX, a surgery-specific variant of the NASA-TLX, was the second highest scoring tool for both frequency of use and CMTA-R (CMTA-R = 15). SURG-TLX was used in eight studies, one (13%) simulated and seven (88%) real-world. The SURG-TLX differs to the NASA-TLX by substituting Performance, Effort and Frustration domains for Task Complexity (“How complex was the procedure?”), Situational Stress (“How anxious did you feel whilst performing the procedure?”), and Distractions (“How distracting was the operating environment?”) [25].

Other than the NASA- and SURG-TLXs, only two other types of subjective measurement were cited in more than one article: the Subjective Mental Effort Questionnaire (SMEQ) and several tools employing adapted Likert Scales for specific uses [14, 15, 26–29]. Both SMEQ and Likert scales utilise a single domain approach.

Objective tools

The review identified 11 objective measuring tools cited in 23 (62%) simulated and 14 (38%) real-world scenarios (Table 2). The most common objective tool was Heart Rate Variability (HRV) (14 articles). HRV assesses the changes in duration between adjacent heartbeats through electrocardiogram (ECG) monitoring. Divided into time and frequency domains, it provides a physiological representation of cognitive stress reflecting the neurocardiac axis and various cardiac inputs [30]. Monitoring may be achieved through 3-lead ECGs, commercial heart rate monitors and smart watches. [31, 32]. HRV was the highest scoring objective tool, showing the greatest potential for use with REBOA operators (CMTA-R = 13) (Table 2).

Another widely used physiological objective tool used eye-tracking (CMTA-R = 11). Used in five studies, eye-tracking represents autonomic responses through measure of pupil size, gaze duration and blink frequency [15, 16, 33]. Monitoring included both wearable and static devices.

Objective measurements of cognitive load can also be achieved through procedural and situational parameters. Alternative objective measures were used across seven articles. Secondary Task Analysis was the most frequently used tool reflecting this approach. In six studies, all under simulated conditions, Secondary Task Analysis tested the reaction speeds of a clinician in termination of an additional stimulus whilst undertaking a procedure [21, 34–39]. This aims to test task-focus. Secondary Task Analysis scored lower on the CMTA-R (CMTA-R = 6).

Participants and study designs

In the included studies the number of participants varied from one to over 100. Surgeons were the most studied (39 studies), followed by anaesthetists (7 studies) and emergency medicine physicians (6 studies). A combined total of 13 randomised controlled trials were recorded. Real-world settings were used in six studies [25, 27, 29, 32, 40, 41], with seven randomised controlled trials occurring in simulation [37, 39, 42–46].

The MERSQI score used for bias assessment varied from 15 to nine, with an average score of 13. Overall, the highest scoring studies included randomised control trials (13 studies) and studies recording real-world patient outcomes (17 studies). MERSQI scores for each article can be found in the summary table of included literature [see Additional File 2].

Discussion

The review has identified a multitude of subjective and objective cognitive load measuring tools with potential application to REBOA operators. Analysis of the benefits and limitations of each tool highlighted the NASA-TLX and HRV as the most suitable subjective and objective tools to assess cognitive load of REBOA operators. Using both tools in conjunction would likely support the idea that intra-procedure monitoring could validate post-procedure questionnaire results.

Pre-hospital REBOA is a time-critical, high-acuity, low-frequency procedure which by occluding the aorta increases cerebral and coronary perfusion. These effects are desirable in both traumatic exsanguination and non-traumatic cardiac arrest. The use of REBOA as a cardiopulmonary resuscitation (CPR) adjunct has been shown to increase rates of Return of Spontaneous Circulation (ROSC) [47, 48]. Meanwhile, when deployed for exsanguinating patients from sub-diaphragmatic haemorrhage, REBOA aids patient survival during transfer to definitive haemostasis [49, 50].

Achieving aortic occlusion with REBOA is a complex process with risk. Significant challenges include patient selection and the fundamental step of achieving femoral arterial access. Obtaining arterial access in the pre-hospital environment on patients either close to exsanguination or with ongoing chest compressions is technically extremely difficult. Potential challenges include visualising the artery under ultrasound and successful guide wire placement. The REBOA operator will be aware that failed attempts increase on-scene time and decrease the probability of patient survival [51].

Another technique that introduces complexity is the use of partial-REBOA (pREBOA). In trauma, pREBOA has the potential to increase total duration of safe aortic occlusion, reduce inflammatory complications and improve patient outcomes [49]. However, the technical application of partial aortic occlusion is challenging as the therapeutic window is narrow with current REBOA technology. The key issue relates to the amount of volume that can be removed from the occluding aortic balloon before an exponential increase in aortic flow distal to the balloon. The meticulously planned process of achieving pREBOA is complicated further by the phenomenon of physiological pREBOA. The dynamic nature of the aorta, affected by catecholamines and fluid status, causes vasoconstriction and dilation leading to partial occlusion regardless of the clinician’s intentions [52, 53]. Such unwanted increases in distal flow can cause proximal collapse of blood pressure, cardiac instability and further bleeding [53].

Technological advancements present opportunities to streamline the aspects of REBOA clinicians find challenging. AI-guided ultrasound and preloaded guide wires offer a potential to improve arterial access [51]. Meanwhile, automated pREBOA systems, like Endovascular Variable Aortic Control (EVAC), can independently adjust aortic balloon inflation based on real-time pressure changes, potentially improving patient outcomes [54, 55].

Such technological advancements could not only enhance procedural efficacy but also reduce the cognitive load on clinicians. By automating technically demanding tasks, pre-hospital clinicians can allocate more cognitive bandwidth to other critical aspects of patient care. While current research into REBOA has necessarily focused on feasibility, processes and survival rates, a deeper understanding of the cognitive demands of REBOA is crucial [52]. By quantifying a clinician’s cognitive load, we can assess the value of technological improvements and optimise skill-uptake during training. Regular simulation training has been shown to reduce cognitive burden, widening the clinician’s capacity for learning [56, 57]. Therefore, measuring cognitive load can guide both reductions in task complexity and the delivery of frequent high-quality training, ultimately improving practice and clinical decision-making in high-stress pre-hospital environments.

This study demonstrates a combination of both subjective and objective cognitive load measuring tools appears best suited to REBOA operators. Over 20% of the included articles used both types of measure, half of which demonstrated real-world application. The synergy between a real-time physiological or procedural marker and a retrospective post-hoc questionnaire, allows for correlations to be drawn between procedural steps and peaks in mental strain. This would be beneficial in the context of REBOA. By identifying the steps incurring the highest cognitive burden, these aspects may be the focus for procedural adaptations and further training.

The cognitive load assessment tools captured in the review were assessed using bespoke criteria for contextual relevance to pre-hospital REBOA (CMTA-R). The CMTA-R criteria was developed by the authors, using domain expertise and real-world experience of pre-hospital physicians deploying REBOA. The authors focused on a score that included operational factors impacting the delivery of the procedure. This approach provided a simple method to compare cognitive load tools using the same benchmarking approach where no such validated tool existed. However, it should be noted that the factors used to rate the cognitive load tools were all derived from pre-hospital clinicians working in one institution. As such, some factors may have been neglected and others over-emphasised.

The NASA-TLX, a versatile tool for assessing cognitive load, is well-suited for REBOA operators. Its six-domain approach, applicable to both real-world and simulated settings, provides a comprehensive understanding of cognitive burden. The weighted scales and adaptability of the NASA-TLX make it ideal for capturing the unique demands of REBOA in diverse operational environments. REBOA performed in different contexts may present differing levels of complexity. For example, using REBOA for trauma in cities has different challenges and mental demands than using it for cardiac arrest in rural areas. Factors such as the duration of balloon inflation, geographical distances travelled, and hospital selection are all affected by context. A useful cognitive load tool will capture and quantify this complexity in its ability to measure cognitive load. For example, by using the NASA-TLX, these differing complexities in decision making would be captured in the Mental Demand domain.

A bespoke REBOA cognitive load assessment questionnaire, building on the NASA-TLX framework, could further refine the tool for pre-hospital settings, focusing on unique task-based and environmental factors. An assessment tool of this sort could be widely accessed and implemented across pre-hospital services currently deploying REBOA. Use of a post-procedure cognitive load questionnaire, such as the NASA-TLX or a bespoke REBOA variant, could generate data helping services guide changes to improve performance for future REBOA patients [58].

This study highlights the adaptability of the NASA-TLX for assessing cognitive load across diverse clinical contexts. By tailoring intrinsic and extrinsic factors to specific procedures (e.g. task complexity in Resuscitative Thoracotomy or image interpretation in Point-of-Care Ultrasound), clinicians gain valuable insights into their cognitive load during challenging interventions like Extra Corporeal Membrane Oxygenation (ECMO) cannulation or pre-hospital REBOA. This enhanced awareness can inform strategies to mitigate cognitive overload and optimise patient care.

HRV, a validated physiological marker of cognitive load, is a promising objective tool for assessing REBOA operators. Previous studies of performance have shown heart rates above 140-145bpm can be associated with inhibition of working memory [8, 59]. Its non-invasive nature and compatibility with wearable devices make it suitable for pre-hospital settings. Unlike other methods, such as Functional Near-Infrared Spectroscopy (fNIR), Electroencephalography (EEG), or Electromyography (EMG), HRV does not require bulky equipment or restrictive setups, allowing for continuous monitoring during procedures.

HRV can be correlated with subjective tools like the NASA-TLX to provide a comprehensive understanding of cognitive burden [23, 60]. A recent case report demonstrated the feasibility of using a commercial smartwatch, to measure HRV, and the NASA-TLX to assess cognitive load in pre-hospital clinicians performing a Resuscitative Thoracotomy [61]. This case report highlights the potential for combining objective and subjective measures in real-world pre-hospital settings.

The present study proposes a novel cognitive load assessment tool for REBOA operators, combining a REBOA-specific NASA-TLX questionnaire and intra-procedural HRV monitoring. This approach offers the first attempt to quantify clinicians’ cognitive load for a pre-hospital procedure. A feasibility study would be required to validate this approach.

Limitations

Cognitive load measuring tools have been utilised throughout many high-stakes professions. The present study only reviewed clinical usage. Tools from other occupational domains may also be suitable. Their assessment would require further research. Overlap in the use of key search terms, such as “mental stress”, “mental effort”, “cognitive load”, and “bandwidth”, may also lead to confounding of different concepts for which these terms are also used. The present study focuses on metrics associated with cognitive load compared to other cognitive states such as psychological trauma and emotional stress. Additionally, the captured literature did not represent pre-hospital REBOA directly. The use of CMTA-R provided a standardised comparative tool to prevent bias when assessing measures in context. However, this approach itself may be limited as factors used to rate the captured tools were derived from pre-hospital clinicians working in one institution.

Conclusion

This scoping review identifies the NASA-TLX and HRV as potential tools for assessing cognitive load during prehospital REBOA. This dual assessment approach could be achieved through development of a bespoke REBOA-adapted NASA-TLX, used post-procedure, with intra-procedural HRV monitoring providing real-time data. Future research should validate this approach in clinical settings. This study provides the first step in obtaining cognitive load measures for prehospital REBOA operator.

Supplementary Information

Supplementary Material 1

Supplementary Material 2: [62–78]