# Comparison of cognitive workload between very short answer questions and multiple-choice questions: an eye-tracking experiment

## TL;DR

This study used eye tracking to find that very short answer questions require more mental effort than multiple-choice questions.

## Contribution

The study objectively compares cognitive workload using eye-tracking data between two question types for the first time.

## Key findings

- VSAQs caused more fixations and revisitations than MCQs, indicating higher cognitive workload.
- Incorrect answers were linked to increased workload, even after controlling for accuracy.
- Heatmaps showed VSAQs led to denser fixations on key features, while MCQs focused on answer options.

## Abstract

Very short answer questions (VSAQs) have gained attention for their superior psychometric properties compared to multiple-choice questions (MCQs). While VSAQs require knowledge recall, MCQs primarily involve knowledge recognition. This difference in cognitive processes may lead to varying cognitive workloads, defined as the amount of mental processing in working memory. Previous studies have not demonstrated consistent differences, likely due to reliance on self-reported measures. Eye tracking provides objective, process-level indicators of cognitive workload. This study investigated whether answering VSAQs requires a higher cognitive workload than answering MCQs. In a within-subject randomized crossover experiment, sixth-year medical students answered both VSAQs and MCQs. Cognitive workload was measured using screen-based eye tracking, focusing on the number of fixations and revisitations as objective indicators of mental effort. Data were analyzed using mixed-effects models. Thirty-four medical students participated, yielding 1,326 observations, which is the multiplication of the number of students by the number of questions (39 questions). Mixed-effects models showed a significant effect of question type on both workload indicators: VSAQs elicited more fixations and revisitations than MCQs (β_std = 0.30–0.39, p < .001). This effect remained after controlling for accuracy. Incorrect answers were associated with higher workload (β_std = −0.15–−0.16, p < .01). Heatmaps confirmed these findings, showing denser fixations on key diagnostic features for VSAQs and on answer options for MCQs. Answering VSAQs imposed a higher cognitive workload than MCQs. The presence of answer options in MCQs may reduce workload by providing unintentional cues, while VSAQs require active retrieval. Eye tracking proved valuable for distinguishing cognitive workload across assessment formats.