Adaptive Captioning with Emotional Cues: Supporting DHH and Neurodivergent Learners in STEM

TL;DR

This study explores four innovative captioning prototypes that incorporate emotional and multimodal cues to enhance comprehension and reduce cognitive load for DHH and neurodivergent STEM learners.

Contribution

It introduces new captioning designs embedding emotional cues and evaluates their effectiveness through user studies with neurodivergent and DHH learners.

Findings

Certain prototypes reduced cognitive load

Improved comprehension scores observed

Customizable features favored by neurodivergent users

Abstract

Real-time captioning is vital for Deaf and Hard of Hearing (DHH) and neurodivergent learners (e.g., those with ADHD), yet it often omits emotional and non-verbal cues essential for comprehension. This omission is particularly consequential in STEM education, where cognitively demanding material can exacerbate the challenges faced by caption users across diverse ability profiles. In this paper, we present a design-oriented exploration of four captioning prototypes that embed emotional and multimodal cues, including facial expressions, body gestures, keyword highlighting, and emoji. Across a pilot and a main study with 24 participants, we found that certain prototypes reduced self-reported cognitive load and improved comprehension scores compared to traditional captions. Qualitative feedback reveals the importance of customizable caption features to accommodate neurodivergent users' preferences (e.g., ADHD or different levels of comfort with emojis). Our findings contribute to ongoing conversations in accessible technology research about how best to integrate emotional cues into captions in a way that is both usable and beneficial for a wide range of learners.