Inclusive Emotion Technologies: Addressing the Needs of d/Deaf and Hard of Hearing Learners in Video-Based Learning

TL;DR

This study investigates how d/Deaf and hard of hearing learners perceive and utilize emotion recognition technologies in video learning, revealing unique needs and concerns that inform more inclusive design.

Contribution

It provides new insights into DHH learners' interactions with emotion data and suggests tailored approaches for inclusive emotion technology design.

Findings

DHH learners rewatch videos to self-report emotions, seeking alternative methods.

Concerns about AER accuracy and intrusiveness among DHH learners.

Differences in emotion technology usage between DHH and hearing learners.

Abstract

Accessibility efforts for d/Deaf and hard of hearing (DHH) learners in video-based learning have mainly focused on captions and interpreters, with limited attention to learners' emotional awareness--an important yet challenging skill for effective learning. Current emotion technologies are designed to support learners' emotional awareness and social needs; however, little is known about whether and how DHH learners could benefit from these technologies. Our study explores how DHH learners perceive and use emotion data from two collection approaches, self-reported and automatic emotion recognition (AER), in video-based learning. By comparing the use of these technologies between DHH (N=20) and hearing learners (N=20), we identified key differences in their usage and perceptions: 1) DHH learners enhanced their emotional awareness by rewatching the video to self-report their emotions and called for alternative methods for self-reporting emotion, such as using sign language or expressive emoji designs; and 2) while the AER technology could be useful for detecting emotional patterns in learning experiences, DHH learners expressed more concerns about the accuracy and intrusiveness of the AER data. Our findings provide novel design implications for improving the inclusiveness of emotion technologies to support DHH learners, such as leveraging DHH peer learners' emotions to elicit reflections.