RTTD-ID: Tracked Captions with Multiple Speakers for Deaf Students

TL;DR

This paper introduces RTTD-ID, a real-time captioning system with speaker identification designed to improve classroom access and inclusion for deaf students by showing who is speaking during group activities.

Contribution

It presents a novel RTTD-ID system with three display schemes and evaluates their effectiveness, demonstrating improved user preference for the pointer method.

Findings

Users preferred the pointer method for speaker identification.

RTTD-ID enhances deaf students' ability to identify speakers in group settings.

The system reduces frustration and improves inclusion for deaf students.

Abstract

Students who are deaf and hard of hearing cannot hear in class and do not have full access to spoken information. They can use accommodations such as captions that display speech as text. However, compared with their hearing peers, the caption accommodations do not provide equal access, because they are focused on reading captions on their tablet and cannot see who is talking. This viewing isolation contributes to student frustration and risk of doing poorly or withdrawing from introductory engineering courses with lab components. It also contributes to their lack of inclusion and sense of belonging. We report on the evaluation of a Real-Time Text Display with Speaker-Identification, which displays the location of a speaker in a group (RTTD-ID). RTTD-ID aims to reduce frustration in identifying and following an active speaker when there are multiple speakers, e.g., in a lab. It has three different display schemes to identify the location of the active speaker, which helps deaf students in viewing both the speaker's words and the speaker's expression and actions. We evaluated three RTTD speaker identification methods: 1) traditional: captions stay in one place and viewers search for the speaker, 2) pointer: captions stay in one place, and a pointer to the speaker is displayed, and 3) pop-up: captions "pop-up" next to the speaker. We gathered both quantitative and qualitative information through evaluations with deaf and hard of hearing users. The users preferred the pointer identification method over the traditional and pop-up methods.