Deaf and Hard of Hearing Access to Intelligent Personal Assistants: Comparison of Voice-Based Options with an LLM-Powered Touch Interface

TL;DR

This study compares voice-based and touch interface methods for IPAs to improve accessibility for deaf and hard of hearing users, highlighting the need for better native recognition of deaf accents.

Contribution

It introduces an LLM-powered touch interface as an alternative to voice commands for DHH users and evaluates its usability against traditional voice methods.

Findings

No significant difference in usability between spoken English and LLM-assisted touch.

User opinions on usability varied across methods.

Highlighting the need for native deaf-accented speech recognition.

Abstract

We investigate intelligent personal assistants (IPAs) accessibility for deaf and hard of hearing (DHH) people who can use their voice in everyday communication. The inability of IPAs to understand diverse accents including deaf speech renders them largely inaccessible to non-signing and speaking DHH individuals. Using an Echo Show, we compare the usability of natural language input via spoken English; with Alexa's automatic speech recognition and a Wizard-of-Oz setting with a trained facilitator re-speaking commands against that of a large language model (LLM)-assisted touch interface in a mixed-methods study. The touch method was navigated through an LLM-powered "task prompter," which integrated the user's history and smart environment to suggest contextually-appropriate commands. Quantitative results showed no significant differences across both spoken English conditions vs LLM-assisted touch. Qualitative results showed variability in opinions on the usability of each method. Ultimately, it will be necessary to have robust deaf-accented speech recognized natively by IPAs.