Mobile Sound Recognition for the Deaf and Hard of Hearing

TL;DR

This paper explores developing a smartphone-based environmental sound recognition system to aid deaf and hard of hearing individuals, emphasizing on-device processing and user feedback.

Contribution

It presents a comprehensive requirements analysis and a prototype system that performs all sound recognition tasks on smartphones, enhancing accessibility.

Findings

User feedback was positive and insightful.

On-device processing proved feasible and effective.

The system increased situational awareness for users.

Abstract

Human perception of surrounding events is strongly dependent on audio cues. Thus, acoustic insulation can seriously impact situational awareness. We present an exploratory study in the domain of assistive computing, eliciting requirements and presenting solutions to problems found in the development of an environmental sound recognition system, which aims to assist deaf and hard of hearing people in the perception of sounds. To take advantage of smartphones computational ubiquity, we propose a system that executes all processing on the device itself, from audio features extraction to recognition and visual presentation of results. Our application also presents the confidence level of the classification to the user. A test of the system conducted with deaf users provided important and inspiring feedback from participants.