A Multimodal Assistive System for Helping Visually Impaired in Social Interactions

TL;DR

This paper presents a real-time multimodal assistive system that provides non-verbal social cues to visually impaired users, demonstrating its usefulness and usability concerns through a user study.

Contribution

It introduces a novel multimodal system for social cues and evaluates its impact on visually impaired users in real social interactions.

Findings

Users find the system useful in social interactions.

Usability concerns need further addressing.

Provides insights for future assistive technology development.

Abstract

Access to non-verbal cues in social interactions is vital for people with visual impairment. It has been shown that non-verbal cues such as eye contact, number of people, their names and positions are helpful for individuals who are blind. While there is an increasing interest in developing systems to provide these cues less emphasis has been put in evaluating its impact on the visually impaired users. In this paper, we provide this analysis by conducting a user study with 12 visually impaired participants in a typical social interaction setting. We design a real time multi-modal system that provides such non-verbal cues via audio and haptic interfaces. The study shows that such systems are generally perceived as useful in social interaction and brings forward some concerns that are not being addressed in its usability aspects. The study provides important insight about developing such technology for this significant part of society.