TouchScribe: Augmenting Non-Visual Hand-Object Interactions with Automated Live Visual Descriptions

TL;DR

TouchScribe is a system that enhances non-visual hand-object interactions for visually impaired users by providing automated, real-time visual descriptions based on hand gestures and movements, improving object understanding.

Contribution

We introduce TouchScribe, a novel system combining egocentric hand gesture recognition with live visual descriptions to assist visually impaired users during object interaction.

Findings

Improves object understanding through real-time visual descriptions.

Recognizes common and unique gestures used by blind individuals.

User study shows increased confidence and efficiency in object exploration.

Abstract

People who are blind or have low vision regularly use their hands to interact with the physical world to gain access to objects' shape, size, weight, and texture. However, many rich visual features remain inaccessible through touch alone, making it difficult to distinguish similar objects, interpret visual affordances, and form a complete understanding of objects. In this work, we present TouchScribe, a system that augments hand-object interactions with automated live visual descriptions. We trained a custom egocentric hand interaction model to recognize both common gestures (e.g., grab to inspect, hold side-by-side to compare) and unique ones by blind people (e.g., point to explore color, or swipe to read available texts). Furthermore, TouchScribe provides real-time and adaptive feedback based on hand movement, from hand interaction states, to object labels, and to visual details. Our user study and technical evaluations demonstrate that TouchScribe can provide rich and useful descriptions to support object understanding. Finally, we discuss the implications of making live visual descriptions responsive to users' physical reach.