An Accelerometer Based Calculator for Visually Impaired People Using Mobile Devices

TL;DR

This paper presents a gesture recognition system using accelerometer data from mobile devices to assist visually impaired users, achieving high accuracy in classifying gestures for a talking calculator application.

Contribution

It introduces a novel approach using Dynamic Time Warping for high-accuracy, user-independent gesture classification tailored for visually impaired users on mobile devices.

Findings

Achieved 96.7% accuracy with normal users

Achieved 95.5% accuracy with visually impaired users

Dynamic warping window size significantly improves recognition performance

Abstract

Recent trend of touch-screen devices produces an accessibility barrier for visually impaired people. On the other hand, these devices come with sensors such as accelerometer. This calls for new approaches to human computer interface (HCI). In this study, our aim is to find an alternative approach to classify 20 different hand gestures captured by iPhone 3GS's built-in accelerometer and make high accuracy on user-independent classifications using Dynamic Time Warping (DTW) with dynamic warping window sizes. 20 gestures with 1,100 gesture data are collected from 15 normal-visioned people. This data set is used for training. Experiment-1 based on this data set produced an accuracy rate of 96.7~\%. In order for visually impaired people to use the system, a gesture recognition based "talking" calculator is implemented. In Experiment-2, 4 visually impaired end-users used the calculator and obtained 95.5~\% accuracy rate among 17 gestures with 720 gesture data totally. Contributions of the techniques to the end result is also investigated. Dynamic warping window size is found to be the most effective one. The data and the code is available.