ImAiR: Airwriting Recognition framework using Image Representation of IMU Signals

TL;DR

This paper introduces a novel airwriting recognition framework that encodes IMU sensor data as images and uses deep learning models for letter identification, advancing gesture recognition for human-computer interaction.

Contribution

The work proposes a new method of encoding IMU signals as images and applying deep learning models, improving airwriting recognition accuracy with multiple image encoding techniques and standard CNN architectures.

Findings

Effective encoding of IMU data as images improves recognition.

Deep learning models achieve high accuracy on public datasets.

Multi-model averaging enhances prediction reliability.

Abstract

The problem of Airwriting Recognition is focused on identifying letters written by movement of finger in free space. It is a type of gesture recognition where the dictionary corresponds to letters in a specific language. In particular, airwriting recognition using sensor data from wrist-worn devices can be used as a medium of user input for applications in Human-Computer Interaction (HCI). Recognition of in-air trajectories using such wrist-worn devices is limited in literature and forms the basis of the current work. In this paper, we propose an airwriting recognition framework by first encoding the time-series data obtained from a wearable Inertial Measurement Unit (IMU) on the wrist as images and then utilizing deep learning-based models for identifying the written alphabets. The signals recorded from 3-axis accelerometer and gyroscope in IMU are encoded as images using different techniques such as Self Similarity Matrix (SSM), Gramian Angular Field (GAF) and Markov Transition Field (MTF) to form two sets of 3-channel images. These are then fed to two separate classification models and letter prediction is made based on an average of the class conditional probabilities obtained from the two models. Several standard model architectures for image classification such as variants of ResNet, DenseNet, VGGNet, AlexNet and GoogleNet have been utilized. Experiments performed on two publicly available datasets demonstrate the efficacy of the proposed strategy. The code for our implementation will be made available at https://github.com/ayushayt/ImAiR.