Towards an IMU-based Pen Online Handwriting Recognizer

TL;DR

This paper introduces an IMU-based online handwriting recognition system using sensor-equipped pens and deep learning, enabling word recognition without specialized surfaces or language models.

Contribution

It presents a novel IMU-based handwriting recognition approach with a deep neural network trained on sensor data, eliminating the need for sequence segmentation.

Findings

Achieved character error rates of 17.97% and 17.08% on seen and unseen words.

Demonstrated effective recognition using inertial sensors without a dedicated writing surface.

Validated the approach with a dataset collected from multiple sensor-equipped pens.

Abstract

Most online handwriting recognition systems require the use of specific writing surfaces to extract positional data. In this paper we present a online handwriting recognition system for word recognition which is based on inertial measurement units (IMUs) for digitizing text written on paper. This is obtained by means of a sensor-equipped pen that provides acceleration, angular velocity, and magnetic forces streamed via Bluetooth. Our model combines convolutional and bidirectional LSTM networks, and is trained with the Connectionist Temporal Classification loss that allows the interpretation of raw sensor data into words without the need of sequence segmentation. We use a dataset of words collected using multiple sensor-enhanced pens and evaluate our model on distinct test sets of seen and unseen words achieving a character error rate of 17.97% and 17.08%, respectively, without the use of a dictionary or language model