Benchmarking Online Sequence-to-Sequence and Character-based Handwriting Recognition from IMU-Enhanced Pens
Felix Ott, David R\"ugamer, Lucas Heublein, Tim Hamann and, Jens Barth, Bernd Bischl, Christopher Mutschler

TL;DR
This paper introduces new datasets and benchmarks for online handwriting recognition using sensor-enhanced pens, comparing sequence-to-sequence and character-based models on paper and tablet data.
Contribution
It provides the first comprehensive dataset and benchmark models for real-time online handwriting recognition from sensor-enhanced pens on paper.
Findings
Convolutional networks with BiLSTMs outperform Transformers.
Sensor data improves recognition accuracy.
Time-series augmentation enhances model performance.
Abstract
Purpose. Handwriting is one of the most frequently occurring patterns in everyday life and with it come challenging applications such as handwriting recognition (HWR), writer identification, and signature verification. In contrast to offline HWR that only uses spatial information (i.e., images), online HWR (OnHWR) uses richer spatio-temporal information (i.e., trajectory data or inertial data). While there exist many offline HWR datasets, there is only little data available for the development of OnHWR methods on paper as it requires hardware-integrated pens. Methods. This paper presents data and benchmark models for real-time sequence-to-sequence (seq2seq) learning and single character-based recognition. Our data is recorded by a sensor-enhanced ballpoint pen, yielding sensor data streams from triaxial accelerometers, a gyroscope, a magnetometer and a force sensor at 100 Hz. We propose…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHandwritten Text Recognition Techniques · Hand Gesture Recognition Systems · Image Processing and 3D Reconstruction
MethodsInceptionTime · Tanh Activation · Sigmoid Activation · Long Short-Term Memory · Sequence to Sequence
