Online trajectory recovery from offline handwritten Japanese kanji characters

TL;DR

This paper presents a deep learning approach to reconstruct online handwritten stroke sequences of Japanese kanji from offline images, improving offline recognition accuracy by recovering trajectory information.

Contribution

It introduces a neural network model with CNN encoder and LSTM decoder with attention to recover stroke sequences from offline images, addressing the complex challenge of multiple strokes and crossings.

Findings

Improved offline handwritten kanji recognition accuracy.

Effective trajectory reconstruction demonstrated visually and through recognition tests.

Attention mechanism enhances stroke sequence generation.

Abstract

In general, it is straightforward to render an offline handwriting image from an online handwriting pattern. However, it is challenging to reconstruct an online handwriting pattern given an offline handwriting image, especially for multiple-stroke character as Japanese kanji. The multiple-stroke character requires not only point coordinates but also stroke orders whose difficulty is exponential growth by the number of strokes. Besides, several crossed and touch points might increase the difficulty of the recovered task. We propose a deep neural network-based method to solve the recovered task using a large online handwriting database. Our proposed model has two main components: Convolutional Neural Network-based encoder and Long Short-Term Memory Network-based decoder with an attention layer. The encoder focuses on feature extraction while the decoder refers to the extracted features and generates the time-sequences of coordinates. We also demonstrate the effect of the attention layer to guide the decoder during the reconstruction. We evaluate the performance of the proposed method by both visual verification and handwritten character recognition. Although the visual verification reveals some problems, the recognition experiments demonstrate the effect of trajectory recovery in improving the accuracy of offline handwritten character recognition when online recognition for the recovered trajectories are combined.