Unsupervised Training Data Generation of Handwritten Formulas using Generative Adversarial Networks with Self-Attention

TL;DR

This paper presents an innovative attention-based GAN system that synthesizes large datasets of handwritten mathematical formulas from LaTeX, addressing the data scarcity challenge in handwritten formula recognition.

Contribution

It introduces a novel GAN architecture with self-attention for translating rendered equations into handwritten formulas, generating extensive training data from LaTeX documents.

Findings

Generated datasets contain hundreds of thousands of formulas.

Synthesized data improves training for handwritten formula recognition.

Feasibility demonstrated on CROHME 2014 benchmark.

Abstract

The recognition of handwritten mathematical expressions in images and video frames is a difficult and unsolved problem yet. Deep convectional neural networks are basically a promising approach, but typically require a large amount of labeled training data. However, such a large training dataset does not exist for the task of handwritten formula recognition. In this paper, we introduce a system that creates a large set of synthesized training examples of mathematical expressions which are derived from LaTeX documents. For this purpose, we propose a novel attention-based generative adversarial network to translate rendered equations to handwritten formulas. The datasets generated by this approach contain hundreds of thousands of formulas, making it ideal for pretraining or the design of more complex models. We evaluate our synthesized dataset and the recognition approach on the CROHME 2014 benchmark dataset. Experimental results demonstrate the feasibility of the approach.