Offline Handwritten Mathematical Recognition using Adversarial Learning and Transformers

TL;DR

This paper introduces an encoder-decoder model with adversarial learning and transformers for offline handwritten mathematical expression recognition, significantly improving accuracy on the CROHME dataset.

Contribution

It proposes a novel adversarial learning framework with semantic-invariant feature extraction combined with DenseNet and transformer architecture for improved recognition.

Findings

Achieved approximately 4% improvement on CROHME 2019 test set.

Enhanced recognition accuracy over previous methods.

Demonstrated effectiveness of adversarial learning in offline HMER.

Abstract

Offline Handwritten Mathematical Expression Recognition (HMER) is a major area in the field of mathematical expression recognition. Offline HMER is often viewed as a much harder problem as compared to online HMER due to a lack of temporal information and variability in writing style. In this paper, we purpose a encoder-decoder model that uses paired adversarial learning. Semantic-invariant features are extracted from handwritten mathematical expression images and their printed mathematical expression counterpart in the encoder. Learning of semantic-invariant features combined with the DenseNet encoder and transformer decoder, helped us to improve the expression rate from previous studies. Evaluated on the CROHME dataset, we have been able to improve latest CROHME 2019 test set results by 4% approx.