Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition

TL;DR

This paper introduces a multi-scale attention model with a dense encoder for handwritten mathematical expression recognition, significantly improving accuracy by capturing fine details and handling scale variations.

Contribution

It proposes a novel multi-scale attention mechanism combined with a densely connected encoder to enhance recognition of complex handwritten math expressions.

Findings

Achieved 52.8% accuracy on CROHME 2014

Achieved 50.1% accuracy on CROHME 2016

Outperforms previous state-of-the-art methods

Abstract

Handwritten mathematical expression recognition is a challenging problem due to the complicated two-dimensional structures, ambiguous handwriting input and variant scales of handwritten math symbols. To settle this problem, we utilize the attention based encoder-decoder model that recognizes mathematical expression images from two-dimensional layouts to one-dimensional LaTeX strings. We improve the encoder by employing densely connected convolutional networks as they can strengthen feature extraction and facilitate gradient propagation especially on a small training set. We also present a novel multi-scale attention model which is employed to deal with the recognition of math symbols in different scales and save the fine-grained details that will be dropped by pooling operations. Validated on the CROHME competition task, the proposed method significantly outperforms the state-of-the-art methods with an expression recognition accuracy of 52.8% on CROHME 2014 and 50.1% on CROHME 2016, by only using the official training dataset.