Textual Description for Mathematical Equations

TL;DR

This paper introduces MED, a deep learning model inspired by image captioning, that generates textual descriptions of mathematical equations to facilitate reading and understanding of complex mathematical expressions.

Contribution

The paper proposes a novel end-to-end neural network model for generating textual descriptions of mathematical equations, addressing the challenge of reading complex mathematical expressions.

Findings

Students can accurately write equations from textual descriptions

Generated datasets enable training and evaluation of the model

MED effectively interprets mathematical equations into natural language

Abstract

Reading of mathematical expression or equation in the document images is very challenging due to the large variability of mathematical symbols and expressions. In this paper, we pose reading of mathematical equation as a task of generation of the textual description which interprets the internal meaning of this equation. Inspired by the natural image captioning problem in computer vision, we present a mathematical equation description (MED) model, a novel end-to-end trainable deep neural network based approach that learns to generate a textual description for reading mathematical equation images. Our MED model consists of a convolution neural network as an encoder that extracts features of input mathematical equation images and a recurrent neural network with attention mechanism which generates description related to the input mathematical equation images. Due to the unavailability of mathematical equation image data sets with their textual descriptions, we generate two data sets for experimental purpose. To validate the effectiveness of our MED model, we conduct a real-world experiment to see whether the students are able to write equations by only reading or listening their textual descriptions or not. Experiments conclude that the students are able to write most of the equations correctly by reading their textual descriptions only.