Link prediction Graph Neural Networks for structure recognition of Handwritten Mathematical Expressions

TL;DR

This paper introduces a GNN-based method for recognizing handwritten mathematical expressions by modeling them as graphs, combining deep learning and graph parsing to improve structural recognition accuracy.

Contribution

It presents a novel integration of GNNs with deep BLSTM and 2D-CFG parsing for enhanced handwritten math expression structure recognition.

Findings

Effective graph-based recognition of HMEs

Improved accuracy over baseline methods

Refined structure prediction with GNN link prediction

Abstract

We propose a Graph Neural Network (GNN)-based approach for Handwritten Mathematical Expression (HME) recognition by modeling HMEs as graphs, where nodes represent symbols and edges capture spatial dependencies. A deep BLSTM network is used for symbol segmentation, recognition, and spatial relation classification, forming an initial primitive graph. A 2D-CFG parser then generates all possible spatial relations, while the GNN-based link prediction model refines the structure by removing unnecessary connections, ultimately forming the Symbol Label Graph. Experimental results demonstrate the effectiveness of our approach, showing promising performance in HME structure recognition.