Semantic Graph Representation Learning for Handwritten Mathematical Expression Recognition

TL;DR

This paper introduces a semantic graph-based method with a semantic aware module to improve handwritten mathematical expression recognition by explicitly modeling symbol interactions, leading to better accuracy.

Contribution

It proposes a novel semantic aware module that enhances symbol relationship understanding in HMER models, improving recognition performance.

Findings

Achieves state-of-the-art results on CROHME and HME100K datasets.

Effectively models symbol interactions through semantic graphs.

Improves recognition accuracy by explicitly learning symbol relationships.

Abstract

Handwritten mathematical expression recognition (HMER) has attracted extensive attention recently. However, current methods cannot explicitly study the interactions between different symbols, which may fail when faced similar symbols. To alleviate this issue, we propose a simple but efficient method to enhance semantic interaction learning (SIL). Specifically, we firstly construct a semantic graph based on the statistical symbol co-occurrence probabilities. Then we design a semantic aware module (SAM), which projects the visual and classification feature into semantic space. The cosine distance between different projected vectors indicates the correlation between symbols. And jointly optimizing HMER and SIL can explicitly enhances the model's understanding of symbol relationships. In addition, SAM can be easily plugged into existing attention-based models for HMER and consistently bring improvement. Extensive experiments on public benchmark datasets demonstrate that our proposed module can effectively enhance the recognition performance. Our method achieves better recognition performance than prior arts on both CROHME and HME100K datasets.