The Return of Structural Handwritten Mathematical Expression Recognition

TL;DR

This paper presents a novel structural recognition system for handwritten mathematical expressions that improves interpretability and error analysis by explicitly modeling symbol spatial relations and traces, leveraging an auto-annotation system and modular architecture.

Contribution

It introduces an automatic annotation system and a modular structural recognition approach that explicitly models spatial relations in handwritten math expressions, enhancing interpretability.

Findings

Achieved competitive performance on CROHME-2023 benchmark.

Generated complete graph structures linking traces to symbols.

Enabled transparent error analysis and interpretability.

Abstract

Handwritten Mathematical Expression Recognition is foundational for educational technologies, enabling applications like digital note-taking and automated grading. While modern encoder-decoder architectures with large language models excel at LaTeX generation, they lack explicit symbol-to-trace alignment, a critical limitation for error analysis, interpretability, and spatially aware interactive applications requiring selective content updates. This paper introduces a structural recognition approach with two innovations: 1 an automatic annotation system that uses a neural network to map LaTeX equations to raw traces, automatically generating annotations for symbol segmentation, classification, and spatial relations, and 2 a modular structural recognition system that independently optimizes segmentation, classification, and relation prediction. By leveraging a dataset enriched with structural annotations from our auto-labeling system, the proposed recognition system combines graph-based trace sorting, a hybrid convolutional-recurrent network, and transformer-based correction to achieve competitive performance on the CROHME-2023 benchmark. Crucially, our structural recognition system generates a complete graph structure that directly links handwritten traces to predicted symbols, enabling transparent error analysis and interpretable outputs.