SemiHMER: Semi-supervised Handwritten Mathematical Expression Recognition using pseudo-labels

TL;DR

SemiHMER introduces a semi-supervised learning framework with consistency regularization and pseudo-labels to improve handwritten mathematical expression recognition, leveraging unlabeled data and a novel counting module.

Contribution

The paper proposes a novel semi-supervised framework with a dual-branch consistency regularization and a global dynamic counting module for improved HMER accuracy.

Findings

Achieved up to 5.47% accuracy improvement on CROHME datasets.

Effectively leverages unlabeled data with pseudo-labels.

Reduces recognition errors in long-distance formulas.

Abstract

In this paper, we study semi-supervised Handwritten Mathematical Expression Recognition (HMER) via exploring both labeled data and extra unlabeled data. We propose a novel consistency regularization framework, termed SemiHMER, which introduces dual-branch semi-supervised learning. Specifically, we enforce consistency between the two networks for the same input image. The pseudo-label, generated by one perturbed recognition network, is utilized to supervise the other network using the standard cross-entropy loss. The SemiHMER consistency encourages high similarity between the predictions of the two perturbed networks for the same input image and expands the training data by leveraging unlabeled data with pseudo-labels. We further introduce a weak-to-strong strategy by applying different levels of augmentation to each branch, effectively expanding the training data and enhancing the quality of network training. Additionally, we propose a novel module, the Global Dynamic Counting Module (GDCM), to enhance the performance of the HMER decoder by alleviating recognition inaccuracies in long-distance formula recognition and reducing the occurrence of repeated characters. The experimental results demonstrate that our work achieves significant performance improvements, with an average accuracy increase of 5.47% on CROHME14, 4.87% on CROHME16, and 5.25% on CROHME19, compared to our baselines.