MDiff4STR: Mask Diffusion Model for Scene Text Recognition

TL;DR

This paper introduces MDiff4STR, a novel Mask Diffusion Model tailored for Scene Text Recognition, which improves accuracy over vanilla diffusion models and surpasses state-of-the-art auto-regressive models while maintaining high efficiency.

Contribution

The paper presents MDiff4STR, a diffusion-based approach for STR with two key strategies to address training-inference gap and overconfidence, achieving superior accuracy and efficiency.

Findings

MDiff4STR outperforms popular STR models across various benchmarks.

It surpasses state-of-the-art auto-regressive models in accuracy.

Maintains fast inference with only three denoising steps.

Abstract

Mask Diffusion Models (MDMs) have recently emerged as a promising alternative to auto-regressive models (ARMs) for vision-language tasks, owing to their flexible balance of efficiency and accuracy. In this paper, for the first time, we introduce MDMs into the Scene Text Recognition (STR) task. We show that vanilla MDM lags behind ARMs in terms of accuracy, although it improves recognition efficiency. To bridge this gap, we propose MDiff4STR, a Mask Diffusion model enhanced with two key improvement strategies tailored for STR. Specifically, we identify two key challenges in applying MDMs to STR: noising gap between training and inference, and overconfident predictions during inference. Both significantly hinder the performance of MDMs. To mitigate the first issue, we develop six noising strategies that better align training with inference behavior. For the second, we propose a token-replacement noise mechanism that provides a non-mask noise type, encouraging the model to reconsider and revise overly confident but incorrect predictions. We conduct extensive evaluations of MDiff4STR on both standard and challenging STR benchmarks, covering diverse scenarios including irregular, artistic, occluded, and Chinese text, as well as whether the use of pretraining. Across these settings, MDiff4STR consistently outperforms popular STR models, surpassing state-of-the-art ARMs in accuracy, while maintaining fast inference with only three denoising steps. Code: https://github.com/Topdu/OpenOCR.