Masked Next-Scale Prediction for Self-supervised Scene Text Recognition

TL;DR

The paper introduces Masked Next-Scale Prediction (MNSP), a self-supervised framework that models hierarchical structures in scene text recognition by predicting higher-resolution features from lower-resolution contexts, improving accuracy and robustness.

Contribution

MNSP is the first unified self-supervised approach to explicitly model cross-scale structural evolution in scene text recognition, combining cross-scale prediction with masked reconstruction.

Findings

Achieves 86.2% accuracy on Union14M benchmark.

Reaches 96.7% accuracy across six standard datasets.

Improves robustness under extreme scale and layout variations.

Abstract

Scene Text Recognition requires modeling visual structures that evolve from coarse layouts to fine-grained character strokes. Training such models relies on large amounts of annotated data. Recent self-supervised approaches, such as Masked Image Modeling (MIM), alleviate this dependency by leveraging large-scale unlabeled data. Yet most existing MIM methods operate at a single spatial scale and fail to capture the hierarchical nature of scene text. In this work, we introduce Masked Next-Scale Prediction (MNSP), a unified self-supervised framework designed to explicitly model cross-scale structural evolution. The framework incorporates Next-Scale Prediction (NSP), which learns hierarchical representations by predicting higher-resolution features from lower-resolution contexts. Naive scale prediction, however, tends to produce spatially diffuse attention, directing the model toward background regions rather than textual structures. MNSP resolves this limitation by jointly learning cross-scale prediction and masked image reconstruction. NSP captures global layout priors across resolutions, while masked reconstruction imposes strong local constraints that guide attention toward informative text regions. A Multi-scale Linguistic Alignment module further maintains semantic consistency across different resolutions. Extensive experiments demonstrate that MNSP achieves state-of-the-art performance, reaching 86.2\% average accuracy on the challenging Union14M benchmark and 96.7\% across six standard datasets. Additional analyses show that our method improves robustness under extreme scale and layout variations. Code is available at https://github.com/CzhczhcHczh/MNSP