Reading Chinese in Natural Scenes with a Bag-of-Radicals Prior

TL;DR

This paper introduces a novel radical-embedding approach for Chinese scene text recognition, leveraging ideographic descriptions and multi-task training to significantly improve performance over traditional methods.

Contribution

The paper proposes a radical-embedding representation and multi-task training strategy to enhance Chinese scene text recognition accuracy.

Findings

Model with RE + CVFM + multi-task training outperforms baselines on six datasets.

Radical-based supervision improves ideographic structure perception.

Significant performance gains on Chinese STR datasets.

Abstract

Scene text recognition (STR) on Latin datasets has been extensively studied in recent years, and state-of-the-art (SOTA) models often reach high accuracy. However, the performance on non-Latin transcripts, such as Chinese, is not satisfactory. In this paper, we collect six open-source Chinese STR datasets and evaluate a series of classic methods performing well on Latin datasets, finding a significant performance drop. To improve the performance on Chinese datasets, we propose a novel radical-embedding (RE) representation to utilize the ideographic descriptions of Chinese characters. The ideographic descriptions of Chinese characters are firstly converted to bags of radicals and then fused with learnable character embeddings by a character-vector-fusion-module (CVFM). In addition, we utilize a bag of radicals as supervision signals for multi-task training to improve the ideographic structure perception of our model. Experiments show performance of the model with RE + CVFM + multi-task training is superior compared with the baseline on six Chinese STR datasets. In addition, we utilize a bag of radicals as supervision signals for multi-task training to improve the ideographic structure perception of our model. Experiments show performance of the model with RE + CVFM + multi-task training is superior compared with the baseline on six Chinese STR datasets.