Multi-Modal Character Localization and Extraction for Chinese Text Recognition

TL;DR

This paper introduces LER, a novel multi-modal approach for Chinese text recognition that explicitly decouples characters and leverages multimodal information to improve accuracy, outperforming existing methods on large-scale benchmarks.

Contribution

The paper presents a new method that explicitly decouples Chinese characters and utilizes multimodal information for improved recognition accuracy.

Findings

Significantly outperforms existing Chinese text recognition methods.

Achieves impressive results on six English benchmarks.

Effective in recognizing complex Chinese characters.

Abstract

Scene text recognition (STR) methods have demonstrated their excellent capability in English text images. However, due to the complex inner structures of Chinese and the extensive character categories, it poses challenges for recognizing Chinese text in images. Recently, studies have shown that the methods designed for English text recognition encounter an accuracy bottleneck when recognizing Chinese text images. This raises the question: Is it appropriate to apply the model developed for English to the Chinese STR task? To explore this issue, we propose a novel method named LER, which explicitly decouples each character and independently recognizes characters while taking into account the complex inner structures of Chinese. LER consists of three modules: Localization, Extraction, and Recognition. Firstly, the localization module utilizes multimodal information to determine the character's position precisely. Then, the extraction module dissociates all characters in parallel. Finally, the recognition module considers the unique inner structures of Chinese to provide the text prediction results. Extensive experiments conducted on large-scale Chinese benchmarks indicate that our method significantly outperforms existing methods. Furthermore, extensive experiments conducted on six English benchmarks and the Union14M benchmark show impressive results in English text recognition by LER. Code is available at https://github.com/Pandarenlql/LER.