CLEval: Character-Level Evaluation for Text Detection and Recognition Tasks

TL;DR

CLEval introduces a character-level evaluation metric for text detection and recognition tasks, enabling fairer, more detailed assessment of end-to-end OCR systems by handling split/merge cases and partial correctness.

Contribution

It proposes a novel character-level evaluation metric that improves fine-grained assessment of OCR methods, addressing limitations of existing metrics.

Findings

Provides a more accurate comparison of OCR methods.

Enables detailed analysis of detection and recognition modules.

Supports development of improved text recognition systems.

Abstract

Despite the recent success of text detection and recognition methods, existing evaluation metrics fail to provide a fair and reliable comparison among those methods. In addition, there exists no end-to-end evaluation metric that takes characteristics of OCR tasks into account. Previous end-to-end metric contains cascaded errors from the binary scoring process applied in both detection and recognition tasks. Ignoring partially correct results raises a gap between quantitative and qualitative analysis, and prevents fine-grained assessment. Based on the fact that character is a key element of text, we hereby propose a Character-Level Evaluation metric (CLEval). In CLEval, the \textit{instance matching} process handles split and merge detection cases, and the \textit{scoring process} conducts character-level evaluation. By aggregating character-level scores, the CLEval metric provides a fine-grained evaluation of end-to-end results composed of the detection and recognition as well as individual evaluations for each module from the end-performance perspective. We believe that our metrics can play a key role in developing and analyzing state-of-the-art text detection and recognition methods. The evaluation code is publicly available at https://github.com/clovaai/CLEval.