Out-of-Vocabulary Challenge Report

TL;DR

This paper reports on the OOV 2022 challenge, highlighting the difficulty of recognizing unseen scene text instances and providing a new dataset to evaluate and improve OCR models' robustness and generalization.

Contribution

It introduces a new OOV dataset for scene text recognition and analyzes the performance gap of current models on unseen text instances.

Findings

State-of-the-art models perform poorly on OOV data

The new dataset reveals significant challenges in recognizing unseen scene text

The challenge encourages development of more robust OCR models

Abstract

This paper presents final results of the Out-Of-Vocabulary 2022 (OOV) challenge. The OOV contest introduces an important aspect that is not commonly studied by Optical Character Recognition (OCR) models, namely, the recognition of unseen scene text instances at training time. The competition compiles a collection of public scene text datasets comprising of 326,385 images with 4,864,405 scene text instances, thus covering a wide range of data distributions. A new and independent validation and test set is formed with scene text instances that are out of vocabulary at training time. The competition was structured in two tasks, end-to-end and cropped scene text recognition respectively. A thorough analysis of results from baselines and different participants is presented. Interestingly, current state-of-the-art models show a significant performance gap under the newly studied setting. We conclude that the OOV dataset proposed in this challenge will be an essential area to be explored in order to develop scene text models that achieve more robust and generalized predictions.