EventSTR: A Benchmark Dataset and Baselines for Event Stream based Scene Text Recognition

TL;DR

This paper introduces EventSTR, a large-scale benchmark dataset for event-based scene text recognition, along with a novel framework that leverages event camera data and large language models to improve recognition accuracy under challenging conditions.

Contribution

It provides the first large-scale event-based scene text recognition dataset and proposes a new framework integrating event features with language models for improved accuracy.

Findings

EventSTR dataset contains 9,928 high-definition event samples with Chinese and English characters.

The proposed SimC-ESTR framework outperforms existing methods on EventSTR and simulation datasets.

The approach effectively handles challenging conditions like low illumination and motion blur in scene text recognition.

Abstract

Mainstream Scene Text Recognition (STR) algorithms are developed based on RGB cameras which are sensitive to challenging factors such as low illumination, motion blur, and cluttered backgrounds. In this paper, we propose to recognize the scene text using bio-inspired event cameras by collecting and annotating a large-scale benchmark dataset, termed EventSTR. It contains 9,928 high-definition (1280 * 720) event samples and involves both Chinese and English characters. We also benchmark multiple STR algorithms as the baselines for future works to compare. In addition, we propose a new event-based scene text recognition framework, termed SimC-ESTR. It first extracts the event features using a visual encoder and projects them into tokens using a Q-former module. More importantly, we propose to augment the vision tokens based on a memory mechanism before feeding into the large language models. A similarity-based error correction mechanism is embedded within the large language model to correct potential minor errors fundamentally based on contextual information. Extensive experiments on the newly proposed EventSTR dataset and two simulation STR datasets fully demonstrate the effectiveness of our proposed model. We believe that the dataset and algorithmic model can innovatively propose an event-based STR task and are expected to accelerate the application of event cameras in various industries. The source code and pre-trained models will be released on https://github.com/Event-AHU/EventSTR