Hear the Scene: Audio-Enhanced Text Spotting

TL;DR

This paper presents a novel text spotting method that relies solely on transcription annotations, reducing annotation costs and incorporating audio-based labeling, while achieving competitive accuracy.

Contribution

It introduces a query-based weakly-supervised text spotting framework that uses only transcription annotations and includes a circular curriculum learning strategy and audio annotation support.

Findings

Achieves competitive performance without location annotations

Reduces annotation effort by using transcription and audio labels

Employs a novel coarse-to-fine cross-attention localization mechanism

Abstract

Recent advancements in scene text spotting have focused on end-to-end methodologies that heavily rely on precise location annotations, which are often costly and labor-intensive to procure. In this study, we introduce an innovative approach that leverages only transcription annotations for training text spotting models, substantially reducing the dependency on elaborate annotation processes. Our methodology employs a query-based paradigm that facilitates the learning of implicit location features through the interaction between text queries and image embeddings. These features are later refined during the text recognition phase using an attention activation map. Addressing the challenges associated with training a weakly-supervised model from scratch, we implement a circular curriculum learning strategy to enhance model convergence. Additionally, we introduce a coarse-to-fine cross-attention localization mechanism for more accurate text instance localization. Notably, our framework supports audio-based annotation, which significantly diminishes annotation time and provides an inclusive alternative for individuals with disabilities. Our approach achieves competitive performance against existing benchmarks, demonstrating that high accuracy in text spotting can be attained without extensive location annotations.