MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling

TL;DR

MSTAR introduces a box-free, multi-query scene text retrieval method utilizing attention recycling and progressive vision embedding, achieving superior performance without costly bounding box annotations and establishing a new benchmark dataset.

Contribution

The paper presents MSTAR, a novel box-free approach for multi-query scene text retrieval that unifies diverse query types and introduces the MQTR dataset for comprehensive evaluation.

Findings

MSTAR outperforms previous models by 6.4% MAP on Total-Text.

MSTAR surpasses prior models by 8.5% on the MQTR benchmark.

Eliminates the need for bounding box annotations in scene text retrieval.

Abstract

Scene text retrieval has made significant progress with the assistance of accurate text localization. However, existing approaches typically require costly bounding box annotations for training. Besides, they mostly adopt a customized retrieval strategy but struggle to unify various types of queries to meet diverse retrieval needs. To address these issues, we introduce Muti-query Scene Text retrieval with Attention Recycling (MSTAR), a box-free approach for scene text retrieval. It incorporates progressive vision embedding to dynamically capture the multi-grained representation of texts and harmonizes free-style text queries with style-aware instructions. Additionally, a multi-instance matching module is integrated to enhance vision-language alignment. Furthermore, we build the Multi-Query Text Retrieval (MQTR) dataset, the first benchmark designed to evaluate the multi-query scene text retrieval capability of models, comprising four query types and 16k images. Extensive experiments demonstrate the superiority of our method across seven public datasets and the MQTR dataset. Notably, MSTAR marginally surpasses the previous state-of-the-art model by 6.4% in MAP on Total-Text while eliminating box annotation costs. Moreover, on the MQTR benchmark, MSTAR significantly outperforms the previous models by an average of 8.5%. The code and datasets are available at https://github.com/yingift/MSTAR.