The Devil is in Fine-tuning and Long-tailed Problems:A New Benchmark for Scene Text Detection

TL;DR

This paper identifies the limitations of current scene text detectors in real-world scenarios due to fine-tuning gaps and long-tailed data distributions, proposing a new benchmark and a self-supervised method to improve robustness.

Contribution

It introduces a Long-Tailed Benchmark for scene text detection and a self-supervised baseline method, addressing generalization issues caused by fine-tuning and data imbalance.

Findings

Fine-tuning often reduces model generalization in new domains.

Long-tailed distribution challenges hinder detection of rare text categories.

The proposed benchmark enables comprehensive evaluation of long-tailed scene text detection.

Abstract

Scene text detection has seen the emergence of high-performing methods that excel on academic benchmarks. However, these detectors often fail to replicate such success in real-world scenarios. We uncover two key factors contributing to this discrepancy through extensive experiments. First, a \textit{Fine-tuning Gap}, where models leverage \textit{Dataset-Specific Optimization} (DSO) paradigm for one domain at the cost of reduced effectiveness in others, leads to inflated performances on academic benchmarks. Second, the suboptimal performance in practical settings is primarily attributed to the long-tailed distribution of texts, where detectors struggle with rare and complex categories as artistic or overlapped text. Given that the DSO paradigm might undermine the generalization ability of models, we advocate for a \textit{Joint-Dataset Learning} (JDL) protocol to alleviate the Fine-tuning Gap. Additionally, an error analysis is conducted to identify three major categories and 13 subcategories of challenges in long-tailed scene text, upon which we propose a Long-Tailed Benchmark (LTB). LTB facilitates a comprehensive evaluation of ability to handle a diverse range of long-tailed challenges. We further introduce MAEDet, a self-supervised learning-based method, as a strong baseline for LTB. The code is available at https://github.com/pd162/LTB.