Beyond Detection: A Structure-Aware Framework for Scene Text Tracking

TL;DR

This paper introduces SymTrack, a novel structure-aware framework for scene text tracking in videos, addressing geometric distortions and visual ambiguities, and establishes new state-of-the-art results on multiple benchmarks.

Contribution

It presents the first systematic approach for scene text tracking, incorporating a dual-branch design with calibration and rectification mechanisms, and creates a new benchmark from existing datasets.

Findings

SymTrack outperforms previous trackers by up to 11.97% AUC.

It effectively handles geometric distortions and visual ambiguities.

The framework achieves state-of-the-art results on three benchmark datasets.

Abstract

Modern visual object trackers show impressive results on general targets, yet their performance drops substantially when dealing with scene text. Although currently underexplored, tracking text in videos is essential for dynamic text manipulations such as segmentation, removal, and editing. To fill this gap, this paper formalizes this specific task as Scene Text Tracking and presents the first systematic work for it. We identify three primary challenges in this task: 1) severe geometric distortions from perspective shifts, 2) high visual ambiguity across different instances, and 3) high sensitivity to fine-grained structural details. To address these issues, we propose SymTrack, a unified detection-free framework with synergistic dual-branch design. It integrates a Cross-Expert Calibration mechanism to reduce semantic bias, along with a Predictive Token Rectification mechanism to correct structural imbalances, complemented by an Adaptive Inference Engine that stabilizes predictions under motion constraints. Considering the lack of dedicated benchmarks for this task, we utilize three datasets from video text spotting to construct a benchmark with high-quality annotations. Extensive experiments demonstrate that SymTrack sets the new state-of-the-art on all three benchmarks, outperforming previous best trackers by up to 11.97\% AUC on $ \text{BOVText}_{\text{SOT}} $. Overall, our work promotes efficient and thorough text tracking, paving the way toward more generalized video text manipulation.