SOI is the Root of All Evil: Quantifying and Breaking Similar Object Interference in Single Object Tracking

TL;DR

This paper systematically investigates Similar Object Interference (SOI) in Single Object Tracking, demonstrating its impact on performance, and introduces a benchmark and a new paradigm using vision-language models to improve tracking robustness.

Contribution

It is the first to quantify SOI's impact, create SOIBench for semantic guidance evaluation, and propose VLM-based external guidance to enhance tracking performance.

Findings

Eliminating interference sources improves tracking accuracy significantly.

Existing vision-language tracking methods underperform with semantic guidance.

VLM-based external guidance boosts tracking accuracy up to 0.93 AUC.

Abstract

In this paper, we present the first systematic investigation and quantification of Similar Object Interference (SOI), a long-overlooked yet critical bottleneck in Single Object Tracking (SOT). Through controlled Online Interference Masking (OIM) experiments, we quantitatively demonstrate that eliminating interference sources leads to substantial performance improvements (AUC gains up to 4.35) across all SOTA trackers, directly validating SOI as a primary constraint for robust tracking and highlighting the feasibility of external cognitive guidance. Building upon these insights, we adopt natural language as a practical form of external guidance, and construct SOIBench-the first semantic cognitive guidance benchmark specifically targeting SOI challenges. It automatically mines SOI frames through multi-tracker collective judgment and introduces a multi-level annotation protocol to generate precise semantic guidance texts. Systematic evaluation on SOIBench reveals a striking finding: existing vision-language tracking (VLT) methods fail to effectively exploit semantic cognitive guidance, achieving only marginal improvements or even performance degradation (AUC changes of -0.26 to +0.71). In contrast, we propose a novel paradigm employing large-scale vision-language models (VLM) as external cognitive engines that can be seamlessly integrated into arbitrary RGB trackers. This approach demonstrates substantial improvements under semantic cognitive guidance (AUC gains up to 0.93), representing a significant advancement over existing VLT methods. We hope SOIBench will serve as a standardized evaluation platform to advance semantic cognitive tracking research and contribute new insights to the tracking research community.