Exploring Reliable Spatiotemporal Dependencies for Efficient Visual Tracking

TL;DR

This paper introduces STDTrack, a lightweight visual tracker that leverages dense spatiotemporal sampling and novel modules to improve accuracy while maintaining real-time speed, bridging the gap with high-performance trackers.

Contribution

The paper proposes a novel framework integrating reliable spatiotemporal dependencies into lightweight trackers using dense sampling and new modules for enhanced performance.

Findings

Achieves state-of-the-art results on six benchmarks.

Operates at 192 FPS on GPU and 41 FPS on CPU.

Rivals high-performance non-real-time trackers like MixFormer.

Abstract

Recent advances in transformer-based lightweight object tracking have established new standards across benchmarks, leveraging the global receptive field and powerful feature extraction capabilities of attention mechanisms. Despite these achievements, existing methods universally employ sparse sampling during training--utilizing only one template and one search image per sequence--which fails to comprehensively explore spatiotemporal information in videos. This limitation constrains performance and cause the gap between lightweight and high-performance trackers. To bridge this divide while maintaining real-time efficiency, we propose STDTrack, a framework that pioneers the integration of reliable spatiotemporal dependencies into lightweight trackers. Our approach implements dense video sampling to maximize spatiotemporal information utilization. We introduce a temporally propagating spatiotemporal token to guide per-frame feature extraction. To ensure comprehensive target state representation, we disign the Multi-frame Information Fusion Module (MFIFM), which augments current dependencies using historical context. The MFIFM operates on features stored in our constructed Spatiotemporal Token Maintainer (STM), where a quality-based update mechanism ensures information reliability. Considering the scale variation among tracking targets, we develop a multi-scale prediction head to dynamically adapt to objects of different sizes. Extensive experiments demonstrate state-of-the-art results across six benchmarks. Notably, on GOT-10k, STDTrack rivals certain high-performance non-real-time trackers (e.g., MixFormer) while operating at 192 FPS(GPU) and 41 FPS(CPU).