Towards Sequence-Level Training for Visual Tracking

TL;DR

This paper proposes a sequence-level training approach for visual tracking using reinforcement learning, aiming to align training with the sequence-based nature of the task and improve model robustness and accuracy.

Contribution

It introduces a sequence-level training strategy that enhances existing tracking models without changing their architectures, using reinforcement learning and improved data sampling, objectives, and augmentation.

Findings

Consistent performance improvements on LaSOT, TrackingNet, and GOT-10k benchmarks.

Enhanced robustness and accuracy of four representative tracking models.

No architectural modifications needed for the proposed training methods.

Abstract

Despite the extensive adoption of machine learning on the task of visual object tracking, recent learning-based approaches have largely overlooked the fact that visual tracking is a sequence-level task in its nature; they rely heavily on frame-level training, which inevitably induces inconsistency between training and testing in terms of both data distributions and task objectives. This work introduces a sequence-level training strategy for visual tracking based on reinforcement learning and discusses how a sequence-level design of data sampling, learning objectives, and data augmentation can improve the accuracy and robustness of tracking algorithms. Our experiments on standard benchmarks including LaSOT, TrackingNet, and GOT-10k demonstrate that four representative tracking models, SiamRPN++, SiamAttn, TransT, and TrDiMP, consistently improve by incorporating the proposed methods in training without modifying architectures.