Optimizing Multi-Modality Trackers via Significance-Regularized Tuning

TL;DR

This paper introduces a significance-regularized fine-tuning framework for multi-modality trackers, improving transferability and performance by balancing stability and adaptability through intrinsic parameter significance.

Contribution

It proposes a novel regularization method that leverages parameter significance to optimize pre-trained models for multi-modality tracking tasks.

Findings

Outperforms state-of-the-art on multiple benchmarks.

Enhances transferability across different modalities.

Improves stability and adaptability in multi-modal tracking.

Abstract

This paper tackles the critical challenge of optimizing multi-modality trackers by effectively adapting pre-trained models for RGB data. Existing fine-tuning paradigms oscillate between excessive flexibility and over-restriction, both leading to suboptimal plasticity-stability trade-offs. To mitigate this dilemma, we propose a novel significance-regularized fine-tuning framework, which delicately refines the learning process by incorporating intrinsic parameter significance. Through a comprehensive investigation of the transition from pre-trained to multi-modality contexts, we identify that parameters crucial to preserving foundational patterns and managing cross-domain shifts are the primary drivers of this issue. Specifically, we first probe the tangent space of pre-trained weights to measure and orient prior significance, dedicated to preserving generalization. Subsequently, we characterize transfer significance during the fine-tuning phase, emphasizing adaptability and stability. By incorporating these parameter significance terms as unified regularization, our method markedly enhances transferability across modalities. Extensive experiments showcase the superior performance of our method, surpassing current state-of-the-art techniques across various multi-modal tracking benchmarks. The source code and models are publicly available at https://github.com/zhiwen-xdu/SRTrack.