MTNet: Learning modality-aware representation with transformer for RGBT tracking

TL;DR

This paper introduces MTNet, a transformer-based RGBT tracking method that effectively learns modality-specific cues and global dependencies, improving tracking accuracy and robustness in real-time.

Contribution

The paper proposes a novel modality-aware transformer network with specialized modules and a dynamic update strategy for enhanced RGBT tracking.

Findings

Achieves state-of-the-art results on three RGBT benchmarks.

Operates in real-time with improved accuracy.

Effectively handles scale variation and deformation.

Abstract

The ability to learn robust multi-modality representation has played a critical role in the development of RGBT tracking. However, the regular fusion paradigm and the invariable tracking template remain restrictive to the feature interaction. In this paper, we propose a modality-aware tracker based on transformer, termed MTNet. Specifically, a modality-aware network is presented to explore modality-specific cues, which contains both channel aggregation and distribution module(CADM) and spatial similarity perception module (SSPM). A transformer fusion network is then applied to capture global dependencies to reinforce instance representations. To estimate the precise location and tackle the challenges, such as scale variation and deformation, we design a trident prediction head and a dynamic update strategy which jointly maintain a reliable template for facilitating inter-frame communication. Extensive experiments validate that the proposed method achieves satisfactory results compared with the state-of-the-art competitors on three RGBT benchmarks while reaching real-time speed.