Cross-Modal Object Tracking: Modality-Aware Representations and A Unified Benchmark

TL;DR

This paper introduces a new cross-modal object tracking dataset and a modality-aware tracking algorithm that effectively handles RGB and NIR data, addressing the challenges of heterogeneous visual properties in surveillance applications.

Contribution

The work provides the first large-scale cross-modal tracking dataset and proposes a novel, flexible modality-aware representation method for improved tracking across RGB and NIR modalities.

Findings

The proposed algorithm outperforms 17 state-of-the-art methods on the new dataset.

The dataset contains 654 sequences with over 481,000 frames, supporting robust evaluation.

The modality-aware approach effectively mitigates appearance gaps between RGB and NIR data.

Abstract

In many visual systems, visual tracking often bases on RGB image sequences, in which some targets are invalid in low-light conditions, and tracking performance is thus affected significantly. Introducing other modalities such as depth and infrared data is an effective way to handle imaging limitations of individual sources, but multi-modal imaging platforms usually require elaborate designs and cannot be applied in many real-world applications at present. Near-infrared (NIR) imaging becomes an essential part of many surveillance cameras, whose imaging is switchable between RGB and NIR based on the light intensity. These two modalities are heterogeneous with very different visual properties and thus bring big challenges for visual tracking. However, existing works have not studied this challenging problem. In this work, we address the cross-modal object tracking problem and contribute a new video dataset, including 654 cross-modal image sequences with over 481K frames in total, and the average video length is more than 735 frames. To promote the research and development of cross-modal object tracking, we propose a new algorithm, which learns the modality-aware target representation to mitigate the appearance gap between RGB and NIR modalities in the tracking process. It is plug-and-play and could thus be flexibly embedded into different tracking frameworks. Extensive experiments on the dataset are conducted, and we demonstrate the effectiveness of the proposed algorithm in two representative tracking frameworks against 17 state-of-the-art tracking methods. We will release the dataset for free academic usage, dataset download link and code will be released soon.