Diff-MM: Exploring Pre-trained Text-to-Image Generation Model for Unified Multi-modal Object Tracking

TL;DR

This paper introduces Diff-MM, a unified multi-modal object tracker leveraging pre-trained text-to-image models to improve tracking performance across various modalities with limited training data.

Contribution

It proposes a novel approach using a pre-trained Stable Diffusion model for multi-modal tracking and introduces a multi-modal sub-module tuning method for better modality integration.

Findings

Outperforms recent trackers, e.g., 8.3% higher AUC on TNL2K.

Utilizes pre-trained generative models for multi-modal understanding.

Achieves unified tracking with a single set of parameters.

Abstract

Multi-modal object tracking integrates auxiliary modalities such as depth, thermal infrared, event flow, and language to provide additional information beyond RGB images, showing great potential in improving tracking stabilization in complex scenarios. Existing methods typically start from an RGB-based tracker and learn to understand auxiliary modalities only from training data. Constrained by the limited multi-modal training data, the performance of these methods is unsatisfactory. To alleviate this limitation, this work proposes a unified multi-modal tracker Diff-MM by exploiting the multi-modal understanding capability of the pre-trained text-to-image generation model. Diff-MM leverages the UNet of pre-trained Stable Diffusion as a tracking feature extractor through the proposed parallel feature extraction pipeline, which enables pairwise image inputs for object tracking. We further introduce a multi-modal sub-module tuning method that learns to gain complementary information between different modalities. By harnessing the extensive prior knowledge in the generation model, we achieve a unified tracker with uniform parameters for RGB-N/D/T/E tracking. Experimental results demonstrate the promising performance of our method compared with recently proposed trackers, e.g., its AUC outperforms OneTracker by 8.3% on TNL2K.