Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking

TL;DR

This paper introduces a dual-adapter framework that enhances multi-modal object tracking by incorporating frequency-aware prompts and a multi-level memory system, improving robustness and accuracy without extensive fine-tuning.

Contribution

It proposes a novel frequency-guided visual adapter and a multilevel memory adapter to better utilize modality-specific frequency information and temporal dependencies in a frozen tracker.

Findings

Achieves state-of-the-art results on RGB-Thermal, RGB-Depth, and RGB-Event benchmarks.

Demonstrates improved robustness against occlusion, motion blur, and illumination changes.

Maintains high efficiency with parameter-effective design.

Abstract

Prompt-learning-based multi-modal trackers have made strong progress by using lightweight visual adapters to inject auxiliary-modality cues into frozen foundation models. However, they still underutilize two essentials: modality-specific frequency structure and long-range temporal dependencies. We present Learning Frequency and Memory-Aware Prompts, a dual-adapter framework that injects lightweight prompts into a frozen RGB tracker. A frequency-guided visual adapter adaptively transfers complementary cues across modalities by jointly calibrating spatial, channel, and frequency components, narrowing the modality gap without full fine-tuning. A multilevel memory adapter with short, long, and permanent memory stores, updates, and retrieves reliable temporal context, enabling consistent propagation across frames and robust recovery from occlusion, motion blur, and illumination changes. This unified design preserves the efficiency of prompt learning while strengthening cross-modal interaction and temporal coherence. Extensive experiments on RGB-Thermal, RGB-Depth, and RGB-Event benchmarks show consistent state-of-the-art results over fully fine-tuned and adapter-based baselines, together with favorable parameter efficiency and runtime. Code and models are available at https://github.com/xuboyue1999/mmtrack.git.