SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker

TL;DR

SEATrack is a novel multimodal tracker that enhances cross-modal alignment and global relation modeling, achieving a better balance of performance and efficiency across various tracking tasks.

Contribution

It introduces AMG-LoRA for dynamic attention alignment and HMoE for efficient global relation modeling, advancing multimodal tracking performance and efficiency.

Findings

Outperforms state-of-the-art methods in RGB-T, RGB-D, and RGB-E tracking.

Achieves a better balance of accuracy and computational efficiency.

Demonstrates the effectiveness of AMG-LoRA and HMoE modules.

Abstract

Parameter-efficient fine-tuning (PEFT) in multimodal tracking reveals a concerning trend where recent performance gains are often achieved at the cost of inflated parameter budgets, which fundamentally erodes PEFT's efficiency promise. In this work, we introduce SEATrack, a Simple, Efficient, and Adaptive two-stream multimodal tracker that tackles this performance-efficiency dilemma from two complementary perspectives. We first prioritize cross-modal alignment of matching responses, an underexplored yet pivotal factor that we argue is essential for breaking the trade-off. Specifically, we observe that modality-specific biases in existing two-stream methods generate conflicting matching attention maps, thereby hindering effective joint representation learning. To mitigate this, we propose AMG-LoRA, which seamlessly integrates Low-Rank Adaptation (LoRA) for domain adaptation with Adaptive Mutual Guidance (AMG) to dynamically refine and align attention maps across modalities. We then depart from conventional local fusion approaches by introducing a Hierarchical Mixture of Experts (HMoE) that enables efficient global relation modeling, effectively balancing expressiveness and computational efficiency in cross-modal fusion. Equipped with these innovations, SEATrack advances notable progress over state-of-the-art methods in balancing performance with efficiency across RGB-T, RGB-D, and RGB-E tracking tasks. \href{https://github.com/AutoLab-SAI-SJTU/SEATrack}{\textcolor{cyan}{Code is available}}.