Exploring Modality-Aware Fusion and Decoupled Temporal Propagation for Multi-Modal Object Tracking

TL;DR

This paper introduces MDTrack, a multimodal object tracking framework that employs modality-specific fusion and decoupled temporal propagation to improve tracking accuracy across various sensor modalities.

Contribution

The paper proposes a novel modality aware fusion method with expert gating and decoupled temporal propagation using separate state space models for each modality, enhancing temporal feature discrimination.

Findings

Achieves state-of-the-art performance on five multimodal tracking benchmarks.

Effectively captures modality-specific temporal information through decoupled models.

Demonstrates superior adaptive fusion via Mixture of Experts with gating mechanism.

Abstract

Most existing multimodal trackers adopt uniform fusion strategies, overlooking the inherent differences between modalities. Moreover, they propagate temporal information through mixed tokens, leading to entangled and less discriminative temporal representations. To address these limitations, we propose MDTrack, a novel framework for modality aware fusion and decoupled temporal propagation in multimodal object tracking. Specifically, for modality aware fusion, we allocate dedicated experts to each modality, including infrared, event, depth, and RGB, to process their respective representations. The gating mechanism within the Mixture of Experts dynamically selects the optimal experts based on the input features, enabling adaptive and modality specific fusion. For decoupled temporal propagation, we introduce two separate State Space Model structures to independently store and update the hidden states of the RGB and X modal streams, effectively capturing their distinct temporal information. To ensure synergy between the two temporal representations, we incorporate a set of cross attention modules between the input features of the two SSMs, facilitating implicit information exchange. The resulting temporally enriched features are then integrated into the backbone through another set of cross attention modules, enhancing MDTrack's ability to leverage temporal information. Extensive experiments demonstrate the effectiveness of our proposed method. Both MDTrack S and MDTrack U achieve state of the art performance across five multimodal tracking benchmarks.