Mamba-FETrack V2: Revisiting State Space Model for Frame-Event based Visual Object Tracking

TL;DR

This paper introduces Mamba-FETrack V2, an efficient RGB-Event object tracking framework that leverages a linear-complexity Vision Mamba network for effective cross-modal feature fusion, achieving high performance with reduced computational cost.

Contribution

The paper presents a novel lightweight Prompt Generator and a Vision Mamba-based backbone for multimodal tracking, improving efficiency and cross-modal interaction over existing transformer-based methods.

Findings

Superior tracking accuracy on multiple benchmarks

Reduced computational complexity compared to transformer-based models

Effective multimodal feature fusion demonstrated

Abstract

Combining traditional RGB cameras with bio-inspired event cameras for robust object tracking has garnered increasing attention in recent years. However, most existing multimodal tracking algorithms depend heavily on high-complexity Vision Transformer architectures for feature extraction and fusion across modalities. This not only leads to substantial computational overhead but also limits the effectiveness of cross-modal interactions. In this paper, we propose an efficient RGB-Event object tracking framework based on the linear-complexity Vision Mamba network, termed Mamba-FETrack V2. Specifically, we first design a lightweight Prompt Generator that utilizes embedded features from each modality, together with a shared prompt pool, to dynamically generate modality-specific learnable prompt vectors. These prompts, along with the modality-specific embedded features, are then fed into a Vision Mamba-based FEMamba backbone, which facilitates prompt-guided feature extraction, cross-modal interaction, and fusion in a unified manner. Finally, the fused representations are passed to the tracking head for accurate target localization. Extensive experimental evaluations on multiple RGB-Event tracking benchmarks, including short-term COESOT dataset and long-term datasets, i.e., FE108 and FELT V2, demonstrate the superior performance and efficiency of the proposed tracking framework. The source code and pre-trained models will be released on https://github.com/Event-AHU/Mamba_FETrack