MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition

TL;DR

MoMa introduces a novel adapter framework that enhances image foundation models for video recognition by fully modeling spatial-temporal dynamics efficiently, outperforming prior methods on multiple benchmarks.

Contribution

MoMa uniquely integrates Mamba's state space modeling into IFMs using SeqMod, enabling comprehensive spatial-temporal understanding with minimal computational overhead.

Findings

Achieves superior accuracy on video benchmarks.

Maintains computational efficiency compared to existing methods.

Effectively captures complex video dynamics.

Abstract

Video understanding is a complex challenge that requires effective modeling of spatial-temporal dynamics. With the success of image foundation models (IFMs) in image understanding, recent approaches have explored parameter-efficient fine-tuning (PEFT) to adapt IFMs for video. However, most of these methods tend to process spatial and temporal information separately, which may fail to capture the full intricacy of video dynamics. In this paper, we propose MoMa, an efficient adapter framework that achieves full spatial-temporal modeling by integrating Mamba's selective state space modeling into IFMs. We propose a novel SeqMod operation to inject spatial-temporal information into pre-trained IFMs, without disrupting their original features. By incorporating SeqMod into a Divide-and-Modulate architecture, MoMa enhances video understanding while maintaining computational efficiency. Extensive experiments on multiple video benchmarks demonstrate the effectiveness of MoMa, achieving superior performance with reduced computational cost.