ABMAMBA: Multimodal Large Language Model with Aligned Hierarchical Bidirectional Scan for Efficient Video Captioning

TL;DR

ABMamba is a scalable multimodal large language model for video captioning that uses a novel hierarchical scan to process long videos efficiently with linear complexity.

Contribution

It introduces ABMamba, a fully open MLLM with a hierarchical bidirectional scan and linear complexity, enabling efficient processing of long video sequences.

Findings

Achieves three times higher throughput than existing models.

Demonstrates competitive performance on VATEX and MSR-VTT benchmarks.

Replaces quadratic attention with a hierarchical scan for efficiency.

Abstract

In this study, we focus on video captioning by fully open multimodal large language models (MLLMs). The comprehension of visual sequences is challenging because of their intricate temporal dependencies and substantial sequence length. The core attention mechanisms of existing Transformer-based approaches scale quadratically with the sequence length, making them computationally prohibitive. To address these limitations, we propose Aligned Hierarchical Bidirectional Scan Mamba (ABMamba), a fully open MLLM with linear computational complexity that enables the scalable processing of video sequences. ABMamba extends Deep State Space Models as its language backbone, replacing the costly quadratic attention mechanisms, and employs a novel Aligned Hierarchical Bidirectional Scan module that processes videos across multiple temporal resolutions. On standard video captioning benchmarks such as VATEX and MSR-VTT, ABMamba demonstrates competitive performance compared to typical MLLMs while achieving approximately three times higher throughput.