Multimodal Instruction Tuning with Hybrid State Space Models

TL;DR

This paper introduces a hybrid transformer-MAMBA model that efficiently processes extremely long multimodal contexts exceeding 100,000 tokens, significantly improving inference speed and flexibility for high-resolution images and videos.

Contribution

The paper presents a novel hybrid transformer-MAMBA architecture that handles long multimodal contexts efficiently, surpassing existing models in both performance and inference speed.

Findings

Handles over 100k tokens in multimodal inputs

Achieves 4x inference efficiency gains for high-res images and videos

First to train on low-res data while inferring on high-res inputs

Abstract

Handling lengthy context is crucial for enhancing the recognition and understanding capabilities of multimodal large language models (MLLMs) in applications such as processing high-resolution images or high frame rate videos. The rise in image resolution and frame rate substantially increases computational demands due to the increased number of input tokens. This challenge is further exacerbated by the quadratic complexity with respect to sequence length of the self-attention mechanism. Most prior works either pre-train models with long contexts, overlooking the efficiency problem, or attempt to reduce the context length via downsampling (e.g., identify the key image patches or frames) to decrease the context length, which may result in information loss. To circumvent this issue while keeping the remarkable effectiveness of MLLMs, we propose a novel approach using a hybrid transformer-MAMBA model to efficiently handle long contexts in multimodal applications. Our multimodal model can effectively process long context input exceeding 100k tokens, outperforming existing models across various benchmarks. Remarkably, our model enhances inference efficiency for high-resolution images and high-frame-rate videos by about 4 times compared to current models, with efficiency gains increasing as image resolution or video frames rise. Furthermore, our model is the first to be trained on low-resolution images or low-frame-rate videos while being capable of inference on high-resolution images and high-frame-rate videos, offering flexibility for inference in diverse scenarios.