TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding

TL;DR

TimeViper is a hybrid Mamba-Transformer model that efficiently processes long videos exceeding 10,000 frames by combining state-space models with attention mechanisms and introducing a token transfer module.

Contribution

The paper presents a novel hybrid Mamba-Transformer architecture with a token transfer module for efficient long video understanding, extending capabilities beyond existing models.

Findings

TimeViper processes hour-long videos with over 10,000 frames.

It competes with state-of-the-art models on multiple benchmarks.

Analysis of attention behaviors offers new interpretability insights.

Abstract

We introduce TimeViper, a hybrid vision-language model designed to tackle challenges of long video understanding. Processing long videos demands both an efficient model architecture and an effective mechanism for handling extended temporal contexts. To this end, TimeViper adopts a hybrid Mamba-Transformer backbone that combines the efficiency of state-space models with the expressivity of attention mechanisms. Through this hybrid design, we reveal the vision-to-text information aggregation phenomenon, where information progressively flows from vision tokens to text tokens across increasing LLM depth, resulting in severe vision token redundancy. Motivated by this observation, we propose TransV, a token information transfer module that transfers and compresses vision tokens into instruction tokens while maintaining multimodal understanding capabilities. This design enables TimeViper to process hour-long videos exceeding 10,000 frames. Extensive experiments across multiple benchmarks demonstrate that TimeViper competes with state-of-the-art models while extending frame numbers. We further analyze attention behaviors of both Mamba and Transformer layers, offering new insights into hybrid model interpretability. This work represents an initial step towards developing, interpreting, and compressing hybrid Mamba-Transformer architectures.