CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanisms

TL;DR

CrossLMM introduces a dual cross-attention mechanism to efficiently process long video sequences in large multimodal models, significantly reducing computational costs while maintaining high performance.

Contribution

It proposes a novel dual cross-attention framework that decouples long video sequences from LMMs, enabling efficient token reduction and improved multimodal understanding.

Findings

Achieves comparable or better performance with fewer computational resources.

Reduces visual tokens significantly through pooling and cross-attention.

Maintains fine-grained informational fidelity in long video processing.

Abstract

The advent of Large Multimodal Models (LMMs) has significantly enhanced Large Language Models (LLMs) to process and interpret diverse data modalities (e.g., image and video). However, as input complexity increases, particularly with long video sequences, the number of required tokens has grown significantly, leading to quadratically computational costs. This has made the efficient compression of video tokens in LMMs, while maintaining performance integrity, a pressing research challenge. In this paper, we introduce CrossLMM, decoupling long video sequences from LMMs via a dual cross-attention mechanism, which substantially reduces visual token quantity with minimal performance degradation. Specifically, we first implement a significant token reduction from pretrained visual encoders through a pooling methodology. Then, within LLM layers, we employ a visual-to-visual cross-attention mechanism, wherein the pooled visual tokens function as queries against the original visual token set. This module enables more efficient token utilization while retaining fine-grained informational fidelity. In addition, we introduce a text-to-visual cross-attention mechanism, for which the text tokens are enhanced through interaction with the original visual tokens, enriching the visual comprehension of the text tokens. Comprehensive empirical evaluation demonstrates that our approach achieves comparable or superior performance across diverse video-based LMM benchmarks, despite utilizing substantially fewer computational resources.