TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations

TL;DR

This paper introduces TC-LLaVA, a novel approach that enhances large language models for video understanding by incorporating temporal information through improved attention mechanisms, achieving state-of-the-art results.

Contribution

The paper proposes two new attention strategies to incorporate temporal information into LLMs, specifically enhancing RoPE with Temporal-Aware Dual RoPE and attention masks with Frame-wise Block Causal Attention.

Findings

Achieves state-of-the-art performance on video understanding benchmarks.

Effectively models temporal information without modifying the core LLM architecture.

Improves inter-frame visual token interactions while maintaining causal inference.

Abstract

Multimodal Large Language Models (MLLMs) have significantly improved performance across various image-language applications. Recently, there has been a growing interest in adapting image pre-trained MLLMs for video-related tasks. However, most efforts concentrate on enhancing the vision encoder and projector components, while the core part, Large Language Models (LLMs), remains comparatively under-explored. In this paper, we propose two strategies to enhance the model's capability in video understanding tasks by improving inter-layer attention computation in LLMs. Specifically, the first approach focuses on the enhancement of Rotary Position Embedding (RoPE) with Temporal-Aware Dual RoPE, which introduces temporal position information to strengthen the MLLM's temporal modeling capabilities while preserving the relative position relationships of both visual and text tokens. The second approach involves enhancing the Attention Mask with the Frame-wise Block Causal Attention Mask, a simple yet effective method that broadens visual token interactions within and across video frames while maintaining the causal inference mechanism. Based on these proposed methods, we adapt LLaVA for video understanding tasks, naming it Temporal-Considered LLaVA (TC-LLaVA). Our TC-LLaVA achieves new state-of-the-art performance across various video understanding benchmarks with only supervised fine-tuning (SFT) on video-related datasets.