Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Manner

TL;DR

This paper proposes a training-free interpolation method for Video-LLMs, enabling longer video sequence processing by rearranging video tokens and extending the LLM context window without additional training.

Contribution

It introduces a novel training-free interpolation technique that overcomes fixed encoder and limited context length constraints in Video-LLMs.

Findings

Enables processing of longer videos without retraining

Rearranges video tokens to bypass fixed encoder limitations

Extends LLM context window training-free

Abstract

Advancements in Large Language Models (LLMs) inspire various strategies for integrating video modalities. A key approach is Video-LLMs, which incorporate an optimizable interface linking sophisticated video encoders to LLMs. However, due to computation and data limitations, these Video-LLMs are typically pre-trained to process only short videos, limiting their broader application for understanding longer video content. Additionally, fine-tuning Video-LLMs to handle longer videos is cost-prohibitive. Consequently, it becomes essential to explore the interpolation of Video-LLMs under a completely training-free setting. In this paper, we first identify the primary challenges in interpolating Video-LLMs: (1) the video encoder and modality alignment projector are fixed, preventing the integration of additional frames into Video-LLMs, and (2) the LLM backbone is limited in its content length capabilities, which complicates the processing of an increased number of video tokens. To address these challenges, we propose a specific INTerPolation method for Video-LLMs (INTP-Video-LLMs). We introduce an alternative video token rearrangement technique that circumvents limitations imposed by the fixed video encoder and alignment projector. Furthermore, we introduce a training-free LLM context window extension method to enable Video-LLMs to understand a correspondingly increased number of visual tokens.