From Image to Video, what do we need in multimodal LLMs?

TL;DR

This paper presents RED-VILLM, a resource-efficient pipeline that leverages existing Image LLMs to develop robust Video LLMs, reducing data and training requirements while enhancing performance.

Contribution

It introduces a novel temporal adaptation structure and a pipeline that effectively builds Video LLMs from Image LLMs, including the first Chinese Video LLM.

Findings

Video LLMs outperform conventional models with less data.

The approach reduces training resources significantly.

The pipeline is scalable and cost-effective.

Abstract

Covering from Image LLMs to the more complex Video LLMs, the Multimodal Large Language Models (MLLMs) have demonstrated profound capabilities in comprehending cross-modal information as numerous studies have illustrated. Previous methods delve into designing comprehensive Video LLMs through integrating video foundation models with primitive LLMs. Despite its effectiveness, such paradigm renders Video LLM's structure verbose and typically requires substantial video data for pre-training. Crucially, it neglects leveraging the foundational contributions of ready-made Image LLMs. In this paper, we introduce RED-VILLM, a Resource-Efficient Development pipeline which builds robust Video LLMs through leveraging the prior knowledge of Image LLMs. Specifically, since a video is naturally a combination of images along the temporal dimension, we devise a temporal adaptation plug-and-play structure, endowing the backbone Image LLM with the capability to grasp temporal information. Moreover, through applying this pipeline, we achieve the first Video LLM within the Chinese-speaking community. Extensive experiments demonstrate that Video LLMs developed through our approach surpass conventional Video LLMs, requiring minimal instructional data and training resources. Our approach highlights the potential for a more cost-effective and scalable advancement in multimodal models.