Learning Video Context as Interleaved Multimodal Sequences

TL;DR

MovieSeq is a multimodal language model that represents narrative videos as interleaved sequences of images, text, and audio, enabling comprehensive understanding across various video analysis tasks.

Contribution

The paper introduces MovieSeq, a novel approach that models videos as interleaved multimodal sequences and uses instruction-tuning to improve video understanding capabilities.

Findings

Effective across six datasets and five tasks

Improves performance in video classification and question-answering

Demonstrates the benefit of multimodal interleaved sequences

Abstract

Narrative videos, such as movies, pose significant challenges in video understanding due to their rich contexts (characters, dialogues, storylines) and diverse demands (identify who, relationship, and reason). In this paper, we introduce MovieSeq, a multimodal language model developed to address the wide range of challenges in understanding video contexts. Our core idea is to represent videos as interleaved multimodal sequences (including images, plots, videos, and subtitles), either by linking external knowledge databases or using offline models (such as whisper for subtitles). Through instruction-tuning, this approach empowers the language model to interact with videos using interleaved multimodal instructions. For example, instead of solely relying on video as input, we jointly provide character photos alongside their names and dialogues, allowing the model to associate these elements and generate more comprehensive responses. To demonstrate its effectiveness, we validate MovieSeq's performance on six datasets (LVU, MAD, Movienet, CMD, TVC, MovieQA) across five settings (video classification, audio description, video-text retrieval, video captioning, and video question-answering). The code will be public at https://github.com/showlab/MovieSeq.