When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding

TL;DR

This paper introduces a novel unified paradigm for video coding that leverages multimodal large language models to enhance semantic and perceptual quality, enabling flexible encoding-decoding modes and efficient frame interpolation.

Contribution

It pioneers the integration of multimodal large language models into video coding, disentangling video content into modalities and enabling multiple reconstruction modes for improved quality.

Findings

TT2V mode achieves effective semantic reconstruction.

IT2V mode exhibits competitive perceptual consistency.

Proposed frame interpolation model ensures smooth motion cues.

Abstract

Existing codecs are designed to eliminate intrinsic redundancies to create a compact representation for compression. However, strong external priors from Multimodal Large Language Models (MLLMs) have not been explicitly explored in video compression. Herein, we introduce a unified paradigm for Cross-Modality Video Coding (CMVC), which is a pioneering approach to explore multimodality representation and video generative models in video coding. Specifically, on the encoder side, we disentangle a video into spatial content and motion components, which are subsequently transformed into distinct modalities to achieve very compact representation by leveraging MLLMs. During decoding, previously encoded components and video generation models are leveraged to create multiple encoding-decoding modes that optimize video reconstruction quality for specific decoding requirements, including Text-Text-to-Video (TT2V) mode to ensure high-quality semantic information and Image-Text-to-Video (IT2V) mode to achieve superb perceptual consistency. In addition, we propose an efficient frame interpolation model for IT2V mode via Low-Rank Adaption (LoRA) tuning to guarantee perceptual quality, which allows the generated motion cues to behave smoothly. Experiments on benchmarks indicate that TT2V achieves effective semantic reconstruction, while IT2V exhibits competitive perceptual consistency. These results highlight potential directions for future research in video coding.