DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs

TL;DR

DisCo introduces a novel visual encapsulation method for video MLLMs that enhances semantic distinctness and temporal coherence of visual tokens, leading to improved performance on video understanding benchmarks.

Contribution

DisCo presents a new approach combining a Visual Concept Discriminator and a Temporal Focus Calibrator to improve visual token quality in video MLLMs.

Findings

Outperforms previous state-of-the-art methods on multiple benchmarks.

Achieves higher token efficiency by reducing semantic indistinctness.

Enhances temporal coherence of visual tokens across video frames.

Abstract

In video Multimodal Large Language Models (video MLLMs), the visual encapsulation process plays a pivotal role in converting video contents into representative tokens for LLM input. While linear projectors are widely employed for encapsulation, they introduce semantic indistinctness and temporal incoherence when applied to videos. Conversely, the structure of resamplers shows promise in tackling these challenges, but an effective solution remains unexplored. Drawing inspiration from resampler structures, we introduce DisCo, a novel visual encapsulation method designed to yield semantically distinct and temporally coherent visual tokens for video MLLMs. DisCo integrates two key components: (1) A Visual Concept Discriminator (VCD) module, assigning unique semantics for visual tokens by associating them in pair with discriminative concepts in the video. (2) A Temporal Focus Calibrator (TFC) module, ensuring consistent temporal focus of visual tokens to video elements across every video frame. Through extensive experiments on multiple video MLLM frameworks, we demonstrate that DisCo remarkably outperforms previous state-of-the-art methods across a variety of video understanding benchmarks, while also achieving higher token efficiency thanks to the reduction of semantic indistinctness. The code: https://github.com/ZJHTerry18/DisCo.