Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understanding

TL;DR

This paper investigates the importance of explicit temporal modeling in multimodal large language models for video understanding, proposing a flexible encoder to compare it with implicit methods and demonstrating its effectiveness.

Contribution

Introduction of the Stackable Temporal Encoder (STE), a flexible module for explicit temporal modeling, enabling systematic comparison with implicit methods in video MLLMs.

Findings

Explicit temporal modeling significantly improves video understanding performance.

STE effectively balances token compression and temporal receptive fields.

Explicit modeling enhances temporal-specific understanding in videos.

Abstract

Applying Multimodal Large Language Models (MLLMs) to video understanding presents significant challenges due to the need to model temporal relations across frames. Existing approaches adopt either implicit temporal modeling, relying solely on the LLM decoder, or explicit temporal modeling, employing auxiliary temporal encoders. To investigate this debate between the two paradigms, we propose the Stackable Temporal Encoder (STE). STE enables flexible explicit temporal modeling with adjustable temporal receptive fields and token compression ratios. Using STE, we systematically compare implicit and explicit temporal modeling across dimensions such as overall performance, token compression effectiveness, and temporal-specific understanding. We also explore STE's design considerations and broader impacts as a plug-in module and in image modalities. Our findings emphasize the critical role of explicit temporal modeling, providing actionable insights to advance video MLLMs.