Enhancing Temporal Modeling of Video LLMs via Time Gating

TL;DR

This paper introduces TG-Vid, a novel Video LLM with a Time Gating module that significantly improves temporal understanding in videos, leading to better performance on temporal-sensitive benchmarks.

Contribution

The paper proposes a new Time Gating module for Video LLMs, enhancing their ability to model temporal information effectively.

Findings

TG-Vid outperforms existing Video LLMs on temporal benchmarks.

Ablation studies confirm the effectiveness of the Time Gating module.

The model achieves robust temporal understanding in videos.

Abstract

Video Large Language Models (Video LLMs) have achieved impressive performance on video-and-language tasks, such as video question answering. However, most existing Video LLMs neglect temporal information in video data, leading to struggles with temporal-aware video understanding. To address this gap, we propose a Time Gating Video LLM (TG-Vid) designed to enhance temporal modeling through a novel Time Gating module (TG). The TG module employs a time gating mechanism on its sub-modules, comprising gating spatial attention, gating temporal attention, and gating MLP. This architecture enables our model to achieve a robust understanding of temporal information within videos. Extensive evaluation of temporal-sensitive video benchmarks (i.e., MVBench, TempCompass, and NExT-QA) demonstrates that our TG-Vid model significantly outperforms the existing Video LLMs. Further, comprehensive ablation studies validate that the performance gains are attributed to the designs of our TG module. Our code is available at https://github.com/LaVi-Lab/TG-Vid.