VideoChat: Chat-Centric Video Understanding

TL;DR

VideoChat is an innovative end-to-end system that combines video foundation models and large language models through a learnable interface, enabling advanced spatiotemporal reasoning, event localization, and causal inference in videos.

Contribution

The paper introduces a novel chat-centric video understanding system and a comprehensive video instruction dataset, advancing the integration of vision and language models for video analysis.

Findings

Preliminary experiments show promising capabilities in video understanding tasks.

The dataset emphasizes spatiotemporal reasoning and causal relationships.

The system demonstrates potential across diverse video applications.

Abstract

In this paper, we initiate an attempt of developing an end-to-end chat-centric video understanding system, coined as VideoChat. It integrates video foundation models and large language models via a learnable neural interface, excelling in spatiotemporal reasoning, event localization, and causal relationship inference. To instructively tune this system, we build a video-centric instruction dataset, composed of thousands of videos associated with detailed descriptions and conversations. This dataset emphasizes spatiotemporal reasoning and captures causal relationships, providing a valuable asset for training our chat-centric video understanding system. Preliminary qualitative experiments demonstrate the potential of our system across a broad spectrum of video applications, which could serve as a simple prototype system for future research on chat-centric video understanding. Access our code and data at https://github.com/OpenGVLab/Ask-Anything