Video Event Reasoning and Prediction by Fusing World Knowledge from LLMs with Vision Foundation Models

TL;DR

This paper introduces a novel framework that combines vision foundation models with large language models to enhance high-level video understanding, reasoning, and future prediction capabilities.

Contribution

It presents a new fusion architecture inspired by Q-Former, enabling effective grounding of visual features in language for advanced reasoning tasks.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Demonstrates strong zero-shot generalization to unseen tasks.

Validates the importance of each architectural component through ablation studies.

Abstract

Current video understanding models excel at recognizing "what" is happening but fall short in high-level cognitive tasks like causal reasoning and future prediction, a limitation rooted in their lack of commonsense world knowledge. To bridge this cognitive gap, we propose a novel framework that synergistically fuses a powerful Vision Foundation Model (VFM) for deep visual perception with a Large Language Model (LLM) serving as a knowledge-driven reasoning core. Our key technical innovation is a sophisticated fusion module, inspired by the Q-Former architecture, which distills complex spatiotemporal and object-centric visual features into a concise, language-aligned representation. This enables the LLM to effectively ground its inferential processes in direct visual evidence. The model is trained via a two-stage strategy, beginning with large-scale alignment pre-training on video-text data, followed by targeted instruction fine-tuning on a curated dataset designed to elicit advanced reasoning and prediction skills. Extensive experiments demonstrate that our model achieves state-of-the-art performance on multiple challenging benchmarks. Notably, it exhibits remarkable zero-shot generalization to unseen reasoning tasks, and our in-depth ablation studies validate the critical contribution of each architectural component. This work pushes the boundary of machine perception from simple recognition towards genuine cognitive understanding, paving the way for more intelligent and capable AI systems in robotics, human-computer interaction, and beyond.