Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark

TL;DR

This paper introduces SceneBench, a new benchmark for long-video understanding that reveals current vision-language models struggle with scene-level context retention, and proposes Scene-RAG to improve long-term memory in models.

Contribution

The paper presents SceneBench, a scene-aware long-video benchmark, and Scene-RAG, a retrieval-augmented method to enhance long-context reasoning in vision-language models.

Findings

VLMs show significant accuracy drop on scene-level questions.

Scene-RAG improves VLM performance by +2.50%.

Current models struggle with long-range scene context retention.

Abstract

Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable progress, existing benchmarks mainly focus on either fine-grained perception or coarse summarization, offering limited insight into temporal understanding over long contexts. In this work, we define a scene as a coherent segment of a video in which both visual and semantic contexts remain consistent, aligning with human perception. This leads us to a key question: can current VLMs reason effectively over long, scene-level contexts? To answer this, we introduce a new benchmark, SceneBench, designed to provide scene-level challenges. Our evaluation reveals a sharp drop in accuracy when VLMs attempt to answer scene-level questions, indicating significant forgetting of long-range context. To further validate these findings, we propose Scene Retrieval-Augmented Generation (Scene-RAG), which constructs a dynamic scene memory by retrieving and integrating relevant context across scenes. This Scene-RAG improves VLM performance by +2.50%, confirming that current models still struggle with long-context retention. We hope SceneBench will encourage future research toward VLMs with more robust, human-like video comprehension.