VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Management

TL;DR

VideoMem introduces an adaptive memory management framework for ultra-long video understanding, enabling models to effectively retain critical information over long sequences and outperform existing methods on various benchmarks.

Contribution

The paper presents VideoMem, a novel approach that models long video understanding as a sequential generation task with dynamic memory updates and a new training algorithm, PRPO.

Findings

Outperforms existing models on multiple ultra-long video benchmarks

Efficiently retains critical long-term information with adaptive memory

Accelerates training convergence through novel reward strategies

Abstract

Ultra long video understanding remains an open challenge, as existing vision language models (VLMs) falter on such content due to limited context length and inefficient long term memory retention. To address this, recent works have attempted to construct external knowledge bases and corresponding retrieval agumented generation (RAG) systems, yet these incur enormous storage and computational overhead. In this paper, we propose VideoMem, a novel framework that pioneers models long video understanding as a sequential generation task via adaptive memory management. Specifically, VideoMem dynamically updates a global memory buffer, which adaptively retains critical information while discarding redundant content across the video timeline. To efficiently train VLMs for such long-term tasks, VideoMem integrates the Progressive Grouped Relative Policy Optimization (PRPO) algorithm, equipped with two core modules: Progressive State Propagation (PSP) adaptively retains valid current states, propagates them to the next rollout step, and gradually narrows the model exploration space. Temporal Cascading Reward (TCR) further alleviates reward sparsity, improving sample utilization and accelerating convergence. Extensive experiments demonstrate that VideoMem significantly outperforms existing open-source models across diverse benchmarks for ultra-long video understanding tasks.