Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval

TL;DR

This paper introduces Context-as-Memory, a novel approach for long video generation that effectively utilizes historical context through memory retrieval based on camera pose overlap, improving scene consistency and generalization.

Contribution

It presents a simple memory storage and conditioning method combined with a memory retrieval module based on FOV overlap, enhancing scene consistency in long video generation.

Findings

Outperforms state-of-the-art methods in memory capabilities

Generalizes well to open-domain scenarios

Reduces computational overhead with effective memory retrieval

Abstract

Recent advances in interactive video generation have shown promising results, yet existing approaches struggle with scene-consistent memory capabilities in long video generation due to limited use of historical context. In this work, we propose Context-as-Memory, which utilizes historical context as memory for video generation. It includes two simple yet effective designs: (1) storing context in frame format without additional post-processing; (2) conditioning by concatenating context and frames to be predicted along the frame dimension at the input, requiring no external control modules. Furthermore, considering the enormous computational overhead of incorporating all historical context, we propose the Memory Retrieval module to select truly relevant context frames by determining FOV (Field of View) overlap between camera poses, which significantly reduces the number of candidate frames without substantial information loss. Experiments demonstrate that Context-as-Memory achieves superior memory capabilities in interactive long video generation compared to SOTAs, even generalizing effectively to open-domain scenarios not seen during training. The link of our project page is https://context-as-memory.github.io/.