Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generation

TL;DR

This paper introduces Memorize-and-Generate (MAG), a novel framework for long-term consistent real-time video generation that effectively balances memory efficiency and scene coherence by separating memory compression from frame synthesis.

Contribution

MAG decouples memory compression and frame generation, enabling long-term scene consistency without excessive memory use, and introduces MAG-Bench for rigorous evaluation of memory retention.

Findings

MAG outperforms existing models in scene consistency over long videos.

MAG maintains competitive performance on standard benchmarks.

MAG-Bench effectively evaluates historical memory retention.

Abstract

Frame-level autoregressive (frame-AR) models have achieved significant progress, enabling real-time video generation comparable to bidirectional diffusion models and serving as a foundation for interactive world models and game engines. However, current approaches in long video generation typically rely on window attention, which naively discards historical context outside the window, leading to catastrophic forgetting and scene inconsistency; conversely, retaining full history incurs prohibitive memory costs. To address this trade-off, we propose Memorize-and-Generate (MAG), a framework that decouples memory compression and frame generation into distinct tasks. Specifically, we train a memory model to compress historical information into a compact KV cache, and a separate generator model to synthesize subsequent frames utilizing this compressed representation. Furthermore, we introduce MAG-Bench to strictly evaluate historical memory retention. Extensive experiments demonstrate that MAG achieves superior historical scene consistency while maintaining competitive performance on standard video generation benchmarks.