I3DM: Implicit 3D-aware Memory Retrieval and Injection for Consistent Video Scene Generation

TL;DR

I3DM introduces an implicit 3D-aware memory mechanism for consistent video scene generation that avoids explicit 3D reconstruction and improves revisit consistency, fidelity, and camera control.

Contribution

The paper proposes a novel implicit 3D-aware memory retrieval and injection method that enhances long-term scene consistency without explicit 3D modeling.

Findings

Outperforms state-of-the-art methods in revisit consistency

Achieves higher generation fidelity

Provides more accurate camera control

Abstract

Despite remarkable progress in video generation, maintaining long-term scene consistency upon revisiting previously explored areas remains challenging. Existing solutions rely either on explicitly constructing 3D geometry, which suffers from error accumulation and scale ambiguity, or on naive camera Field-of-View (FoV) retrieval, which typically fails under complex occlusions. To overcome these limitations, we propose I3DM, a novel implicit 3D-aware memory mechanism for consistent video scene generation that bypasses explicit 3D reconstruction. At the core of our approach is a 3D-aware memory retrieval strategy, which leverages the intermediate features of a pre-trained Feed-Forward Novel View Synthesis (FF-NVS) model to score view relevance, enabling robust retrieval even in highly occluded scenarios. Furthermore, to fully utilize the retrieved historical frames, we introduce a 3D-aligned memory injection module. This module implicitly warps historical content to the target view and adaptively conditions the generation on reliable warping regions, leading to improved revisit consistency and accurate camera control. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches, achieving superior revisit consistency, generation fidelity, and camera control precision.