AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories

TL;DR

AnchorWeave introduces a novel memory-augmented framework for video generation that uses multiple local geometric memories to improve spatial consistency over long horizons, addressing cross-view misalignment issues.

Contribution

The paper proposes AnchorWeave, which replaces global 3D scene reconstruction with multiple local memories and a multi-anchor weaving controller for better long-term spatial consistency.

Findings

Significantly improves long-term scene consistency.

Maintains high visual quality in generated videos.

Validates effectiveness through extensive experiments.

Abstract

Maintaining spatial world consistency over long horizons remains a central challenge for camera-controllable video generation. Existing memory-based approaches often condition generation on globally reconstructed 3D scenes by rendering anchor videos from the reconstructed geometry in the history. However, reconstructing a global 3D scene from multiple views inevitably introduces cross-view misalignment, as pose and depth estimation errors cause the same surfaces to be reconstructed at slightly different 3D locations across views. When fused, these inconsistencies accumulate into noisy geometry that contaminates the conditioning signals and degrades generation quality. We introduce AnchorWeave, a memory-augmented video generation framework that replaces a single misaligned global memory with multiple clean local geometric memories and learns to reconcile their cross-view inconsistencies. To this end, AnchorWeave performs coverage-driven local memory retrieval aligned with the target trajectory and integrates the selected local memories through a multi-anchor weaving controller during generation. Extensive experiments demonstrate that AnchorWeave significantly improves long-term scene consistency while maintaining strong visual quality, with ablation and analysis studies further validating the effectiveness of local geometric conditioning, multi-anchor control, and coverage-driven retrieval.