Mem4D: Decoupling Static and Dynamic Memory for Dynamic Scene Reconstruction

TL;DR

Mem4D introduces a dual-memory framework that separately models static and dynamic scene components, enabling accurate, high-fidelity, and consistent reconstruction of dynamic scenes from monocular videos.

Contribution

The paper proposes a novel dual-memory architecture that decouples static and dynamic scene modeling, overcoming the memory demand dilemma in dynamic scene reconstruction.

Findings

Achieves state-of-the-art reconstruction quality on benchmarks.

Maintains static scene consistency while capturing dynamic details.

Operates efficiently with a dual-memory approach.

Abstract

Reconstructing dense geometry for dynamic scenes from a monocular video is a critical yet challenging task. Recent memory-based methods enable efficient online reconstruction, but they fundamentally suffer from a Memory Demand Dilemma: The memory representation faces an inherent conflict between the long-term stability required for static structures and the rapid, high-fidelity detail retention needed for dynamic motion. This conflict forces existing methods into a compromise, leading to either geometric drift in static structures or blurred, inaccurate reconstructions of dynamic objects. To address this dilemma, we propose Mem4D, a novel framework that decouples the modeling of static geometry and dynamic motion. Guided by this insight, we design a dual-memory architecture: 1) The Transient Dynamics Memory (TDM) focuses on capturing high-frequency motion details from recent frames, enabling accurate and fine-grained modeling of dynamic content; 2) The Persistent Structure Memory (PSM) compresses and preserves long-term spatial information, ensuring global consistency and drift-free reconstruction for static elements. By alternating queries to these specialized memories, Mem4D simultaneously maintains static geometry with global consistency and reconstructs dynamic elements with high fidelity. Experiments on challenging benchmarks demonstrate that our method achieves state-of-the-art or competitive performance while maintaining high efficiency. Codes will be publicly available.