MeMix: Writing Less, Remembering More for Streaming 3D Reconstruction

TL;DR

MeMix is a training-free module that enhances streaming 3D reconstruction by selectively updating memory patches, reducing forgetting and improving accuracy without additional training or memory overhead.

Contribution

Introducing MeMix, a plug-and-play, training-free memory module that improves long-sequence streaming 3D reconstruction by mitigating forgetting with minimal memory overhead.

Findings

Reduces reconstruction error by 15.3% on average.

Maintains O(1) inference memory.

Effective across multiple benchmarks.

Abstract

Reconstruction is a fundamental task in 3D vision and a fundamental capability for spatial intelligence. Particularly, streaming 3D reconstruction is central to real-time spatial perception, yet existing recurrent online models often suffer from progressive degradation on long sequences due to state drift and forgetting, motivating inference-time remedies. We present MeMix, a training-free, plug-and-play module that improves streaming reconstruction by recasting the recurrent state into a Memory Mixture. MeMix partitions the state into multiple independent memory patches and updates only the least-aligned memory patches while exactly preserving others. This selective update mitigates catastrophic forgetting while retaining $O(1)$ inference memory, and requires no fine-tuning or additional learnable parameters, making it directly applicable to existing recurrent reconstruction models. Across standard benchmarks (ScanNet, 7-Scenes, KITTI, etc.), under identical backbones and inference settings, MeMix reduces reconstruction completeness error by 15.3% on average (up to 40.0%) across 300--500 frame streams on 7-Scenes. The code is available at https://dongjiacheng06.github.io/MeMix/