Mem3R: Streaming 3D Reconstruction with Hybrid Memory via Test-Time Training

TL;DR

Mem3R introduces a hybrid memory streaming model for 3D reconstruction that enhances long-sequence accuracy and efficiency using test-time training and fixed-size states.

Contribution

It proposes a novel hybrid memory architecture with implicit fast-weight memory and explicit token-based states, improving long-sequence performance and reducing model size.

Findings

Significantly improves long-sequence performance over CUT3R.

Reduces model size from 793M to 644M parameters.

Decreases Absolute Trajectory Error by up to 39% on long sequences.

Abstract

Streaming 3D perception is well suited to robotics and augmented reality, where long visual streams must be processed efficiently and consistently. Recent recurrent models offer a promising solution by maintaining fixed-size states and enabling linear-time inference, but they often suffer from drift accumulation and temporal forgetting over long sequences due to the limited capacity of compressed latent memories. We propose Mem3R, a streaming 3D reconstruction model with a hybrid memory design that decouples camera tracking from geometric mapping to improve temporal consistency over long sequences. For camera tracking, Mem3R employs an implicit fast-weight memory implemented as a lightweight Multi-Layer Perceptron updated via Test-Time Training. For geometric mapping, Mem3R maintains an explicit token-based fixed-size state. Compared with CUT3R, this design not only significantly improves long-sequence performance but also reduces the model size from 793M to 644M parameters. Mem3R supports existing improved plug-and-play state update strategies developed for CUT3R. Specifically, integrating it with TTT3R decreases Absolute Trajectory Error by up to 39% over the base implementation on 500 to 1000 frame sequences. The resulting improvements also extend to other downstream tasks, including video depth estimation and 3D reconstruction, while preserving constant GPU memory usage and comparable inference throughput. Project page: https://lck666666.github.io/Mem3R/