Memory Over Maps: 3D Object Localization Without Reconstruction

TL;DR

This paper introduces a map-free method for 3D object localization that relies on storing posed RGB-D keyframes, enabling faster scene indexing and effective robot navigation without dense 3D reconstruction.

Contribution

It proposes a lightweight visual memory approach that bypasses traditional 3D scene reconstruction, reducing preprocessing time and storage while maintaining strong localization performance.

Findings

Scene indexing is over 100 times faster than reconstruction-based methods.

The approach achieves competitive object localization accuracy without dense 3D models.

The method performs well on downstream object-goal navigation benchmarks.

Abstract

Target localization is a prerequisite for embodied tasks such as navigation and manipulation. Conventional approaches rely on constructing explicit 3D scene representations to enable target localization, such as point clouds, voxel grids, or scene graphs. While effective, these pipelines incur substantial mapping time, storage overhead, and scalability limitations. Recent advances in vision-language models suggest that rich semantic reasoning can be performed directly on 2D observations, raising a fundamental question: is a complete 3D scene reconstruction necessary for object localization? In this work, we revisit object localization and propose a map-free pipeline that stores only posed RGB-D keyframes as a lightweight visual memory--without constructing any global 3D representation of the scene. At query time, our method retrieves candidate views, re-ranks them with a vision-language model, and constructs a sparse, on-demand 3D estimate of the queried target through depth backprojection and multi-view fusion. Compared to reconstruction-based pipelines, this design drastically reduces preprocessing cost, enabling scene indexing that is over two orders of magnitude faster to build while using substantially less storage. We further validate the localized targets on downstream object-goal navigation tasks. Despite requiring no task-specific training, our approach achieves strong performance across multiple benchmarks, demonstrating that direct reasoning over image-based scene memory can effectively replace dense 3D reconstruction for object-centric robot navigation. Project page: https://ruizhou-cn.github.io/memory-over-maps/