RenderMem: Rendering as Spatial Memory Retrieval

TL;DR

RenderMem introduces a novel spatial memory system that uses rendering as an interface to 3D scene representations, enabling embodied agents to perform viewpoint-dependent reasoning about visibility and occlusion.

Contribution

It presents RenderMem, a new framework that links 3D scene representations with rendering to improve spatial reasoning in embodied agents.

Findings

Improves accuracy on visibility and occlusion queries

Compatible with existing vision-language models

Enhances reasoning about line-of-sight and occlusion

Abstract

Embodied reasoning is inherently viewpoint-dependent: what is visible, occluded, or reachable depends critically on where the agent stands. However, existing spatial memory systems for embodied agents typically store either multi-view observations or object-centric abstractions, making it difficult to perform reasoning with explicit geometric grounding. We introduce RenderMem, a spatial memory framework that treats rendering as the interface between 3D world representations and spatial reasoning. Instead of storing fixed observations, RenderMem maintains a 3D scene representation and generates query-conditioned visual evidence by rendering the scene from viewpoints implied by the query. This enables embodied agents to reason directly about line-of-sight, visibility, and occlusion from arbitrary perspectives. RenderMem is fully compatible with existing vision-language models and requires no modification to standard architectures. Experiments in the AI2-THOR environment show consistent improvements on viewpoint-dependent visibility and occlusion queries over prior memory baselines.