TrajRAG: Retrieving Geometric-Semantic Experience for Zero-Shot Object Navigation

TL;DR

TrajRAG is a retrieval-augmented framework that enhances zero-shot object navigation by accumulating and utilizing episodic geometric-semantic experiences through a hierarchical, topological trajectory representation.

Contribution

It introduces a novel topological-polar trajectory representation and a hierarchical chunking structure for lifelong experience accumulation and retrieval in object navigation.

Findings

TrajRAG improves zero-shot ObjectNav performance on MP3D, HM3D-v1, and HM3D-v2 datasets.

Hierarchical chunking organizes similar trajectories into summaries for efficient retrieval.

Retrieving relevant past experiences guides better waypoint selection during navigation.

Abstract

Existing zero-shot Object Goal Navigation (ObjectNav) methods often exploit commonsense knowledge from large language or vision-language models to guide navigation. However, such knowledge arises from internet-scale text rather than embodied 3D experience, and episodic observations collected during navigation are typically discarded, preventing the accumulation of lifelong experience. To this end, we propose Trajectory RAG (TrajRAG), a retrieval-augmented generation framework that enhances large-model reasoning by retrieving geometric-semantic experiences. TrajRAG incrementally accumulates episodic observations from past navigation episodes. To structure these observations, we propose a topological-polar (topo-polar) trajectory representation that compactly encodes spatial layouts and semantic contexts, effectively removing redundancies in raw episodic observations. A hierarchical chunking structure further organizes similar topo-polar trajectories into unified summaries, enabling coarse-to-fine retrieval. During navigation, candidate frontiers generate multiple trajectory hypotheses that query TrajRAG for similar past trajectories, guiding large-model reasoning for waypoint selection. New experiences are continually consolidated into TrajRAG, enabling the accumulation of lifelong navigation experience. Experiments on MP3D, HM3D-v1, and HM3D-v2 show that TrajRAG effectively retrieves relevant geometric-semantic experiences and improves zero-shot ObjectNav performance.