Language-enhanced RNR-Map: Querying Renderable Neural Radiance Field maps with natural language

TL;DR

Le-RNR-Map enhances neural radiance maps with CLIP-based language embeddings, enabling natural language queries for visual navigation and localization without extra labels, demonstrated through effective single and multi-object searches.

Contribution

This work introduces a language-enhanced neural radiance map that integrates CLIP embeddings, allowing natural language queries for navigation and localization in visual environments.

Findings

Effective natural language search in visual maps.

High accuracy in image navigation and localization.

Compatibility with large language models for query resolution.

Abstract

We present Le-RNR-Map, a Language-enhanced Renderable Neural Radiance map for Visual Navigation with natural language query prompts. The recently proposed RNR-Map employs a grid structure comprising latent codes positioned at each pixel. These latent codes, which are derived from image observation, enable: i) image rendering given a camera pose, since they are converted to Neural Radiance Field; ii) image navigation and localization with astonishing accuracy. On top of this, we enhance RNR-Map with CLIP-based embedding latent codes, allowing natural language search without additional label data. We evaluate the effectiveness of this map in single and multi-object searches. We also investigate its compatibility with a Large Language Model as an "affordance query resolver". Code and videos are available at https://intelligolabs.github.io/Le-RNR-Map/