Renderable Neural Radiance Map for Visual Navigation

TL;DR

This paper introduces the RNR-Map, a novel neural radiance map for visual navigation that encodes environmental visual information for improved localization and navigation, demonstrating superior performance in various scenarios.

Contribution

The paper presents the RNR-Map, a new renderable neural radiance map that effectively encodes visual environment data for navigation tasks, with frameworks for localization and navigation.

Findings

RNR-Map-based localization achieves fast, accurate results with robustness to environmental changes.

The navigation framework outperforms existing methods, especially in challenging scenarios.

Achieved 65.7% success rate in curved scenarios, surpassing previous state-of-the-art by 18.6%.

Abstract

We propose a novel type of map for visual navigation, a renderable neural radiance map (RNR-Map), which is designed to contain the overall visual information of a 3D environment. The RNR-Map has a grid form and consists of latent codes at each pixel. These latent codes are embedded from image observations, and can be converted to the neural radiance field which enables image rendering given a camera pose. The recorded latent codes implicitly contain visual information about the environment, which makes the RNR-Map visually descriptive. This visual information in RNR-Map can be a useful guideline for visual localization and navigation. We develop localization and navigation frameworks that can effectively utilize the RNR-Map. We evaluate the proposed frameworks on camera tracking, visual localization, and image-goal navigation. Experimental results show that the RNR-Map-based localization framework can find the target location based on a single query image with fast speed and competitive accuracy compared to other baselines. Also, this localization framework is robust to environmental changes, and even finds the most visually similar places when a query image from a different environment is given. The proposed navigation framework outperforms the existing image-goal navigation methods in difficult scenarios, under odometry and actuation noises. The navigation framework shows 65.7% success rate in curved scenarios of the NRNS dataset, which is an improvement of 18.6% over the current state-of-the-art. Project page: https://rllab-snu.github.io/projects/RNR-Map/

Tables

Table 1. Table 1 : Image-goal Navigation Result . SR: Success Rate. SPL: Success weighted by Path Length.

	Stragint								Curved
	Easy		Medium		Hard		Overall		Easy		Medium		Hard		Overall
	SR	SPL	SR	SPL	SR	SPL	SR	SPL	SR	SPL	SR	SPL	SR	SPL	SR	SPL
BC + RNN	39.4	27.9	25.7	15.9	13.3	9.0	26.1	17.6	26.4	12.6	20.3	10.5	8.4	4.8	18.4	9.3
DDPPO [50]	43.2	38.5	36.4	34.8	7.4	7.2	29.0	26.8	22.2	16.5	20.7	18.5	4.2	3.7	15.7	12.9
ANS + Target Pred [10]	68.8	55.1	54.0	30.3	42.4	22.9	55.1	36.1	48.0	21.0	46.0	20.5	31.3	14.6	41.8	18.7
NRNS [23]	64.1	55.4	47.9	39.5	25.2	18.1	45.7	37.7	27.3	10.6	23.1	10.4	10.5	5.6	20.3	8.8
ZSEL [2]	-	-	-	-	-	-	-	-	41.0	28.2	27.3	18.6	9.3	6.0	25.9	17.6
OVRL [52]	53.6	34.7	48.6	33.3	32.5	21.9	44.9	30.0	53.6	31.8	47.6	30.2	35.6	22.0	45.6	28.0
NRNS + SLING [49]	85.3	74.4	66.8	49.3	41.1	28.8	64.4	50.8	58.6	16.1	47.6	16.8	24.9	10.1	43.7	14.3
OVRL + SLING [49]	71.2	54.1	60.3	44.4	43.0	29.1	58.2	42.5	68.4	47.0	57.7	39.8	40.2	25.5	55.4	37.4
RNR-Map (ours)	76.4	55.3	73.6	46.1	54.6	30.2	68.2	43.9	75.3	52.5	70.9	42.3	51.0	27.4	65.7	40.8

Table 2. Table 2 : Ablation study . The values are the average of the straight and curved scenarios.

			Easy		Medium		Hard		Overall
Noise	$F_{loc}$	$F_{track}$	SR	SPL	SR	SPL	SR	SPL	SR	SPL
✘	✘	-	60.3	41.0	57.7	36.8	39.9	23.4	52.6	33.7
✘	✔	-	72.8	61.1	67.7	48.9	48.8	31.1	63.1	47.0
✔	✔	✘	74.6	53.6	64.0	40.9	40.8	23.6	59.8	39.4
✔	✔	✔	75.3	53.9	72.2	44.2	52.8	28.9	66.9	42.3

Table 3. Table 3 : Runtime analysis of RNR-Map.

Mapping $F_{\mathrm{reg}}$	Tracking $F_{\mathrm{track}}$	Localization $F_{\mathrm{loc}}$	Rendering $F_{\mathrm{dec}}$
91.9 Hz (10.9 ms)	5.0 Hz (200 ms)	56.8 Hz (17.6 ms)	14.7 Hz (68.0 ms)

Table 4. Table 4 : Localization Results. (a) Camera Tracking. ATE: Average Trajectory Error. The inference time is the average time of mapping and tracking time for a single frame. (b) Image-Based Localization. N 𝑁 N -cm Recall refers to the ratio of the cases which the localization error is below N 𝑁 N -cm. The inference time is the amount of calculation time for a single query, assuming a map is given.

	(a) Camera Tracking + Mapping		(b) Image-Based Localization
	ATE RMSE (m)	Inference Time (s)	25cm Recall (%)	50cm Recall (%)	1m Recall (%)	Inference Time (s)
Raw Noise	0.877	-	-	-	-	-
MapNet [24]	0.332	0.104	8.6	15.4	21.2	0.027
Nice-SLAM [55]	0.941	6.743	85.9	93.9	94.3	24.520
Nice-SLAM* [55]	0.071	25.977	91.1	93.7	94.4	24.520
RNR-Map (Ours)	0.108	0.271	76.6	99.2	99.5	0.018

Table 5. Table 5 : Localization Results on Object Change scenarios. Avg Img. Diff.: Average image differences between the query image and the originally observed image. Dist. Err.: Distance error between the localized position and the query position.

Difficulty	OC 1	OC 2	OC 3	OC 4
Avg Img. diff.	7%	14.1%	23.8%	33.9%
Dist. Err.(m)	0.083	0.091	0.111	0.140
25cm Recall (%)	76.7	72.8	71.5	69.6
50cm Recall (%)	99.8	98.8	98.6	97.4
1m Recal (%)	99.8	99.0	99.2	98.5

Table 6. Table 6 : Novel Environment Localization Results. The rows show each search method that finds the most visually similar place based on own metric or method (random, ours). Each column reports the normalized visual similarity from each metric, measuring the image found by each search method in rows. Naturally, the measured similarity of a metric found by the same metric would be 100%.

	CLIP [39]	CL	SSIM	L1
Random	88.23	30.91	78.66	88.44
Max CLIP [39]	100.0	61.36	84.58	90.48
Max CL	93.26	100.0	91.29	91.60
Max SSIM	93.75	66.98	100.0	93.40
Max Inv. L1	90.77	49.68	90.91	100.0
RNR-Map (Ours)	94.52	82.9	90.93	91.41

Table 7. Table 7 : Ablation studies of the navigation module on Gibson-curved scenarios. Exploration score and Latent score are used in the exploration module for planning where to visit. GT Point Navi and GT Stopper are ablations of the point navigation policy and the stopper module F stop subscript 𝐹 stop F_{\mathrm{stop}} , respectively. Each module is replaced with a simple function that has access to ground-truth information from the simulator.

	Gibson-Curved				Easy		Medium		Hard		Overall
	Exloration Score	Latent Score	GT Point Navi	GT Stopper	SR	SPL	SR	SPL	SR	SPL	SR	SPL
Ground-truth Information	✔	✔	✔	✔	98.0	85.0	95.0	70.8	86.4	53.9	93.1	69.9
	✔	✔	✘	✔	94.6	81.8	90.1	66.9	72.2	45.0	85.6	64.6
	✔	✔	✔	✘	72.4	61.2	65.2	47.1	55.2	33.2	64.3	47.1
Exploration Strategy	✔	✘	✘	✘	59.0	39.5	57.0	35.5	37.9	21.5	51.3	32.2
	✘	✔	✘	✘	71.1	60.5	65.9	45.4	40.4	24.2	59.2	43.4
Full Model	✔	✔	✘	✘	71.7	59.9	67.2	47.5	45.2	29.2	61.4	45.6

Table 8. Table 8 : Image-goal navigation results on MP3D [ 8 ] -curved scenarios in NRNS [ 23 ] dataset. The first column shows the training dataset each method has trained on.

Training Dataset	Method	Easy		Medium		Hard		Overall
		SR	SPL	SR	SPL	SR	SPL	SR	SPL
MP3D	NRNS [23]	23.7	12.7	16.2	8.3	9.1	5.1	16.3	8.7
MP3D	SLING+NRNS [49]	43.2	19.7	32.5	15.1	22.1	9.9	32.6	14.9
Gibson	OVRL [52]	16.9	8.3	25.8	13.5	14.3	7.0	10.6	4.6
MP3D	OVRL [52]	52.4	35.2	42.6	26.3	29.7	16.9	41.6	26.1
Gibson	SLING+OVRL [49]	47.5	30.2	28.4	17.0	19.3	9.3	31.7	18.8
MP3D	SLING+OVRL [49]	62.6	41.1	48.4	31.5	29.2	17.7	46.7	30.1
Gibson	RNR-Map (ours)	58.42	49.01	50.35	35.57	34.38	22	47.7	35.5

Equations

$$\begin{array}{rl}\left[\begin{array}{c}{q}_x(h,w)\\ q_y(h,w)\\ q_z(h,w)\end{array}\right]& =d_{h,w}{\mathbb{R}}^{-1}{\mathbb{K}}^{-1}\left[\begin{array}{c}h\\ w\\ 1\end{array}\right]-t.\end{array}$$

$$X_{(u,v)}$$

$$m(u,v)$$

$$\begin{array}{rl}{m}_t^l,n_t^l& =F_{\mathrm{reg}}(I_t,p_t;{\theta }_{\mathrm{enc}}),\end{array}$$

$$\begin{array}{rl}(m_t^g,n_t^g)& =F_{\mathrm{map}}(I_t,p_t,m_{t-1}^g,n_{t-1}^g;{\theta }_{\mathrm{enc}})\\ m_t^g(u,v)& =\frac{m_t^l(u,v)\cdot n_t^l(u,v)+m_{t-1}^g(u,v)\cdot n_{t-1}^g(u,v)}{n_t^l(u,v)+n_{t-1}^g(u,v)}\\ n_t^g(u,v)& =n_t^l(u,v)+n_{t-1}^g(u,v).\end{array}$$

$$m_i^g,n_i^g$$

$$\mathrm{Loss}({\theta }_{\mathrm{enc}},{\theta }_{\mathrm{dec}})$$

$$\begin{array}{r}{F}_{\mathrm{loc}}(m,m_{\mathrm{trg}};{\theta }_{loc})=F_E(\mathrm{Conv}(F_k(m),\{F_q(m_{\mathrm{trg}}{)}_r{\}}_1^R)),\end{array}$$

$$\begin{array}{rl}(\hat{E},{\hat{a}}_{\mathrm{trg}})& =F_{\mathrm{loc}}(m,m_{\mathrm{trg}};{\theta }_{loc})\\ \mathrm{Loss}({\theta }_{\mathrm{loc}})& =D_{KL}(E_{\mathrm{gt}},\hat{E})+CE(a_{\mathrm{trg}},{\hat{a}}_{\mathrm{trg}}),\end{array}$$

$$\widehat{p_t}=F_{\mathrm{track}}(m_{t-1},{\bar{p}}_t)=\arg \underset{\delta p_t}{\min }|F_{\mathrm{dec}}(m_{t-1},{\bar{p}}_t+\delta p_t)-I_t|,$$

$$\begin{array}{rl}c(\mathbf{r},m)& ={\int }_{u_n}^{u_f}Tr(u)\sigma (\mathbf{r}(u),q)\mathbf{a}(r(u),\mathbf{d},q)du\\ Tr(u)& =\exp (-{\int }_{u_n}^u\sigma (\mathbf{r}(u),q)du),\end{array}$$

Full text

1 Introduction

In this paper, we address how to explicitly embed the visual information from a 3D environment into a grid form and how to use it for visual navigation. We present renderable neural radiance map (RNR-Map), a novel type of a grid map for navigation.We point out three main properties of RNR-Map which make RNR-Map navigation-friendly. First, it is visually descriptive. Commonly used grid-based maps such as occupancy maps [10, 11, 17] and semantic maps [21, 48, 9], record obstacle information or object information into grids. In contrast, RNR-Map converts image observations to latent codes which are then embedded in grid cells. Each latent code in a grid cell can be converted to a neural radiance field, which can render the corresponding region. We can utilize the implicit visual information of these latent codes to understand and reason about the observed environment. For example, we can locate places based on an image or determine which region is the most related to a given image. RNR-Map enables image-based localization only with a simple forward pass in a neural network, by directly utilizing the latent codes without rendering images. We build a navigation framework with RNR-Map, to navigate to the most plausible place given a query image. Through extensive experiments, we validate that the latent codes can serve as important visual clues for both image-based localization and image-goal navigation. More importantly, a user has an option to utilize the renderable property of RNR-Map for more fine-level of localization such as camera tracking.

RNR-Map is generalizable. There have been a number of studies that leverage neural radiance fields (NeRF) for various applications other than novel view synthesis. The robotic applications of NeRF are also now beginning to emerge [44, 55, 1, 15, 31]. However, many of the approaches require pretrained neural radiance fields about a specific scene and are not generalizable to various scenes. This can be a serious problem when it comes to visual navigation tasks, which typically assume that an agent performs the given task in an unseen environment [16]. In contrast, RNR-Map is applicable in arbitrary scenes without additional optimization. Even with the unseen environment, the RNR-Map can still embed the useful information from images to the map and render images. The neural radiance fields of RNR-Map are conditioned on the latent codes. A pair of encoder and decoder is trained to make these latent codes from images of arbitrary scenes and reconstruct images using neural radiance fields. These pretrained encoder and decoder enable the generalization to unseen environments.

Third, RNR-Map is real-time capable. The majority of the present NeRF-based navigation methods require a significant time for inference because of required computation-heavy image rendering and rendering-based optimization steps. The RNR-Map is designed to operate fast enough not to hinder the navigation system. By directly utilizing the latent codes, we can eliminate the rendering step in mapping and localization. The mapping and image-based localization frameworks operate at 91.9Hz and 56.8Hz, respectively. The only function which needs rendering-based optimization is camera tracking, which can localize under odometry noises, and it operates at 5Hz.

To the best of our knowledge, the RNR-Map is the first method having all three of the aforementioned characteristics as a navigation map. The RNR-Map and its localization and navigation frameworks are evaluated in various visual navigation tasks, including camera tracking, image-based localization, and image-goal navigation. Experimental results show that the proposed RNR-Map serves as an informative map for visual navigation. Our localization framework exhibits competitive localization accuracy and inference speed when compared to existing approaches. On the image-goal navigation task, the navigation framework displays 65.7% success rate in curved scenarios of the NRNS[23] dataset, where the current state-of-the-art method[49] shows a success rate of 55.4%.

As RNR-Map finds a place based on the visual information of the map and the query image, we also consider a variant version of image-based localization. In real-world scenarios, there can be partial changes in the target place (changes in furniture placement, lighting conditions, …). Also, the user might only have images from similar but different environments. We test the proposed localization framework in both cases. We find that the RNR-Map is robust to environmental changes and is able to find the most visually similar places even when a novel query image from a different environment is provided.

The contributions of this paper can be summarized as follows:

• •

We present RNR-Map, a novel type of renderable grid map for navigation, utilizing neural radiance fields for embedding the visual appearance of the environment.

• •

We demonstrate efficient and effective methods for utilizing the visual information in RNR-Map for searching an image goal by developing RNR-Map-based localization and navigation framework.

• •

Extensive experiments show that the proposed method shows the state-of-the-art performance in both localization and image-goal navigation.

2 Related Work

Embodied AI with spatial memories.

One of the central issues in recent embodied AI research is how to construct a useful memory for the embodied agent [16]. A memory that contains the navigation history, as well as information about the observed environment, is required for successful task execution. There is a large body of works using occupancy maps for visual navigation [17, 41, 10, 11]. An occupancy map expresses the environment in a grid form, and each grid cell has obstacle information about the corresponding region. An occupancy map represents the overall structure of the environment and can guide a robot to navigate the environment safely. There have been various kinds of research about building a spatial memory which contains additional information more than obstacles. The additional information can be object classes of the observed objects [21, 48, 9, 7], or implicitly learned useful information for a specific task [22, 36, 24, 38]. MapNet [24], SMNet [7] and ISS [38] have a similar mapping architecture with our method. Like our approach, they learn how to convert RGBD observations into useful latent information, and record it in the spatial memories using 3D inverse camera projection. Using the recorded latent information, MapNet [24] learns to localize the agent pose, and SMNet learns to generate a semantic map. ISS [38] is more related to ours since this method addresses scene generation and novel view image synthesis from a grid map. Our research is focused on how we can utilize such latent information for visual navigation. We develop localization and navigation frameworks which actively utilize the embedded visual information in RNR-Map.

Robotics with neural radiance fields.

The neural radiance field (NeRF) [33] has gained significant popularity in various AI tasks. Not only in computer vision or graphics tasks, but NeRF is also adopted for robot applications in recent years. NeRF predicts the RGB color and density of a point in a scene so that an image from an arbitrary viewpoint can be rendered. This property enables pose estimation [44, 30, 31, 1] based on the photometric loss between the observed image and the rendered image, or manipulation of tricky objects [25, 12, 29, 14, 5]. A pretrained NeRF can also work as a virtual simulator, in which a robot can plan its trajectory [1] or can be used to train an action policy for the real-world [6]. Among the existing approaches, NICE-SLAM [55] is relevant to our work because it performs mapping and localization in arbitrary environments. NICE-SLAM builds a 3D implicit representation of the environment from image observations. The camera pose is inferred from optimizing the photometric loss between the observation image and the rendered image. Our method, on the other hand, places less emphasis on mapping quality, and it is designed for successfully completing navigation tasks. We focus on how the RNR-Map can be efficiently used for visual navigation, in terms of both speed and performance. The mapping and the target-searching function of RNR-Map are designed to operate fast enough to not hinder the other parts of the system. Also, the proposed RNR-Map method is generalizable in various environments without additional fine-tuning.

3 RNR-Map

A robot agent has an RGB-D camera, and also knows its odometry information. Here, the odometry means how much the agent has moved from its previous position and we consider 3-DoF pose in this paper. At time $t$, the robot observes an RGBD image $I_{t}$ and its relative pose $\Delta p_{t}=(\Delta x_{t},\Delta y_{t},\Delta a_{t})$ from the previous pose (xy position and heading angle). By cumulating pose differences, the agent can determine its relative pose $p_{t}$ from the start pose $p_{0}=(0,0,0)$.

A pair of the pretrained encoder and decoder is used when building a RNR-Map. The training process of these encoder and decoder resembles the autoencoder method. However, unlike 2D images, autoencoding a 3D environment is not a trivial problem. We build a reconstruction framework as shown in Figure 1(a). The encoder encodes an image and embeds the pixel features into the RNR-Map. We denote each embedded feature in a grid of RNR-Map as a latent code. A query pose is then provided, and the decoder samples latent codes along each camera ray corresponding to each pixel and renders the corresponding images. We present details of each part in the following section.

Registration $\mathbf{F_{\mathrm{reg}},F_{\mathrm{map}}}$.

When an RGBD image $I_{t}\in\mathbb{R}^{H\times W\times 4}$ comes in, the encoder encodes the image into a same-sized feature $C_{t}\in\mathbb{R}^{H\times W\times D}$, where $H$ and $W$ refers to height and width of the image, respectively, and $D$ refers to the channel size. First, each pixel $c_{h,w}\in\mathbb{R}^{D}$ in $C_{t}$ is mapped to its corresponding 3D world position $[q_{x},q_{y},q_{z}]^{T}$ using the known camera intrinsic $\mathbb{K}$, the extrinsic matrix $[\mathbb{R|t}]$ based on the agent pose, and the depth information $d_{h,w}$ of the pixel. The world position of each pixel is calculated using inverse camera projection, as follows:

[TABLE]

Then we digitize each position by normalizing with the map resolution $s$, and get map position $(u,v)$ as shown in (3). We aggregate the pixel features that belong to the same 2D map position, and average the aggregated features into a single feature vector. The pixel features are registered in the corresponding grid in the RNR-Map $m\in\mathbb{R}^{U\times V\times D}$, where $U$ and $V$ refer to the height and width of the RNR-Map, respectively. The number of averaged features at each 2D map position is also recorded in mask $n\in\mathbb{R}^{U\times V}$. We denote the element of $m$ at the map position $(u,v)$ by $m(u,v)$ and the mask at $(u,v)$ by $n(u,v)$. The registered feature $m(u,v)\in\mathbb{R}^{D}$ is the latent code which contains visual information of the region corresponding to the map position $(u,v)$. This process can be written as follows:

[TABLE]

The registration process $F_{\mathrm{reg}}$ includes the encoding of $C_{t}$, inverse projection, and feature registration. The $F_{\mathrm{reg}}$ is represented as:

[TABLE]

where $\theta_{enc}$ refers to the network parameters of the encoder. The registration process $F_{\mathrm{reg}}$ outputs a local map $m_{t}^{l}$ and a local mask $n_{t}^{l}$. The local map $m_{t}^{l}$ only contains the information from the image $I_{t}$, and this will be integrated with other local maps to form the global map $m^{g}$.111For simplicity, $m$ without any superscript refers to the global map ($m^{g}$) in the rest of the paper. When multiple observations are given, we can use $n$ to compute the average value from the original and new latent codes. We name this integration process $F_{\mathrm{map}}$, which operates $F_{\mathrm{reg}}$ over multiple observations. $F_{\mathrm{map}}$ at time $t$ with previous $m^{g}_{t-1}$ is formulated as follows:

[TABLE]

Decoding $\mathbf{F_{\mathrm{dec}}}$.

To make these latent codes contain visual information, we reconstruct the image observation from the latent codes. We use a decoder which has a structure similar to the generative scene network (GSN) [13] for rendering an RGBD image from the 2D latent map. Originally, GSN generates a random indoor floorplan from random variables. Then GSN proposed how to render images from the generated floorplan, based on locally conditioned radiance fields. Our approach involves designing the encoder $\theta_{\mathrm{enc}}$ and the registration process $F_{\mathrm{reg}}$ to transform image observations into latent codes, which can be converted to the locally conditioned radiance fields. We utilize the locally conditioned radiance fields from GSN to render an image from $m$. Given the camera parameters, we can sample latent codes on points along the camera ray, corresponding to the pixel location which will be rendered in the camera. The sampled latent codes are converted into modulation linear layer-based locally conditioned radiance fields [4, 13]. The decoder is trained to render an RGBD image from the latent codes to be close to the image observations $I_{t}$. The reader can refer to [13] for a more detailed explanation about the rendering procedure.

By training the rendered RGBD images $\tilde{I}_{t}$ to be close to the original observation $I_{t}$, the encoder is trained to extract the visual information from the image. This mechanism is similar to training an autoencoder, to extract and summarize useful information from an image into a latent vector. We sample $N$ images in a random scene and embed them into the RNR-Map. Then, we render each image from the final RNR-Map and compare them with the original images. The encoder and the decoder are trained using the following loss:

[TABLE]

where $\theta_{\mathrm{enc}}$ and $\theta_{\mathrm{dec}}$ are weight parameters of the encoder and decoder, respectively.

Since the rendering process is conditioned on the latent codes from the image observation, our proposed reconstruction framework is capable of embedding and rendering arbitrary images from unseen environments. This leads to the generalizable property of RNR-Map. Also, the decoder can synthesize a novel view, different from the observed direction, based on $m$. Examples of reconstructed observations are shown in Figure 1(b). The decoder can render images from novel viewpoints in the observed region, based on the latent codes. Note that the rendering boundary is limited to the observed 3D points. We can see that the decoder generates grey color for the unobserved regions, in the last row of Figure 1(b). More examples of reconstructed images are provided in the supplementary material A, as well as the network architectures of the encoder and decoder B.1.

Mapping.

After training, we can use $F_{\mathrm{map}}$ for logging visual information from the environment. Note that the rendering function $F_{\mathrm{dec}}$ is not needed for mapping. The RNR-Map is built incrementally during navigation, based on the odometry information and the known camera intrinsics. At time $t$, the agent obtains $m_{t}$ and $n_{t}$ using $F_{\mathrm{map}}$, as formulated in (4). If the same 3D point in the environment is observed multiple times, $F_{\mathrm{map}}$ averages the latent codes based on the number of observation instances.

4 Localization

One of the crucial components of the navigation task is localization. In the following sections, we describe how RNR-Map is used for two localization tasks: image-based localization and camera tracking. Here, we consider 3-DoF pose $(x,y,a)$ for localization.

4.1 Image-Based Localization

The implicit visual information of the latent codes in RNR-Map can provide useful clues for finding out which grid cell is the closest to the given target image $I_{\mathrm{trg}}$. Inspired by the fast localization method in [24], we directly compare latent codes from the target image $I_{\mathrm{trg}}$ and the RNR-Map $m$ for localization. We denote this image-based localization function as $F_{\mathrm{loc}}$. Suppose that the target image is taken at the pose $p_{\mathrm{trg}}=(x_{\mathrm{trg}},y_{\mathrm{trg}},a_{\mathrm{trg}})$, and the corresponding map position is $(u_{\mathrm{trg}},v_{\mathrm{trg}})$. $F_{\mathrm{loc}}$ outputs a heatmap $E\in\mathbb{R}^{U\times V}$ and the predicted orientation of the target $a_{\mathrm{trg}}$. $E$ highlights which grid cell corresponds to the target location $(u_{\mathrm{trg}},v_{\mathrm{trg}})$, among the observed area in $m$.

The localization process $F_{\mathrm{loc}}$ is shown in Figure 2. The RNR-Map $m$ is constructed from a partial observation about the environment. The query image is transformed into $m_{\mathrm{trg}}$ using $F_{\mathrm{reg}}(I_{\mathrm{trg}},p_{0};\theta_{\mathrm{enc}}).$ Here, we use origin $p_{0}=(0,0,0)$ as an input to $F_{\mathrm{reg}}$ since the agent does not know the position of the target location. $F_{\mathrm{loc}}$ includes three convolutional neural networks, $F_{k}$, $F_{q}$, and $F_{E}$. $F_{k}$ and $F_{q}$ are for proccessing $m$ and $m_{\mathrm{trg}}$ into $m^{\prime}_{k}$ and $m^{\prime}_{q}$, respectively. We found it helpful to introduce neural networks ($F_{k}$, $F_{q}$) for better localization results. Then, we search query $m^{\prime}_{q}$ by convolving (cross-correlation, denoted as $\mathrm{Conv}$) with $m^{\prime}_{k}$. The query map $m^{\prime}_{q}$ is rotated into $R$ different angles $\{0^{\circ},...,360^{\circ}\times\frac{R-1}{R}\}$. $(m^{\prime}_{q})_{r}$ denotes $Rot_{r}(m^{\prime}_{q})$, where $Rot_{r}$ represents the $r$-th from the $R$ possible rotations. Each rotated query $(m^{\prime}_{q})_{r}$ works as a filter, and the output from the $\mathrm{Conv}$ is forwarded to $F_{E}$. $F_{E}$ outputs $E\in\mathbb{R}^{U\times V}$ which highlights the most query-related region in $m$. Each pixel $e_{u,v}\in E$ in the heatmap represents the probability of the query image being taken at $(u,v)$. Also, $F_{E}$ has an additional head which predict the orientation of the target observation.

The overall localization process can be put together as:

[TABLE]

where $\theta_{loc}$ denotes the weight parameters of $F_{k}$, $F_{q}$, and $F_{E}$. Detailed explanations about the network architecture are provided in the supplementary material B.2.

We make a ground-truth Gaussian heatmap $E_{\mathrm{gt}}$ which highlights the ground-truth map location $(u_{\mathrm{trg}},v_{\mathrm{trg}})$ of the query image. The representation of an orientation $a_{\mathrm{trg}}$ is discretized into 18 bins, and $F_{\mathrm{E}}$ is trained to select the appropriate bin using cross-entropy loss. The framework is trained to generate a distribution similar to the ground-truth heatmap $E_{\mathrm{gt}}$ and predicts the $a_{\mathrm{trg}}$. The following loss term is used to train $F_{\mathrm{loc}}$:

[TABLE]

where $D_{KL}$ refers to KL divergence, and $CE$ refers to cross-entropy loss. We provide quantitative and qualitative experimental results of $F_{\mathrm{loc}}$ in the supplementary material C.2.

4.2 Camera Tracking

During the navigation, the agent needs to be aware of its own pose to accurately record the observed visual observation. By cumulating odometry readings, the agent can calculate its relative pose to the start pose. However, in the real world, it is difficult to determine the accurate pose of a robot due to noises in odometry readings. The differential rendering function of $F_{\mathrm{dec}}$ can be used for adjusting the rough estimation of the agent pose $p_{t}$. The pose optimization is based on the photometric loss between the rendered image and the current observation. As the rendering process $F_{\mathrm{dec}}$ is differential, the pose of the agent $p_{t}$ can be optimized with the gradient descent method. We name this camera tracking function as $F_{\mathrm{track}}$. At time $t$, the agent has the previously estimated pose $\hat{p}_{t-1}$, and takes the odometry data $\Delta p_{t}$. The rough estimation of the current pose $p_{t}$ can be calculated by simply adding $\Delta p_{t}$ to the previous pose $\hat{p}_{t-1}$. $\bar{p}_{t}$ denotes such roughly estimated pose: $\bar{p}_{t}=\hat{p}_{t-1}+\Delta p_{t}$. Using $F_{\mathrm{track}}$, $\bar{p}_{t}$ is optimized to $\hat{p}_{t}$. The output of pose optimization can be derived by the following equation:

[TABLE]

which minimizes the error between the current observation and the rendered image from the latent map. $\bar{p}_{t}$ is the initial value of the pose in the optimization process. By sampling a small subset of pixels from the image, we can make this optimization process fast enough to use it in navigation. We provide the accuracy and inference speed of the proposed camera tracking method in the supplementary material C.1.

5 Navigation

Now we describe how the RNR-Map and RNR-Map-based localization functions can be utilized in a navigation system. We consider the visual navigation task, especially image-goal navigation. The objective of image-goal navigation is to find the target location given the image taken from the target location. We build a visual navigation framework which includes $F_{\mathrm{map}}$ for mapping, $F_{\mathrm{track}}$ for localization, and $F_{\mathrm{loc}}$ for target searching. Figure 3 shows an overview of the proposed navigation system. The system consists of three modules, mapping, localization, and navigation. In the following section, we describe how each module works during navigation.

5.1 Mapping Module

The mapping module builds RNR-Map using the pretrained encoder and $F_{\mathrm{map}}$. At the start of the episode ($t=0$), the mapping module transforms the target image $I_{\mathrm{trg}}$ into an RNR-Map $m_{\mathrm{trg}}$. While maintaining $m_{\mathrm{trg}}$, the module updates the current RNR-Map $m_{t}$ using each image observation $I_{t}$ with $F_{\mathrm{map}}$. Also, the mapping module builds an occupancy map using depth information. This occupancy map is used for collision avoidance in a point navigation policy. The navigation module also uses this occupancy map to add more exploration property to the navigation system.

5.2 Localization Module

The localization framework described in Section 4 works as a localization module in the proposed navigation system. This localization module has two objectives during navigation. First, the localization module finds the most probable area which is similar to the target location. Second, considering a noisy odometry sensor, the localization module is needed to figure out the precise current pose of the agent. The image-based localization function $F_{\mathrm{loc}}$ and camera tracking function $F_{\mathrm{track}}$ are actively used in this module. With high probability, the target location may not be in the current RNR-Map $m_{t}$ in the navigation scenario. Hence, $F_{\mathrm{loc}}$ for the navigation task is trained to find the region in $m_{t}$ which is closer to the target location.

5.3 Navigation Module

The navigation module consists of three submodules: exploration, point navigation, and stopper.

Exploration.

The objective of the exploration module is to select the most plausible region to explore. The module decides where to visit in order to search the target location, based on the probabilty heatmap $E$ from $F_{\mathrm{loc}}$ in the localization module. We have adopted the concept from robot exploration [27, 53, 56, 34], which builds a generalized Voronoi graph on the occupancy map. We draw a Voronoi graph on the occupancy map and calculate visitation priority scores for each node of the created graph. Based on the scores, the exploration module selects the nodes to explore. Two types of scores are used for selecting exploration candidates, the latent score and the exploration score. The latent score is based on the heatmap $E$ and represents how probable the node is near the target location. The exploration score of each node is simply calculated based on the values in the occupancy map. The occupancy map has three types of value: occupied, free and unseen. The exploration score of a node is proportional to the number of unseen pixels in the neighborhood of a node. The visitation priority of a node is determined based on the sum of the latent score and the exploration score.

Point navigation policy and Stopper.

The point navigation policy is a simple occupancy-map-based path-following algorithm. When the exploration target node is selected, the point navigation module draws the shortest path to the target position. Following the path, the point navigation policy heuristically avoids obstacles using the occupancy map. The stopper module determines the arrival at the target location and calculates the relative pose from the target location. We employ a neural network $F_{\mathrm{stop}}$ which decides whether the agent is near the target location. This neural network is trained to output a binary value (1 if the target location is reached and 0, otherwise) based on the $m_{trg}$ and $m_{t}$. For efficient target reaching, we adopted the recent last-mile navigation method[49] in stopper module. Based on keypoint matching, the relative pose between the target location and the current location is calculated using Perspective-n-Point [28] and RANSAC [19]. After $F_{\mathrm{stop}}$ detects the target location, the point navigation policy navigates to the target using the estimated relative pose. We provide detailed explanations about graph generation and exploration planning in the supplementary material G.

5.4 Implementation Details

The modules that require training are the encoder and decoder, and the neural networks used in $F_{\mathrm{loc}}$ and $F_{\mathrm{stop}}$. We trained them using the same dataset. We have collected 200 random navigation trajectories from each scene in 72 Gibson [51] environments with the Habitat simulator [32]. The pair of encoder and decoder is first trained, then $F_{\mathrm{loc}}$ and $F_{\mathrm{stop}}$ are trained based on the pretrained encoder. Further implementation details (network architectures and training details) are provided in the supplementary material B.

6 Experiments

We have evaluated RNR-Map in both localization and navigation tasks. However, as the main objective of the proposed RNR-Map is visual navigation, we focus on analyzing the experiment results of the navigation task in the main manuscript. For localization tasks, we summarize the experimental results here and provide detailed information and analyses in the supplementary material C.

6.1 Localization

We have tested $F_{\mathrm{track}}$ and $F_{\mathrm{loc}}$ with other related baselines [24, 55]. In camera tracking task, the RNR-Map $F_{\mathrm{track}}$ shows high speed (5Hz) and accuracy (0.108m error) which are adequate for a real-time navigation system. More than localizing the current pose of the agent, the RNR-Map $F_{\mathrm{loc}}$ is able to locate the previously seen images (image-based localization task) with a high recall rate (inliers less than 50cm with 99%) in the recorded map. We can leverage this RNR-Map for searching the most similar place to the query image even if the exact place is not in the current environment. Two scenarios can be considered: (1) (Object Change) There have been large changes in the environment so that the object configuration of the environment is different from the information in the RNR-Map. (2) (Novel Environment) The user only has a query image from a different environment but wants to find the most similar place in the current environment. We have tested the RNR-Map in both scenarios and observed that $F_{\mathrm{loc}}$ robustly localizes query images even if some object configuration changes. When 33.9% of the observed images have changed, $F_{\mathrm{loc}}$ localizes the query image with less than 50cm error in 97.4% of cases, and 20∘ error in 97.5% of cases. Also, when given a query image from a different scene, the localized location achieves 94.5% of visual similarity compared to the best possible visual similarity in the given scene. The examples of the image-based localization tasks are shown in Figure 4. We can see that $F_{\mathrm{loc}}$ finds visually similar places based on the RNR-Map, even when the environment has changed or given in a novel environment. Additionally, we found that the suggested method $F_{\mathrm{loc}}$ mislocates when there are multiple, similar locations in the environment or when the query has poor visual information. We provide more detailed experiments with baselines (MapNet[24], NICE-SLAM[55]) and examples in the supplementary material C.

6.2 Image-Goal Navigation

6.2.1 Baselines

We compare RNR-Map with learning-based agents using behavior cloning (BC+RNN) and reinforcement learning (DDPPO [50]). Also, to figure out the advantages of the RNR-Map over an occupancy map, we include the occupancy-map-based coverage method [10] as a baseline. The objective of this method is to visit all possible area in the given scene. We modified this method with the target distance prediction from [23], to make the agent reach the target when it is detected while exploring the environment (ANS [10]+Pose Pred). We also compare our method with the recent state-of-the-art image-goal navigation methods. ZSEL[2], and OVRL[52] are reinforcement learning-based methods which learn image-goal navigation task with specially designed rewards and the pretrained visual representation. NRNS builds a topological map and select the nodes to explore by predicting the distances between the target image and the node images with a distance prediction network. SLING is a last-mile navigation method which predict the relative pose of the target based on keypoint matching, after the target is detected. This method needs to be integrated with the exploratory algorithm such as NRNS or OVRL. Note that our method adopted this method, for efficient target reaching. The digits of the baseline DDPPO, OVRL, (NRNS,OVRL)+SLING are from [49] and their open-sourced code222https://github.com/Jbwasse2/SLING, and NRNS, ZSEL are from the original papers [23], [2], respectively.

6.2.2 Task setup

We have tested each method in the public image-goal navigation datasets from NRNS[23] and Gibson [51] with Habitat simulator [32]. Gibson dataset consists of 72 houses for training split, and 14 houses for the validation split. NRNS dataset consists of three difficulty levels (easy, medium, hard) with two path types (straight and curved). Each difficulty level has 1000 episodes for each path type, except for the hard-straight set (806). The objective of the image-goal navigation is to find the target place given the image of the target location. The agent only has access to the current RGBD observations and an odometry reading. We consider a noisy setting [10] where the odometry sensor and the robot actuation include noises as in the real world. The RGBD image observation comes from a directional camera with $90^{\circ}$ of HFOV. A discrete action space is used in this paper with four types of actions: move forward $0.25m$. turn right $10^{\circ}$, turn left $10^{\circ}$, and stop. The maximum time step of each episode is set to 500. An episode is considered a success when the agent takes a stop action within $1m$ from the target location. Two evaluation metrics are used: success rate (SR) and success weighted by (normalized inverse) path length (SPL) [3], which represents the efficiency of a navigation path.

6.2.3 Image-goal navigation Results

RNR-Map helps efficient navigation.

Table 1 shows the average SR and SPL of each method. We can see that the proposed navigation framework with the RNR-Map shows competitive or higher performance compared to the baselines, on image-goal navigation tasks. Many of the baselines (DDPPO, ZSEL, OVRL, OVRL+SLING) include reinforcement learning which is sample inefficient and computationally heavy, while having relatively simple representation about the environment. In contrast, the RNR-Map shows higher performances while only requiring an offline trajectory dataset for training neural networks. Based on this result, we argue that having a good internal representation of the environment and finding how to extract exploratory signals from such representation are important. An agent with an informative environmental representation can navigate well without the numerous inefficient interactions often required in reinforcement learning. Compared to baselines which have their own internal representation of the environment (NRNS, NRNS+SLING, ANS), our method shows a much higher performances in curved scenarios. From this result, we can infer that the RNR-Map indeed provides useful information for searching targets, more than coverage signals, and better than the existing methods. The ablation study shown in Table 2 also displays a similar trend. We ablated the main functions of the navigation framework $F_{\mathrm{loc}}$ and $F_{\mathrm{track}}$, as well as noises. Without the latent score from $F_{\mathrm{loc}}$, the success rate and SPL dramatically drop. We also have tested the proposed method in MP3D [8] dataset, and observed similar results with Gibson dataset. We provide the results in the supplementary material E, and additional ablation study about the submodules of the navigation module is also provided in D.

$\mathbf{F_{\mathrm{track}}}$ makes RNR-Map robust to noise.

Comparing the third row and the last row of Table 2, we can infer that the pose adjusting function $F_{\mathrm{track}}$ helps the agent to find the image goal more effectively, even with the noisy odometry. The proposed method shows higher success rates in noisy settings, while SPL values are low. We believe this is from the randomness of noises, which helps the local navigation policy to escape from being stuck in a single location.

RNR-Map is real-time capable.

We analyzed the runtime for each feature of RNR-Map and report them in Table 3.

The runtimes are measured on a desktop PC with a Intel i7-9700KF CPU @ 3.60GHz, and an NVIDIA GeForce RTX 2080 Ti GPU. We can see that each function of RNR-Map operates fast enough for real-time navigation, even when including NeRF-based rendering.

Navigation example.

Figure 5 provides an example of an image-goal navigation episode. $F_{\mathrm{loc}}$ highlights two types of places, a place that looks similar to the target image and an explorable place like the end of the aisle, or doorway. We can see that the highlighted area at the start of the episode has a similar observation to the target image. First, the agent navigates to a similar-looking place but decided not to stop at the location. While navigating to another place that shows a high latent score, the agent has found the target location, and the latent score also changed to highlight the target. Additional qualitative results of navigation episodes are provided in the supplementary material F and video.

7 Conclusion

In this paper, we have proposed RNR-Map for visual navigation, which captures visual information about the environment. The RNR-Map is helpful for visual navigation in two ways: (1) The latent codes in the RNR-Map can provide rich signals to find the target location, given a query image. (2) The rendering property of the RNR-Map helps the agent to accurately estimate its pose based on a photometric error, in an unseen environment. The proposed method has outperformed other methods in image-based localization. Also, we have found that the image-based localization of RNR-Map is robust to environmental changes. In image-goal navigation tasks, the proposed method outperforms the current state-of-the-art image-goal navigation methods. Furthermore, the fast inference time of the RNR-Map shows its potential for real-world applications. However, the proposed method still has limitations. Once the observation images are embedded in grids, RNR-Map is hard to correct the past odometry error. We can consider applying loop closure using the proposed localization framework. Inspired by existing graph-based SLAM methods, a pose graph with local RNR-Maps can be leveraged to optimize poses, leading to consensus in the pixel renderings from the global RNR-Map.

Appendix: Renderable Neural Radiance Map for Visual Navigation

We provide additional analyses and examples of the proposed method. The followings are included in this supplementary material: We provide additional analyses and examples of the proposed method. The followings are included in this supplementary material:

• A

Examples of reconstructed images from $F_{\mathrm{dec}}$

• B

Implementation details of the neural networks

• C

Experiments of localization functions $F_{\mathrm{track}}$ and $F_{\mathrm{loc}}$

• E

Image-goal navigation results on MP3D[8] dataset.

• D

Ablation studies of the modules in the navigation system.

• F

Qualitative examples of image-goal navigation episodes

• G

Implementation details of each submodule in the navigation module.

Appendix A Examples of reconstructed images

We provide examples of the reconstructed images from the decoder $F_{\mathrm{dec}}$ in Figure 7. The images are sampled from unseen environments, and not used during training. We embedded eight image observations for each scenario into RNR-Map and reconstructed the images. Also, we sample two novel views, which are not embedded in RNR-Map. We can see that $F_{\mathrm{dec}}$ can render images well in both seen and novel views. More importantly, RNR-Map is able to embed and reconstruct images from arbitrary scenes. RNR-Map can only render the observed region from various viewpoints, and the unobserved regions are rendered as empty space. Examples of the rendered unobserved region are shown in the last three cases in Figure 7.

Although the quality of each image is not state-of-the-art, RNR-Map can reconstruct the overall structure of the image and render large objects. Small objects and the details of the texture are often ignored in rendered images. The experiment results from the image-goal navigation task show that it is enough to accurately infer the camera pose based on the image renderings. Better training techniques and more weight parameters of the encoder and decoder will improve the image quality and navigation performance.

Appendix B Network Architecture and Training Details

B.1 Encoder and Decoder

We use images with a size of $128\times 128\times 4$. The encoder used in $F_{\mathrm{enc}}$ is a simple convolutional network consisting of four convolutional layers. The encoder embeds an image $I$ into a same-sized feature $C$ with 32 channels. We normalize each pixel feature $c_{h,w}$ along the feature dimension. These pixel features are averaged according to their (inverse) projected 3D positions and embedded into RNR-Map $m$.

The network structure of the decoder is adopted from GSN [13]. The illustration of the decoding process is shown in Figure 6. Given a query pose $\mathbf{p}$ and known camera intrinsic, we can calculate which 3D position will be rendered in a specific pixel $(h,w)$. More specifically, assuming a ray which passes the pixel $(h,w)$, we sample 3D points along the ray. Let $u$ be the variable of how far the point is from the camera center along the ray. For each sampled 3D point, we can calculate its map position, and select the corresponding feature $q$ from the $m$. As the calculated map position will have continuous values, we sample the feature using bilinear interpolation between the grids. GSN proposed a local coordinate system, which represents a 3D position of a point with a relative pose in a grid. The decoder network takes this local pose $\mathbf{p^{\prime}}$, view direction $\mathbf{d}$, and the sampled latent code. This can be understood as the decoder rendering how a specific region would look like, from the local pose and view direction. The latent code becomes the modulation linear layer, and outputs the occupancy $\sigma$, and appearance $\mathbf{a}\in\mathcal{R}^{3}$ The pairs of occupancy and appearance are calculated for all the sampled latent codes along the ray. Finally, the rendered color of a ray $\mathbf{r}$ is calculated with implicit volumetric rendering [33]:

[TABLE]

which is the same as the rendering function in GSN [13].

We only have changed the last step of the rendering in GSN. For computational efficiency, GSN renders a small-size of feature map and upsamples it as an image using a convolutional network. Since the objective of GSN is to generate a realistic scene, a whole image with good quality is needed to evaluate the generated scene. However, RNR-Map needs to individually render a small subset of pixels for the efficient calculation in the camera tracking function $F_{\mathrm{track}}$. Hence, we did not use the last convolutional network in GSN, and replaced it with a simple linear layer. With this linear layer, we can get the color of the selected pixels, not requiring whole image rendering.

The size of the RNR-map represents $32m\times 32m$ area with $128\times 128\times 32$ size of a tensor, where 32 refers to the feature dimension. Each grid cell in RNR-map represents $25cm\times 25cm$ of a region.

B.2 Localization Network $F_{loc}$

The image-based localization is based on two RNR-Map, $m$ and $m_{trg}$. $m$ is constructed using partial observations from the environment, and $m_{trg}$ is constructed with the given query image. Note that the query image is embedded in $m_{trg}$ at the origin pose $p_{0}$, ($m_{trg}=F_{\mathrm{reg}}(I_{\mathrm{trg}},p_{0};\theta_{\mathrm{enc}}).$) The localization proccess $F_{\mathrm{loc}}$ is done by convolving the given RNR-Map $m\in\mathcal{R}^{128\times 128\times 32}$ and the target RNR-Map $m_{trg}^{32\times 32\times 32}$. As $m_{trg}$ has only the information of the target image at the origin, most of the grid cells are empty. Hence, we crop and only use the center of $m_{trg}$, to reduce unnecessary computations.

The localization network $F_{\mathrm{loc}}$ consists of three convolutional neural networks, $F_{k}$, $F_{q}$ and $F_{E}$. The last neural network $F_{E}$ has two heads $F_{E_{1}},F_{E_{2}}$, whose outputs are the heatmap $\hat{E}$ and the predicted orientation of the query pose $\hat{a}_{\mathrm{trg}}$, respectively. $F_{k}$, $F_{q}$, and $F_{E_{1}}$ have the same U-Net architecture and output the same-size feature as the input. $F_{E_{2}}$ is based on ResNet-18, which processes the output of the cross-correlation into the 18-angle bins.

B.3 Stopper $F_{stop}$

The stopper $F_{\mathrm{stop}}$ determines whether the agent is near the target location or not, based on $m$ and $m_{\mathrm{trg}}$. We employed the attention mechanism for $F_{\mathrm{stop}}$, the overall procedure is illustrated in Figure 8.

The local area has the majority of the information required to determine the relative distance to the target. Hence we crop the neighborhood of the agent position in $m$, and provide it as an input to $F_{stop}$. The cropped patch $m^{\prime}\in\mathcal{R}^{32\times 32\times D}$ and the target $m_{\mathrm{trg}}$ are flattened and considered as a sequence of the latent codes. We conduct self-attention for each sequence, and then conduct cross-attention between them as shown in Figure 8. The outputs of the cross-attention are unflattened as the original size and forwarded to a convolutional network. This convolutional network consists of four convolutional layers and a linear layer. The last linear layer outputs the prediction of the closeness to the target.

B.4 Training

We have collected 200 random navigation trajectories on 86 (training 72 + validation 14) scenes from the Gibson [51] dataset. We first trained the pair of encoder and decoder, and then trained the $F_{\mathrm{loc}}$ and $F_{\mathrm{stop}}$ with the frozen parameters of the encoder.

The neural networks used in RNR-Map are all trained with the same dataset. The data sample for each neural network is also created in a similar manner. We sample a trajectory from the dataset and select a subset of frames from the trajectory. The pose information of the subset is normalized with the first frame of the sampled subset, considering the first frame as the origin. The images are embedded in RNR-Map according to the normalized pose. Then we select a query frame from the trajectory. For the encoder and decoder, the query image is selected in the subset. They are trained to reconstruct the query image from RNR-Map. When $F_{\mathrm{loc}}$ is trained for image-based localization, the query image is also selected from the subset which is embedded in RNR-Map. In contrast, when $F_{\mathrm{loc}}$ is trained for navigation, the query image is often selected outside of the subset, which is not provided in RNR-Map. Using this data sample, $F_{\mathrm{loc}}$ is trained to estimate which grid cell would be the closest to the target location. The $F_{\mathrm{stop}}$ is trained to determine the query position is in the neighborhood of the origin (the first frame of the sampled subset).

The training of the encoder and decoder is done with four GPUs (24GB NVIDIA GeForce RTX 3090), with a batch size of 16. The $F_{\mathrm{loc}}$ and $F_{\mathrm{stop}}$ are trained with one GPU (24GB NVIDIA GeForce RTX 3090), with a batch size of 32. All neural networks are trained until the validation loss converges.

Appendix C Localization Experiments

C.1 Camera Tracking

We compare the proposed method on the camera tracking task against existing methods that build an environmental map in the latent space. MapNet[24] proposed a method to build an allocentric spatial memory from egocentric observation. This method builds a spatial memory for camera pose localization, which is done by comparing the latent features between the current image and the memory. The localization function $F_{\mathrm{loc}}$ in our method resembles this operation. NICE-SLAM[55] builds a renderable latent map of the environment using a differentiable rendering function (NeRF). Localization and mapping of this method are based on the photometric error between the rendered image and the observations. We used the official repository of each method333MapNet: https://github.com/jotaf98/mapnet, NICE-SLAM: https://github.com/cvg/NICE-SLAM. The objective of this task is to estimate the camera trajectory, following a stream of image observations and noisy odometry sensor readings. We evaluate RNR-Map and MapNet with 1,000 trajectories from the validation scene, and a 10% subset for NICE-SLAM. We were only able to test a small subset for NICE-SLAM due to its large computational time. Each trajectory contains 1,000 frames of observations. We use root mean squared error of average trajectory error (ATE RMSE) [43] as a metric for the accuracy of camera tracking. The inference time is also measured, and it is the average mapping and tracking time for a single frame. The inference times of every method are measured on a desktop PC with a Intel i7-9700KF CPU @ 3.60GHz, and an NVIDIA GeForce RTX 2080 Ti GPU.

The experiment results are shown in Table 4. Raw noise in the first row shows the average trajectory error when the noises are not adjusted by any camera-tracking method. Our method is slightly slower than MapNet. The reason is that the localization function of MapNet is based on cross-correlation between the latent maps, while our camera tracking function $F_{\mathrm{track}}$ is based on rendering-based optimization. However, our method shows higher accuracy in inferring camera poses. MapNet discretizes the environment into grids and selects the most relevant grid cell for localization. In contrast, by rendering the image observations, we can adjust the camera pose at a finer level, smaller than the grid size.

Since our method directly embeds the image feature to the grids, RNR-Map shows a much faster inference speed because NICE-SLAM needs a rendering-based optimization process for mapping. The NICE-SLAM camera tracking results are significantly worse than the performances described in the original paper [55]. This is because our trajectory dataset is for navigation. There is much less overlap between each frame than in the dataset used for SLAM tasks, which results in performance deterioration. We also tested a different hyperparameter set for NICE-SLAM which conducts more optimization steps on both mapping and tracking (NICE-SLAM*). NICE-SLAM* outputs much more accurate results for the camera tracking. There is a significant trade-off between accuracy and inference time in NICE-SLAM, as more optimization steps lead to high accuracy on camera tracking but take considerable time.

C.2 Image-Based Localization

The objective of the image-based localization task is to find the pose of the query image, which is observed in the distant past. This task is different from camera tracking, which asks about the current pose of the agent, requiring relatively recent information. We tested each baseline on the image-based localization, and the results are shown in Table 4. We sampled 10 query images from each trajectory used in camera tracking (a total of 10,000 test samples). Also, in here, we were only able to test 10% of the samples for NICE-SLAM (1,000 test samples).

The RNR-Map is aggregated along the trajectory, preserving past information. As a result, it can be used to locate a target that has been observed in the distant past. MapNet [24] is focused on finding the camera pose at the moment, so it uses a recurrent neural network (RNN) for a better understanding of sequential observations. This RNN makes the method less effective in the image-based localization task because old information can be easily lost.

For NICE-SLAM, we render each pose candidate and select the best pose with the lowest photometric difference. As the rendering process takes a lot of time, we made NICE-SLAM more privileged, providing the possible pose candidates. The pose candidates are the recorded poses from the mapping process. NICE-SLAM shows a much longer inference time because the predicted location always has to be compared in the image domain, requiring a rendering process. In contrast, RNR-Map can find the location at high speed with high accuracy, by directly comparing the latent codes rather than rendering every position. $F_{\mathrm{loc}}$ of RNR-Map locates the target with less than 50cm with an accuracy of 99%, with a fast speed of 0.018 seconds. $F_{\mathrm{loc}}$ selects the most probable grid which corresponds to $25cm\times 25cm$ region. This reduces the performance of the 25cm Recall, which requires precise localization finer than the grid size. However, compared to rendering each image and comparing them to the query, the RNR-Map still exhibits 5.87% higher localization performance in 50cm Recall with an incomparably faster time.

We provide some examples of success cases and failure cases from experiment results in Figure 9. The $F_{\mathrm{loc}}$ generally finds the query observation with a small error. However, when there are multiple visually-similar regions in the given environment, $F_{\mathrm{loc}}$ often selects the wrong places. We can see that the found location has a similar visual appearance to the query image.

C.3 Similar-image-goal localization

We can leverage this RNR-Map for searching the most similar place to the query image even if the exact place is not in the current environment. Two scenarios can be considered: (1) (Object Change) There have been large changes in the environment so the object configuration of the environment is different from the information in the RNR-Map. (2) (Novel Environment) The user only has a query image from a different environment but wants to find the most similar place in the current environment.

Object change.

We divided the difficulty of the localization scenario with the changed environments into four levels: OC1, OC2, OC3, and OC4. We first recorded RNR-Map map for each validation scene without any additional objects. Then we randomly place the random objects (bags, boxes, sofa, chair, toys, etc.) in the scenes, and take pictures of the changed environment. We gave this picture with objects as a query image. The difficulty is determined by how much the image has been changed because of the random objects. The image difference is calculated as the L1 loss between the newly captured image with the random objects and the original image without objects. This value is normalized by the size of an image so that it represents how much the image has been changed from the original. Each difficulty contains 300 samples of the key scene and query image pair. We show the quantitative results in Table 5. We observed that the RNR-Map robustly localizes query images even if some object configuration changes. The RNR-Map finds the query location with less than 50cm error in 97.4% of the cases, even in the most difficult scenario. The qualitative results are shown in Figure 11(a).

Novel environments.

We also tested the novel environment (NE), where the query image is from a different scene. The objective of this task is to find the most visually similar location to the query image. However, the visual similarity is hard to quantify and can vary depending on the metric. To evaluate the localized observation, we use various types of metrics which can measure the similarity between images. We leverage two image encoders which are trained using contrastive learning (CLIP, CL). In contrastive learning, the image encoder converts images into feature vectors and is trained to maximize the cosine similarity between the features from similar images. Using the two contrastive learning models, we calculate the cosine similarity of the feature vectors from the query image and the localized observation. We also utilize the structural similarity index (SSIM) and L1 loss to measure the visual similarity. The following metrics are employed:

• •

CLIP [39]: CLIP [39] is a weakly supervised vision-language model with a web-scale dataset, which is trained to embed images and text into a joint latent space. CLIP encoding contains a semantic understanding of an image, and this model is widely used in various kinds of downstream tasks [40, 18, 20, 35, 37, 45]. We use CLIP for measuring visual similarities between images. The weight parameters from the publicly released model 444https://github.com/openai/CLIP are used. This model only takes RGB images.

• •

CL (SupContrast) [26]: We use a contrastive learning model [26] specially trained on the images from the Gibson [51] and MP3D [8] datasets. During the training, similar images are defined by the physical distance between the positions where the image was taken. This model is trained to encode the images from the same region (maximum 2m apart) into similar feature vectors, which show high cosine similarity. This model takes RGBD images.

• •

SSIM [47] (Structural similarity index): We measured SSIM between the depth images of the query image and the localized observation. This measure can represent the similarity between the 3D geometric structures in images.

• •

(Inverse) L1: We measure the direct pixel differences between the images. The sum of L1 losses from both RGB and depth images are used. As L1 represents the distance between the images, we inverse the value to use it as a similarity measure.

We tested 100 novel environment scenarios. As the query image is not from the same scene, the maximum visual similarity may vary depending on the given scene. We normalize each similarity metric by the maximum possible value from the given scene. The normalized value represents how much the found location is visually similar compared to the most visually-similar location. The experiment results are shown in Table 6. The random in the first row shows how much the metrics would appear when we select a random position.

We observed that showing high similarity on one metric did not always imply high similarity on other metrics. We report the experimental results when a query image is searched based on a specific metric (second to the fifth row of Table 6). We sampled all possible locations from the given scene and evaluated each image with the metrics. For example, the image which shows the highest CLIP visual similarity becomes the localized image by using CLIP as a search method (Max CLIP in Table 6). The CL column of the Max CLIP row shows that the image which has the highest CLIP similarity shows 61.36% of CL similarity compared to the maximum CL similarity. We can see that the maximum of each metric does not always mean the maximum in the other similarity metrics.

Meanwhile, the RNR-Map localizes a query image with consistently high value in all metrics (the last row). The RNR-Map shows general visual similarity in various metrics, in contrast to using a specific metric as a search method. We can infer that the RNR-Map finds a visually similar location even with a query image from a different environment.

Furthermore, we want to stress that the localization process $F_{\mathrm{loc}}$ of RNR-Map is done with a forward pass to the neural network with 56.8Hz of speed. Even with the fast speed, RNR-Map still achieves competitive similarity compared to the cases when all the possible images are compared with the query image one by one. We also plot the correlation matrix between the metrics and the probability value from RNR-Map in Figure 10. The RNR-Map shows positive correlations with the similarity metrics. The examples of the query image and the localized observation pairs are shown in Figure 11(b). In Figure 11(b), the image pairs are sorted by CL similarity and SSIM similarity. We observed that SSIM similarity is high when the overall 3D structures of the images are similar, while CL similarity is high when the overall colors of the images are similar.

Appendix D Navigation ablation studies

In this section, we report the ablation studies of the navigation module. The results are shown in Table 7, and each method is evaluated on Gibson-curved scenario from NRNS dataset [23] without noise setting.

Ground-truth information.

We provided the ground-truth information to the point navigation policy and the stopper module. With ground-truth information, the point-navigation policy can reach the target using the shortest path without collision. The stopper module with the ground-truth information has access to the geodesic distance to the target, so that it can detect the target location and take a stop action accurately. We can observe that navigation performance increases when given ground-truth information. The information about the distance to the target location is critical to the performance. Both the success rate and SPL dramatically increase because the ground-truth information enables efficient navigation without wandering or collision. Also, the agent has no risk of taking the false-stop action (terminating the episode even when the target is far) or overlooking the target location. We observed that the ground-truth point navigation policy often leads to the failure of an episode. Hence, in the medium scenarios, the model without the ground-truth point navigation policy (full model in the last row) shows better performance than the one with the ground-truth point navigation policy. This is because of the relatively inefficient path from our point-navigation module, which leads to a random chance to explore the environment more and find the target location. We anticipate that an improved point navigation policy and better $F_{stop}$ will largely improve the navigation performances.

Exploration strategy.

The exploration module in the navigation module selects where to explore in order to find the goal location. The exploration module draws a Voronoi graph on the occupancy map and selects a node as an exploration candidate from the created graph. Two criteria are used for selecting the exploration candidates, latent score and exploration score. The latent score is based on the localization heatmap value from $F_{loc}$. The occupancy map has three types of values, free (1), occupied (2), and unseen (0). The exploration score is based on the number of unseen pixels in the neighborhood of a pixel. The more unseen pixels are in the neighborhood, the more likely a location has been underexplored and has a high probability of discovering new areas. We ablated each score, and the results are shown in Table 7. Without the latent score, only the exploration score is used for exploring the environment. The exploration score helps the agent visit all the possible places in the scene. However, this score does not consider the information from the target, so the agent may not be able to find the target within the time limitation due to inefficient exploration. We can see that the agent finds the target more successfully when only the latent score is used. In the full model, we use the sum of the exploration score and the latent score. Using the combination of two scores helps the agent to find the target location efficiently. The latent score can guide the agent to the target location based on the RNR-Map. However, when there is not much information or clues about the target location, the exploration score can encourage the agent to explore the environment.

Appendix E Navigation on MP3D Dataset

We provide the image-goal navigation experiment results on MP3D dataset in Table 8. The experiments are done without noise setting, and we report the digits from [23], [49] for the baselines. The proposed RNR-Map-based navigation framework outperforms the baselines, except for the success rate in easy scenarios. The neural networks in RNR-Map are trained using Gibson [51] dataset. Based on the results, we can observe that the RNR-Map-based framework is generalizable to a different dataset, without any fine-tuning.

Appendix F Navigation Examples

We provide examples from the image-goal navigation episodes of RNR-Map in Figure 12. The latent scores usually highlight unexplored areas at the beginning of episodes. This is because the $F_{\mathrm{loc}}$ for navigation is trained to predict the closer area to the target given the partial information of the environment. We can see that the way to other rooms is highlighted in the first column of Figure 12(a) and 12(b). The agent follows the latent score and expands its RNR-Map according to the image observations. During the exploration, the agent observed the target-related region and successfully reached the target location. More examples are provided in the attached video.

Appendix G Implementation Details for the navigation submodules

Graph Generation

For efficient exploration planning, we discretize the region in the observed environment into a graph. Figure 13 shows the process of graph generation. This method is inspired by robot exploration literature [27, 53, 56, 34], which draws an (approximated) Voronoi graph on the occupancy map. Originally, the Voronoi graph consists of nodes that are equally distanced from the neighbor obstacles. We simplify this graph construction with the image skeletonizing method in the image processing library (skimage.morphology.skeletonize) [46]. This function is based on the image thinning algorithm proposed in [54]. As the skeleton only consists of lines, we need to determine which pixels will be the nodes. We select the pixels that have many neighbors as a node. The neighbor of more than two pixels indicates that the pixel is not on a line, but instead in the intersection of several lines. Then we split the skeleton based on the selected nodes, and determine the relationship between the nodes. The created graph is used for planning the exploration of the agent. The exploration module selects the node to explore, rather than sampling a pixel among the free spaces in the occupancy map.

Exploration Score

The exploration module evaluates each node in the graph with two criteria. The first one is the latent score, which is from the heatmap from $F_{loc}$ in the localization module. This heatmap highlights the place related to the target image. The second one is the exploration score, which is based on the number of unseen pixels in the neighborhood of the node. A location is more likely to have been underexplored and have a high likelihood of discovering new areas if there are more unseen pixels nearby. The examples of calculating the exploration score are presented in Figure 13. Drawing a set of rays centered at the node, we count the number of unseen pixels in the rays. We also evaluate whether a ray is blocked by an obstacle, and limit the length of the ray to less than the distance to the blocked obstacle. The number of unseen pixels is normalized into 0.0 to 1.0, and this value becomes the exploration score. Furthermore, we also calculate the distance from the agent to each node and add the inverse value to the exploration score. This encourages greedy exploration, which explores the near neighborhood first.

Point Navigation Module

The point navigation module takes the map position of the agent and the selected node position to explore. This module calculates the collision-free shortest path to the node based on the fast-marching method. We used the open-source library for the fast-marching method 555https://github.com/scikit-fmm/scikit-fmm. After a navigation path is obtained, the point navigation module outputs an appropriate action based on its relative pose to the path points.

Stopper Module

The stopper module has two components, $F_{stop}$ and the last-mile approaching function which is adopted from [49]. First, $F_{stop}$ determines whether the agent is near the target location. Second, if $F_{stop}$ found that the target is near the agent, we conduct keypoint-matching [42] between the current observation and the target image. If a sufficient number of keypoints are matched, we can infer the relative pose between the target location and the current position using the depth information. As proposed in [49], we use Perspective-n-Point [28], and RANSAC [19]. After the relative pose is determined, we set the local goal point, and the point navigation module navigates to the estimated target. If the number of matched keypoints decreases below a certain threshold (20 in our case), the last-mile approach is terminated, and the exploration module selects the next exploration target.