UAV See, UGV Do: Aerial Imagery and Virtual Teach Enabling Zero-Shot Ground Vehicle Repeat

TL;DR

This paper introduces Virtual Teach and Repeat (VirT&R), a novel method enabling GPS-denied, zero-shot autonomous ground vehicle navigation using aerial imagery and neural radiance fields to create high-fidelity environment models for path planning and execution.

Contribution

VirT&R extends the Teach and Repeat framework by integrating NeRF-based environment modeling for autonomous navigation without manual path teaching in untraversed areas.

Findings

Achieved RMSE of around 19.5 cm in path tracking.

Demonstrated similar performance to traditional LT&R without manual teaching.

Validated on over 12 km of autonomous driving data.

Abstract

This paper presents Virtual Teach and Repeat (VirT&R): an extension of the Teach and Repeat (T&R) framework that enables GPS-denied, zero-shot autonomous ground vehicle navigation in untraversed environments. VirT&R leverages aerial imagery captured for a target environment to train a Neural Radiance Field (NeRF) model so that dense point clouds and photo-textured meshes can be extracted. The NeRF mesh is used to create a high-fidelity simulation of the environment for piloting an unmanned ground vehicle (UGV) to virtually define a desired path. The mission can then be executed in the actual target environment by using NeRF-generated point cloud submaps associated along the path and an existing LiDAR Teach and Repeat (LT&R) framework. We benchmark the repeatability of VirT&R on over 12 km of autonomous driving data using physical markings that allow a sim-to-real lateral path-tracking error to be obtained and compared with LT&R. VirT&R achieved measured root mean squared errors (RMSE) of 19.5 cm and 18.4 cm in two different environments, which are slightly less than one tire width (24 cm) on the robot used for testing, and respective maximum errors were 39.4 cm and 47.6 cm. This was done using only the NeRF-derived teach map, demonstrating that VirT&R has similar closed-loop path-tracking performance to LT&R but does not require a human to manually teach the path to the UGV in the actual environment.