Learning autonomous driving from aerial imagery

TL;DR

This paper introduces a novel approach for autonomous ground vehicle perception and control using aerial imagery and Neural Radiance Fields (NeRF) to synthesize views, enabling end-to-end learning and relocalization without extensive simulator setup.

Contribution

It proposes using NeRF as an intermediate representation for view synthesis to improve autonomous driving from aerial imagery, reducing setup costs and enabling real-world relocalization.

Findings

NeRF-based view synthesis improves policy training for autonomous vehicles.

The method successfully relocalizes vehicles in real-world environments.

Demonstrated effectiveness in a custom mini-city environment.

Abstract

In this work, we consider the problem of learning end to end perception to control for ground vehicles solely from aerial imagery. Photogrammetric simulators allow the synthesis of novel views through the transformation of pre-generated assets into novel views.However, they have a large setup cost, require careful collection of data and often human effort to create usable simulators. We use a Neural Radiance Field (NeRF) as an intermediate representation to synthesize novel views from the point of view of a ground vehicle. These novel viewpoints can then be used for several downstream autonomous navigation applications. In this work, we demonstrate the utility of novel view synthesis though the application of training a policy for end to end learning from images and depth data. In a traditional real to sim to real framework, the collected data would be transformed into a visual simulator which could then be used to generate novel views. In contrast, using a NeRF allows a compact representation and the ability to optimize over the parameters of the visual simulator as more data is gathered in the environment. We demonstrate the efficacy of our method in a custom built mini-city environment through the deployment of imitation policies on robotic cars. We additionally consider the task of place localization and demonstrate that our method is able to relocalize the car in the real world.