Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

TL;DR

This paper introduces a novel end-to-end method for generating bird's-eye-view scene representations from images captured by arbitrary camera rigs, improving perception accuracy and robustness for autonomous vehicle navigation.

Contribution

The authors propose a new architecture that lifts images into 3D frustums and splats them into a unified bird's-eye-view, enabling robust multi-camera fusion and end-to-end motion planning.

Findings

Outperforms all baselines on object and map segmentation tasks

Robust to calibration errors in camera rigs

Enables interpretable motion planning using bird's-eye-view representations

Abstract

The goal of perception for autonomous vehicles is to extract semantic representations from multiple sensors and fuse these representations into a single "bird's-eye-view" coordinate frame for consumption by motion planning. We propose a new end-to-end architecture that directly extracts a bird's-eye-view representation of a scene given image data from an arbitrary number of cameras. The core idea behind our approach is to "lift" each image individually into a frustum of features for each camera, then "splat" all frustums into a rasterized bird's-eye-view grid. By training on the entire camera rig, we provide evidence that our model is able to learn not only how to represent images but how to fuse predictions from all cameras into a single cohesive representation of the scene while being robust to calibration error. On standard bird's-eye-view tasks such as object segmentation and map segmentation, our model outperforms all baselines and prior work. In pursuit of the goal of learning dense representations for motion planning, we show that the representations inferred by our model enable interpretable end-to-end motion planning by "shooting" template trajectories into a bird's-eye-view cost map output by our network. We benchmark our approach against models that use oracle depth from lidar. Project page with code: https://nv-tlabs.github.io/lift-splat-shoot .