Model-Based Imitation Learning for Urban Driving

TL;DR

MILE is a novel model-based imitation learning approach that uses 3D geometry and high-resolution videos to learn a compact environment model and policy for urban autonomous driving, improving generalization and interpretability.

Contribution

It introduces the first camera-only method that jointly models static and dynamic scenes along with ego-behavior for urban driving, trained entirely offline.

Findings

Achieves 31% higher driving score on CARLA in new environments.

Predicts diverse, plausible states and actions with interpretability.

Executes complex maneuvers from imagined plans.

Abstract

An accurate model of the environment and the dynamic agents acting in it offers great potential for improving motion planning. We present MILE: a Model-based Imitation LEarning approach to jointly learn a model of the world and a policy for autonomous driving. Our method leverages 3D geometry as an inductive bias and learns a highly compact latent space directly from high-resolution videos of expert demonstrations. Our model is trained on an offline corpus of urban driving data, without any online interaction with the environment. MILE improves upon prior state-of-the-art by 31% in driving score on the CARLA simulator when deployed in a completely new town and new weather conditions. Our model can predict diverse and plausible states and actions, that can be interpretably decoded to bird's-eye view semantic segmentation. Further, we demonstrate that it can execute complex driving manoeuvres from plans entirely predicted in imagination. Our approach is the first camera-only method that models static scene, dynamic scene, and ego-behaviour in an urban driving environment. The code and model weights are available at https://github.com/wayveai/mile.