ObserveNet Control: A Vision-Dynamics Learning Approach to Predictive Control in Autonomous Vehicles

TL;DR

ObserveNet Control introduces a vision-dynamics neural network that predicts future sensory data for autonomous vehicle control, enabling safer and more effective motion planning in complex driving scenarios through self-supervised learning.

Contribution

It presents a novel deep learning approach that predicts raw sensory data for autonomous driving, integrating it with a temporal planner for improved motion control.

Findings

Effective prediction of sensory data up to 10s ahead

Successful deployment in simulation and real-world tests

Outperforms baseline and state-of-the-art methods in aggressive driving scenarios

Abstract

A key component in autonomous driving is the ability of the self-driving car to understand, track and predict the dynamics of the surrounding environment. Although there is significant work in the area of object detection, tracking and observations prediction, there is no prior work demonstrating that raw observations prediction can be used for motion planning and control. In this paper, we propose ObserveNet Control, which is a vision-dynamics approach to the predictive control problem of autonomous vehicles. Our method is composed of a: i) deep neural network able to confidently predict future sensory data on a time horizon of up to 10s and ii) a temporal planner designed to compute a safe vehicle state trajectory based on the predicted sensory data. Given the vehicle's historical state and sensing data in the form of Lidar point clouds, the method aims to learn the dynamics of the observed driving environment in a self-supervised manner, without the need to manually specify training labels. The experiments are performed both in simulation and real-life, using CARLA and RovisLab's AMTU mobile platform as a 1:4 scaled model of a car. We evaluate the capabilities of ObserveNet Control in aggressive driving contexts, such as overtaking maneuvers or side cut-off situations, while comparing the results with a baseline Dynamic Window Approach (DWA) and two state-of-the-art imitation learning systems, that is, Learning by Cheating (LBC) and World on Rails (WOR).