End-to-end Learning of Driving Models from Large-scale Video Datasets

TL;DR

This paper proposes an end-to-end deep learning approach to predict vehicle motion from large-scale crowd-sourced video data, using a novel FCN-LSTM architecture that incorporates scene segmentation to improve robustness and generalization.

Contribution

It introduces a new FCN-LSTM architecture for monocular vehicle motion prediction trained on large-scale crowd-sourced data, enhancing robustness and diversity in visuomotor models.

Findings

Effective prediction of vehicle egomotion from monocular video

Improved performance using scene segmentation as a side task

Demonstrates scalability with large crowd-sourced datasets

Abstract

Robust perception-action models should be learned from training data with diverse visual appearances and realistic behaviors, yet current approaches to deep visuomotor policy learning have been generally limited to in-situ models learned from a single vehicle or a simulation environment. We advocate learning a generic vehicle motion model from large scale crowd-sourced video data, and develop an end-to-end trainable architecture for learning to predict a distribution over future vehicle egomotion from instantaneous monocular camera observations and previous vehicle state. Our model incorporates a novel FCN-LSTM architecture, which can be learned from large-scale crowd-sourced vehicle action data, and leverages available scene segmentation side tasks to improve performance under a privileged learning paradigm.