Deep Learning of Robotic Tasks without a Simulator using Strong and Weak Human Supervision

TL;DR

This paper introduces a comprehensive deep learning framework for training autonomous highway steering agents directly from human supervision without relying on a simulator, emphasizing safety and multi-stage learning.

Contribution

It presents a novel multi-stage training scheme combining unsupervised, supervised, and reinforcement learning, including a safety network to prevent catastrophic errors.

Findings

Successfully trained an autonomous highway steering agent without a simulator

Demonstrated the importance of the last four training elements for vision-based control

Introduced a safety network to prevent catastrophic mistakes during reinforcement learning

Abstract

We propose a scheme for training a computerized agent to perform complex human tasks such as highway steering. The scheme is designed to follow a natural learning process whereby a human instructor teaches a computerized trainee. The learning process consists of five elements: (i) unsupervised feature learning; (ii) supervised imitation learning; (iii) supervised reward induction; (iv) supervised safety module construction; and (v) reinforcement learning. We implemented the last four elements of the scheme using deep convolutional networks and applied it to successfully create a computerized agent capable of autonomous highway steering over the well-known racing game Assetto Corsa. We demonstrate that the use of the last four elements is essential to effectively carry out the steering task using vision alone, without access to a driving simulator internals, and operating in wall-clock time. This is made possible also through the introduction of a safety network, a novel way for preventing the agent from performing catastrophic mistakes during the reinforcement learning stage.