Incorporating Voice Instructions in Model-Based Reinforcement Learning for Self-Driving Cars

TL;DR

This paper introduces a method that integrates natural language voice instructions into model-based deep reinforcement learning to improve the training efficiency of self-driving cars, making human-AI interaction more natural and effective.

Contribution

It proposes a novel approach to incorporate voice instructions into DRL for autonomous vehicles, enhancing training speed and human-in-the-loop learning.

Findings

Voice instructions significantly speed up learning.

Improved training efficiency in CARLA simulator.

Enhanced human-AI communication during training.

Abstract

This paper presents a novel approach that supports natural language voice instructions to guide deep reinforcement learning (DRL) algorithms when training self-driving cars. DRL methods are popular approaches for autonomous vehicle (AV) agents. However, most existing methods are sample- and time-inefficient and lack a natural communication channel with the human expert. In this paper, how new human drivers learn from human coaches motivates us to study new ways of human-in-the-loop learning and a more natural and approachable training interface for the agents. We propose incorporating natural language voice instructions (NLI) in model-based deep reinforcement learning to train self-driving cars. We evaluate the proposed method together with a few state-of-the-art DRL methods in the CARLA simulator. The results show that NLI can help ease the training process and significantly boost the agents' learning speed.