Grounding Human-to-Vehicle Advice for Self-driving Vehicles

TL;DR

This paper introduces a method for improving self-driving vehicle control by incorporating natural language advice from humans, which guides the vehicle's attention and actions, enhancing safety and performance.

Contribution

It presents the first end-to-end vehicle controller that accepts and utilizes natural language advice, linking advice to visual attention and control decisions.

Findings

Advice improves driving performance.

Model attends to salient objects based on advice.

Dataset enables training and evaluation of advice-guided driving models.

Abstract

Recent success suggests that deep neural control networks are likely to be a key component of self-driving vehicles. These networks are trained on large datasets to imitate human actions, but they lack semantic understanding of image contents. This makes them brittle and potentially unsafe in situations that do not match training data. Here, we propose to address this issue by augmenting training data with natural language advice from a human. Advice includes guidance about what to do and where to attend. We present the first step toward advice giving, where we train an end-to-end vehicle controller that accepts advice. The controller adapts the way it attends to the scene (visual attention) and the control (steering and speed). Attention mechanisms tie controller behavior to salient objects in the advice. We evaluate our model on a novel advisable driving dataset with manually annotated human-to-vehicle advice called Honda Research Institute-Advice Dataset (HAD). We show that taking advice improves the performance of the end-to-end network, while the network cues on a variety of visual features that are provided by advice. The dataset is available at https://usa.honda-ri.com/HAD.