Predicting Physical World Destinations for Commands Given to Self-Driving Cars

TL;DR

This paper advances understanding of natural language commands for self-driving cars by focusing on predicting the physical destination in 3D space, enhancing human-vehicle interaction and safety.

Contribution

It introduces a novel annotation scheme for 3D destinations and proposes a model that outperforms existing baselines in predicting these destinations.

Findings

Proposed a new 3D destination annotation method

Developed a model surpassing prior baselines

Improved accuracy in destination prediction

Abstract

In recent years, we have seen significant steps taken in the development of self-driving cars. Multiple companies are starting to roll out impressive systems that work in a variety of settings. These systems can sometimes give the impression that full self-driving is just around the corner and that we would soon build cars without even a steering wheel. The increase in the level of autonomy and control given to an AI provides an opportunity for new modes of human-vehicle interaction. However, surveys have shown that giving more control to an AI in self-driving cars is accompanied by a degree of uneasiness by passengers. In an attempt to alleviate this issue, recent works have taken a natural language-oriented approach by allowing the passenger to give commands that refer to specific objects in the visual scene. Nevertheless, this is only half the task as the car should also understand the physical destination of the command, which is what we focus on in this paper. We propose an extension in which we annotate the 3D destination that the car needs to reach after executing the given command and evaluate multiple different baselines on predicting this destination location. Additionally, we introduce a model that outperforms the prior works adapted for this particular setting.