Commands 4 Autonomous Vehicles (C4AV) Workshop Summary

TL;DR

This paper introduces a new challenge for visual grounding in autonomous vehicles using natural language commands, based on the Talk2Car dataset, and analyzes model performance and failure cases.

Contribution

It presents the C4AV challenge, compares it with existing datasets, and analyzes what makes models successful or prone to failure in this autonomous vehicle context.

Findings

Benchmark differs from existing datasets in scope and complexity.

Successful models leverage specific visual and linguistic features.

Failure cases reveal limitations in current visual grounding approaches.

Abstract

The task of visual grounding requires locating the most relevant region or object in an image, given a natural language query. So far, progress on this task was mostly measured on curated datasets, which are not always representative of human spoken language. In this work, we deviate from recent, popular task settings and consider the problem under an autonomous vehicle scenario. In particular, we consider a situation where passengers can give free-form natural language commands to a vehicle which can be associated with an object in the street scene. To stimulate research on this topic, we have organized the \emph{Commands for Autonomous Vehicles} (C4AV) challenge based on the recent \emph{Talk2Car} dataset (URL: https://www.aicrowd.com/challenges/eccv-2020-commands-4-autonomous-vehicles). This paper presents the results of the challenge. First, we compare the used benchmark against existing datasets for visual grounding. Second, we identify the aspects that render top-performing models successful, and relate them to existing state-of-the-art models for visual grounding, in addition to detecting potential failure cases by evaluating on carefully selected subsets. Finally, we discuss several possibilities for future work.