Hard Cases Detection in Motion Prediction by Vision-Language Foundation Models

TL;DR

This paper explores using vision-language foundation models like GPT-4v to detect hard cases in autonomous driving scenarios, improving safety and training efficiency by identifying challenging situations in traffic prediction tasks.

Contribution

It introduces a novel pipeline leveraging VLMs for hard case detection in autonomous driving, enhancing data selection and model robustness.

Findings

VLMs effectively identify challenging traffic scenarios.

The pipeline improves training efficiency for motion prediction models.

Demonstrated on NuScenes dataset with state-of-the-art methods.

Abstract

Addressing hard cases in autonomous driving, such as anomalous road users, extreme weather conditions, and complex traffic interactions, presents significant challenges. To ensure safety, it is crucial to detect and manage these scenarios effectively for autonomous driving systems. However, the rarity and high-risk nature of these cases demand extensive, diverse datasets for training robust models. Vision-Language Foundation Models (VLMs) have shown remarkable zero-shot capabilities as being trained on extensive datasets. This work explores the potential of VLMs in detecting hard cases in autonomous driving. We demonstrate the capability of VLMs such as GPT-4v in detecting hard cases in traffic participant motion prediction on both agent and scenario levels. We introduce a feasible pipeline where VLMs, fed with sequential image frames with designed prompts, effectively identify challenging agents or scenarios, which are verified by existing prediction models. Moreover, by taking advantage of this detection of hard cases by VLMs, we further improve the training efficiency of the existing motion prediction pipeline by performing data selection for the training samples suggested by GPT. We show the effectiveness and feasibility of our pipeline incorporating VLMs with state-of-the-art methods on NuScenes datasets. The code is accessible at https://github.com/KTH-RPL/Detect_VLM.