Reality Bites: Assessing the Realism of Driving Scenarios with Large Language Models

TL;DR

This paper evaluates the ability of large language models to assess the realism of driving scenarios, demonstrating that GPT outperforms other models in robustness across various conditions, which is vital for autonomous driving testing.

Contribution

It introduces an empirical evaluation framework for assessing LLMs' effectiveness in judging driving scenario realism, highlighting GPT's superior robustness.

Findings

GPT achieved highest robustness across scenarios

Weather and road conditions affect LLM performance

Mistral performed the worst in assessments

Abstract

Large Language Models (LLMs) are demonstrating outstanding potential for tasks such as text generation, summarization, and classification. Given that such models are trained on a humongous amount of online knowledge, we hypothesize that LLMs can assess whether driving scenarios generated by autonomous driving testing techniques are realistic, i.e., being aligned with real-world driving conditions. To test this hypothesis, we conducted an empirical evaluation to assess whether LLMs are effective and robust in performing the task. This reality check is an important step towards devising LLM-based autonomous driving testing techniques. For our empirical evaluation, we selected 64 realistic scenarios from \deepscenario--an open driving scenario dataset. Next, by introducing minor changes to them, we created 512 additional realistic scenarios, to form an overall dataset of 576 scenarios. With this dataset, we evaluated three LLMs (\gpt, \llama, and \mistral) to assess their robustness in assessing the realism of driving scenarios. Our results show that: (1) Overall, \gpt achieved the highest robustness compared to \llama and \mistral, consistently throughout almost all scenarios, roads, and weather conditions; (2) \mistral performed the worst consistently; (3) \llama achieved good results under certain conditions; and (4) roads and weather conditions do influence the robustness of the LLMs.