A Tool for Benchmarking Large Language Models' Robustness in Assessing the Realism of Driving Scenarios

TL;DR

This paper introduces DriveRLR, a benchmark tool that uses large language models to evaluate the realism of driving scenarios, aiding autonomous vehicle testing and safety validation.

Contribution

The paper presents DriveRLR, a novel benchmark for assessing LLM robustness in driving scenario realism evaluation, with validation on multiple state-of-the-art models.

Findings

DriveRLR effectively differentiates LLM robustness levels.

It demonstrates practical utility in scenario realism assessment.

It can guide scenario generation for autonomous driving testing.

Abstract

In recent years, autonomous driving systems have made significant progress, yet ensuring their safety remains a key challenge. To this end, scenario-based testing offers a practical solution, and simulation-based methods have gained traction due to the high cost and risk of real-world testing. However, evaluating the realism of simulated scenarios remains difficult, creating demand for effective assessment methods. Recent advances show that Large Language Models (LLMs) possess strong reasoning and generalization capabilities, suggesting their potential in assessing scenario realism through scenario-related textual prompts. Motivated by this, we propose DriveRLR, a benchmark tool to assess the robustness of LLMs in evaluating the realism of driving scenarios. DriveRLR generates mutated scenario variants, constructs prompts, which are then used to assess a given LLM's ability and robustness in determining the realism of driving scenarios. We validate DriveRLR on the DeepScenario dataset using three state-of-the-art LLMs: GPT-5, Llama 4 Maverick, and Mistral Small 3.2. Results show that DriveRLR effectively reveals differences in the robustness of various LLMs, demonstrating its effectiveness and practical value in scenario realism assessment. Beyond LLM robustness evaluation, DriveRLR can serve as a practical component in applications such as an objective function to guide scenario generation, supporting simulation-based ADS testing workflows.