Conversational Code Generation: a Case Study of Designing a Dialogue System for Generating Driving Scenarios for Testing Autonomous Vehicles

TL;DR

This paper presents a dialogue-based natural language interface leveraging large language models to help non-coding experts generate driving scenarios for autonomous vehicle testing, significantly improving success rates.

Contribution

It introduces a novel conversational system for scenario generation in autonomous vehicle testing, demonstrating the effectiveness of dialogue in improving success rates with limited training data.

Findings

Dialogue increases scenario generation success rate by 4.5 times

Feasibility of converting utterances to symbolic programs with small datasets

Human experiments confirm importance of conversation in simulation success

Abstract

Cyber-physical systems like autonomous vehicles are tested in simulation before deployment, using domain-specific programs for scenario specification. To aid the testing of autonomous vehicles in simulation, we design a natural language interface, using an instruction-following large language model, to assist a non-coding domain expert in synthesising the desired scenarios and vehicle behaviours. We show that using it to convert utterances to the symbolic program is feasible, despite the very small training dataset. Human experiments show that dialogue is critical to successful simulation generation, leading to a 4.5 times higher success rate than a generation without engaging in extended conversation.