ScenicNL: Generating Probabilistic Scenario Programs from Natural Language

TL;DR

ScenicNL is an AI system that converts natural language police crash reports into probabilistic scenario programs using large language models, enabling better simulation of rare events in cyber-physical systems.

Contribution

It introduces a novel method for translating natural language crash reports into probabilistic scenario programs with LLMs, addressing limitations of existing prompting techniques.

Findings

Successfully generates semantically meaningful Scenic code from crash reports.

Demonstrates the system's effectiveness on real-world autonomous vehicle crash data.

Provides insights into the challenges of reasoning about probabilistic scenarios with LLMs.

Abstract

For cyber-physical systems (CPS), including robotics and autonomous vehicles, mass deployment has been hindered by fatal errors that occur when operating in rare events. To replicate rare events such as vehicle crashes, many companies have created logging systems and employed crash reconstruction experts to meticulously recreate these valuable events in simulation. However, in these methods, "what if" questions are not easily formulated and answered. We present ScenarioNL, an AI System for creating scenario programs from natural language. Specifically, we generate these programs from police crash reports. Reports normally contain uncertainty about the exact details of the incidents which we represent through a Probabilistic Programming Language (PPL), Scenic. By using Scenic, we can clearly and concisely represent uncertainty and variation over CPS behaviors, properties, and interactions. We demonstrate how commonplace prompting techniques with the best Large Language Models (LLM) are incapable of reasoning about probabilistic scenario programs and generating code for low-resource languages such as Scenic. Our system is comprised of several LLMs chained together with several kinds of prompting strategies, a compiler, and a simulator. We evaluate our system on publicly available autonomous vehicle crash reports in California from the last five years and share insights into how we generate code that is both semantically meaningful and syntactically correct.