AGENTS-LLM: Augmentative GENeration of Challenging Traffic Scenarios with an Agentic LLM Framework

TL;DR

This paper presents AGENTS-LLM, a novel framework that uses agentic large language models to generate challenging traffic scenarios from natural language descriptions, improving scalability and control in autonomous driving testing.

Contribution

Introduces an agentic LLM-based framework for traffic scenario augmentation that offers fine-grained control and high-quality outputs with smaller models, addressing scalability issues.

Findings

High-quality scenario generation comparable to manual creation

Effective control over scenario details via natural language prompts

Maintains performance with smaller, cost-effective LLMs

Abstract

Rare, yet critical, scenarios pose a significant challenge in testing and evaluating autonomous driving planners. Relying solely on real-world driving scenes requires collecting massive datasets to capture these scenarios. While automatic generation of traffic scenarios appears promising, data-driven models require extensive training data and often lack fine-grained control over the output. Moreover, generating novel scenarios from scratch can introduce a distributional shift from the original training scenes which undermines the validity of evaluations especially for learning-based planners. To sidestep this, recent work proposes to generate challenging scenarios by augmenting original scenarios from the test set. However, this involves the manual augmentation of scenarios by domain experts. An approach that is unable to meet the demands for scale in the evaluation of self-driving systems. Therefore, this paper introduces a novel LLM-agent based framework for augmenting real-world traffic scenarios using natural language descriptions, addressing the limitations of existing methods. A key innovation is the use of an agentic design, enabling fine-grained control over the output and maintaining high performance even with smaller, cost-effective LLMs. Extensive human expert evaluation demonstrates our framework's ability to accurately adhere to user intent, generating high quality augmented scenarios comparable to those created manually.