ChatDyn: Language-Driven Multi-Actor Dynamics Generation in Street Scenes

TL;DR

ChatDyn is a novel system that uses language instructions and multi-agent role-playing to generate realistic, controllable street scene dynamics involving vehicles and pedestrians, advancing simulation capabilities.

Contribution

It introduces the first comprehensive method for generating interactive, controllable street scene dynamics based on natural language instructions using multi-LLM-agent planning.

Findings

Outperforms previous methods on key subtasks

Generates realistic pedestrian and vehicle dynamics

Enables precise control through complex language instructions

Abstract

Generating realistic and interactive dynamics of traffic participants according to specific instruction is critical for street scene simulation. However, there is currently a lack of a comprehensive method that generates realistic dynamics of different types of participants including vehicles and pedestrians, with different kinds of interactions between them. In this paper, we introduce ChatDyn, the first system capable of generating interactive, controllable and realistic participant dynamics in street scenes based on language instructions. To achieve precise control through complex language, ChatDyn employs a multi-LLM-agent role-playing approach, which utilizes natural language inputs to plan the trajectories and behaviors for different traffic participants. To generate realistic fine-grained dynamics based on the planning, ChatDyn designs two novel executors: the PedExecutor, a unified multi-task executor that generates realistic pedestrian dynamics under different task plannings; and the VehExecutor, a physical transition-based policy that generates physically plausible vehicle dynamics. Extensive experiments show that ChatDyn can generate realistic driving scene dynamics with multiple vehicles and pedestrians, and significantly outperforms previous methods on subtasks. Code and model will be available at https://vfishc.github.io/chatdyn.