Natural-language-driven Simulation Benchmark and Copilot for Efficient Production of Object Interactions in Virtual Road Scenes

TL;DR

This paper introduces a natural-language-driven simulation framework and a benchmark dataset for efficiently generating complex object interactions in virtual road scenes, aiding autonomous driving system development.

Contribution

It presents the L2I benchmark dataset with 120,000 descriptions and develops SimCopilot, a model translating natural language into visual scene interactions, advancing NLD simulation research.

Findings

SimCopilot effectively controls object motions in virtual scenes.

The dataset enables training models to generate complex interactions.

Results show good generalization across different road topologies.

Abstract

We advocate the idea of the natural-language-driven(NLD) simulation to efficiently produce the object interactions between multiple objects in the virtual road scenes, for teaching and testing the autonomous driving systems that should take quick action to avoid collision with obstacles with unpredictable motions. The NLD simulation allows the brief natural-language description to control the object interactions, significantly reducing the human efforts for creating a large amount of interaction data. To facilitate the research of NLD simulation, we collect the Language-to-Interaction(L2I) benchmark dataset with 120,000 natural-language descriptions of object interactions in 6 common types of road topologies. Each description is associated with the programming code, which the graphic render can use to visually reconstruct the object interactions in the virtual scenes. As a methodology contribution, we design SimCopilot to translate the interaction descriptions to the renderable code. We use the L2I dataset to evaluate SimCopilot's abilities to control the object motions, generate complex interactions, and generalize interactions across road topologies. The L2I dataset and the evaluation results motivate the relevant research of the NLD simulation.