Are requirements really all you need? A case study of LLM-driven configuration code generation for automotive simulations

TL;DR

This paper evaluates a state-of-the-art LLM's ability to translate high-level automotive requirements into simulation configuration code, highlighting its strengths and limitations in handling real-world industrial tasks.

Contribution

It provides a case study on LLM-driven code generation for automotive simulations, assessing practical applicability and understanding of complex instructions.

Findings

LLMs can generate functional simulation code from abstract requirements

Performance varies depending on instruction clarity and complexity

The study identifies gaps in LLM understanding of domain-specific standards

Abstract

Large Language Models (LLMs) are taking many industries by storm. They possess impressive reasoning capabilities and are capable of handling complex problems, as shown by their steadily improving scores on coding and mathematical benchmarks. However, are the models currently available truly capable of addressing real-world challenges, such as those found in the automotive industry? How well can they understand high-level, abstract instructions? Can they translate these instructions directly into functional code, or do they still need help and supervision? In this work, we put one of the current state-of-the-art models to the test. We evaluate its performance in the task of translating abstract requirements, extracted from automotive standards and documents, into configuration code for CARLA simulations.