Hallucination in LLM-Based Code Generation: An Automotive Case Study

TL;DR

This study examines hallucination issues in LLM-based automotive code generation, revealing high error rates and the importance of context-rich prompts for correct outputs, emphasizing safety-critical application challenges.

Contribution

It provides a detailed case study on hallucinations in automotive code generation with LLMs, highlighting the impact of prompt complexity on output correctness.

Findings

High frequency of syntax and reference errors in models

Context-rich prompts improve correctness in GPT-4.1 and GPT-4o

Simpler prompts often fail to produce valid code

Abstract

Large Language Models (LLMs) have shown significant potential in automating code generation tasks offering new opportunities across software engineering domains. However, their practical application remains limited due to hallucinations - outputs that appear plausible but are factually incorrect, unverifiable or nonsensical. This paper investigates hallucination phenomena in the context of code generation with a specific focus on the automotive domain. A case study is presented that evaluates multiple code LLMs for three different prompting complexities ranging from a minimal one-liner prompt to a prompt with Covesa Vehicle Signal Specifications (VSS) as additional context and finally to a prompt with an additional code skeleton. The evaluation reveals a high frequency of syntax violations, invalid reference errors and API knowledge conflicts in state-of-the-art models GPT-4.1, Codex and GPT-4o. Among the evaluated models, only GPT-4.1 and GPT-4o were able to produce a correct solution when given the most context-rich prompt. Simpler prompting strategies failed to yield a working result, even after multiple refinement iterations. These findings highlight the need for effective mitigation techniques to ensure the safe and reliable use of LLM generated code, especially in safety-critical domains such as automotive software systems.