Smarter, not Bigger: Fine-Tuned RAG-Enhanced LLMs for Automotive HIL Testing

TL;DR

This paper introduces HIL-GPT, a retrieval-augmented LLM system for automotive HIL testing that uses fine-tuned, domain-specific models and semantic retrieval to improve efficiency and user satisfaction.

Contribution

It presents a novel RAG system with domain-adapted LLMs and a scalable retrieval approach, demonstrating superior performance over larger models in automotive HIL testing.

Findings

Fine-tuned compact models outperform larger models in accuracy and efficiency.

RAG-enhanced assistants increase perceived helpfulness and satisfaction.

The approach enables scalable, traceable test artifact retrieval in industrial settings.

Abstract

Hardware-in-the-Loop (HIL) testing is essential for automotive validation but suffers from fragmented and underutilized test artifacts. This paper presents HIL-GPT, a retrieval-augmented generation (RAG) system integrating domain-adapted large language models (LLMs) with semantic retrieval. HIL-GPT leverages embedding fine-tuning using a domain-specific dataset constructed via heuristic mining and LLM-assisted synthesis, combined with vector indexing for scalable, traceable test case and requirement retrieval. Experiments show that fine-tuned compact models, such as \texttt{bge-base-en-v1.5}, achieve a superior trade-off between accuracy, latency, and cost compared to larger models, challenging the notion that bigger is always better. An A/B user study further confirms that RAG-enhanced assistants improve perceived helpfulness, truthfulness, and satisfaction over general-purpose LLMs. These findings provide insights for deploying efficient, domain-aligned LLM-based assistants in industrial HIL environments.