Investigating the Role of LLMs Hyperparameter Tuning and Prompt Engineering to Support Domain Modeling

TL;DR

This paper investigates how hyperparameter tuning and prompt engineering can enhance the performance of LLMs, specifically Llama 3.1, in generating accurate domain models from text across multiple application domains.

Contribution

It demonstrates that combining hyperparameter tuning with prompt engineering improves domain modeling accuracy in LLMs, with a focus on medical data and diverse applications.

Findings

Hyperparameter tuning significantly improved model quality in medical domain.

Prompt engineering combined with tuning enhanced results across multiple domains.

Solutions were not universally applicable but showed broad improvements.

Abstract

The introduction of large language models (LLMs) has enhanced automation in software engineering tasks, including in Model Driven Engineering (MDE). However, using general-purpose LLMs for domain modeling has its limitations. One approach is to adopt fine-tuned models, but this requires significant computational resources and can lead to issues like catastrophic forgetting. This paper explores how hyperparameter tuning and prompt engineering can improve the accuracy of the Llama 3.1 model for generating domain models from textual descriptions. We use search-based methods to tune hyperparameters for a specific medical data model, resulting in a notable quality improvement over the baseline LLM. We then test the optimized hyperparameters across ten diverse application domains. While the solutions were not universally applicable, we demonstrate that combining hyperparameter tuning with prompt engineering can enhance results across nearly all examined domain models.