Prompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability

TL;DR

This paper investigates prompt engineering techniques to optimize Large Language Models for automated software traceability, providing strategies and insights to improve link prediction accuracy and guide future research in the field.

Contribution

It introduces detailed prompt construction strategies and multiple approaches for leveraging LLMs to enhance traceability link generation beyond zero-shot methods.

Findings

Improved ranking of candidate links through prompt refinement

Strategies for effective prompt design in traceability tasks

Lessons learned for future prompt engineering efforts

Abstract

Large Language Models (LLMs) have the potential to revolutionize automated traceability by overcoming the challenges faced by previous methods and introducing new possibilities. However, the optimal utilization of LLMs for automated traceability remains unclear. This paper explores the process of prompt engineering to extract link predictions from an LLM. We provide detailed insights into our approach for constructing effective prompts, offering our lessons learned. Additionally, we propose multiple strategies for leveraging LLMs to generate traceability links, improving upon previous zero-shot methods on the ranking of candidate links after prompt refinement. The primary objective of this paper is to inspire and assist future researchers and engineers by highlighting the process of constructing traceability prompts to effectively harness LLMs for advancing automatic traceability.