TraceLLM: Leveraging Large Language Models with Prompt Engineering for Enhanced Requirements Traceability

TL;DR

TraceLLM introduces a systematic prompt engineering framework leveraging large language models to improve requirements traceability, outperforming traditional methods and enabling semi-automated workflows across diverse domains.

Contribution

The paper presents a novel prompt engineering framework, TraceLLM, that enhances LLM-based requirements traceability with systematic prompt design, demonstration selection, and evaluation across multiple datasets.

Findings

TraceLLM achieves state-of-the-art F2 scores in traceability tasks.

Prompt quality significantly impacts traceability performance.

Diversity-based demonstration sampling improves link extraction accuracy.

Abstract

Requirements traceability, the process of establishing and maintaining relationships between requirements and various software development artifacts, is paramount for ensuring system integrity and fulfilling requirements throughout the Software Development Life Cycle (SDLC). Traditional methods, including manual and information retrieval models, are labor-intensive, error-prone, and limited by low precision. Recently, Large Language Models (LLMs) have demonstrated potential for supporting software engineering tasks through advanced language comprehension. However, a substantial gap exists in the systematic design and evaluation of prompts tailored to extract accurate trace links. This paper introduces TraceLLM, a systematic framework for enhancing requirements traceability through prompt engineering and demonstration selection. Our approach incorporates rigorous dataset splitting, iterative prompt refinement, enrichment with contextual roles and domain knowledge, and evaluation across zero- and few-shot settings. We assess prompt generalization and robustness using eight state-of-the-art LLMs on four benchmark datasets representing diverse domains (aerospace, healthcare) and artifact types (requirements, design elements, test cases, regulations). TraceLLM achieves state-of-the-art F2 scores, outperforming traditional IR baselines, fine-tuned models, and prior LLM-based methods. We also explore the impact of demonstration selection strategies, identifying label-aware, diversity-based sampling as particularly effective. Overall, our findings highlight that traceability performance depends not only on model capacity but also critically on the quality of prompt engineering. In addition, the achieved performance suggests that TraceLLM can support semi-automated traceability workflows in which candidate links are reviewed and validated by human analysts.