Classifier or Prompt: A Case Study on Legal Requirements Traceability

TL;DR

This paper compares automated language model-based solutions for legal requirements traceability, showing that tailored prompts and classifiers significantly improve link detection accuracy over existing methods.

Contribution

It introduces and empirically evaluates two novel approaches, Kashif and RICE_LRT, for legal requirements traceability, demonstrating their superior performance over baseline models.

Findings

Kashif achieves 63% F2 score, outperforming baselines by 21 pp.

RICE_LRT reaches 84% recall and 61% F2 score on GDPR-related documents.

Legal artifact-specific techniques outperform general literature methods.

Abstract

New regulations are introduced to ensure software development aligns with ethical concerns and protects public safety. Showing compliance requires tracing requirements to legal provisions. Requirements traceability is a key task where engineers must analyze technical requirements against target artifacts, often within limited time. Manually analyzing complex systems with hundreds of requirements is infeasible. The legal dimension adds challenges that increase effort. In this paper, we investigate two automated solutions based on language models, including large ones (LLMs). The first solution, Kashif, is a classifier that leverages sentence transformers and semantic similarity. The second solution, RICE_LRT, prompts a recent LLM based on RICE, a prompt engineering framework. Using a publicly available benchmark dataset, we empirically evaluate Kashif and compare it against seven baseline classifiers from the literature (LSI, LDA, GloVe, TraceBERT, RoBERTa, and LLaMa). Kashif can identify trace links with F2 score of 63%, outperforming the best baseline by a substantial margin of 21 percentage points (pp) in F2 score. On a newly created and more complex requirements document traced to the European general data protection regulation (GDPR), RICE_LRT outperforms Kashif and baseline prompts in the literature by achieving an average recall of 84% and F2 score of 61%, improving the F2 score by 34 pp compared to the best baseline prompt. Our results indicate that requirements traceability in legal contexts cannot be adequately addressed by techniques proposed in the literature that are not specifically designed for legal artifacts. Furthermore, we demonstrate that our engineered prompt outperforms both classifier-based approaches and baseline prompts.