On Interpreting the Effectiveness of Unsupervised Software Traceability with Information Theory

TL;DR

This paper introduces an information theory-based approach, TraceXplainer, to evaluate and understand the fundamental limits of unsupervised software traceability techniques, revealing inherent information constraints affecting their effectiveness.

Contribution

It proposes the use of information theory metrics to assess traceability methods and demonstrates their application, uncovering information imbalances and limits in existing datasets and techniques.

Findings

Source code has 1.48 times more entropy than linked documentation.

Average mutual information is 4.81 bits, indicating limited link predictability.

Identified information-theoretic limits constrain the effectiveness of unsupervised traceability methods.

Abstract

Traceability is a cornerstone of modern software development, ensuring system reliability and facilitating software maintenance. While unsupervised techniques leveraging Information Retrieval (IR) and Machine Learning (ML) methods have been widely used for predicting trace links, their effectiveness remains underexplored. In particular, these techniques often assume traceability patterns are present within textual data - a premise that may not hold universally. Moreover, standard evaluation metrics such as precision, recall, accuracy, or F1 measure can misrepresent the model performance when underlying data distributions are not properly analyzed. Given that automated traceability techniques tend to struggle to establish links, we need further insight into the information limits related to traceability artifacts. In this paper, we propose an approach, TraceXplainer, for using information theory metrics to evaluate and better understand the performance (limits) of unsupervised traceability techniques. Specifically, we introduce self-information, cross-entropy, and mutual information (MI) as metrics to measure the informativeness and reliability of traceability links. Through a comprehensive replication and analysis of well-studied datasets and techniques, we investigate the effectiveness of unsupervised techniques that predict traceability links using IR/ML. This application of TraceXplainer illustrates an imbalance in typical traceability datasets where the source code has on average 1.48 more information bits (i.e., entropy) than the linked documentation. Additionally, we demonstrate that an average MI of 4.81 bits, loss of 1.75, and noise of 0.28 bits signify that there are information-theoretic limits on the effectiveness of unsupervised traceability techniques. We hope these findings spur additional research on understanding the limits and progress of traceability research.