Practitioners' Perceptions of the Goals and Visual Explanations of Defect Prediction Models

TL;DR

This study explores software practitioners' perceptions of defect prediction models and visual explanation techniques, revealing preferences for certain explainability methods and emphasizing the need for improved understanding of these models in practice.

Contribution

It provides empirical insights into practitioners' perceptions of defect prediction goals and visual explanation techniques, highlighting preferences and areas for improvement.

Findings

82%-84% of practitioners find defect prediction goals useful

LIME is the most preferred technique for model explanations

ANOVA/VarImp is the second most preferred explanation method

Abstract

Software defect prediction models are classifiers that are constructed from historical software data. Such software defect prediction models have been proposed to help developers optimize the limited Software Quality Assurance (SQA) resources and help managers develop SQA plans. Prior studies have different goals for their defect prediction models and use different techniques for generating visual explanations of their models. Yet, it is unclear what are the practitioners' perceptions of (1) these defect prediction model goals, and (2) the model-agnostic techniques used to visualize these models. We conducted a qualitative survey to investigate practitioners' perceptions of the goals of defect prediction models and the model-agnostic techniques used to generate visual explanations of defect prediction models. We found that (1) 82%-84% of the respondents perceived that the three goals of defect prediction models are useful; (2) LIME is the most preferred technique for understanding the most important characteristics that contributed to a prediction of a file, while ANOVA/VarImp is the second most preferred technique for understanding the characteristics that are associated with software defects in the past. Our findings highlight the significance of investigating how to improve the understanding of defect prediction models and their predictions. Hence, model-agnostic techniques from explainable AI domain may help practitioners to understand defect prediction models and their predictions.