Reducing Labeling Effort in Architecture Technical Debt Detection through Active Learning and Explainable AI

TL;DR

This paper presents a method to reduce labeling effort in detecting Architecture Technical Debt by combining keyword filtering, active learning, and explainable AI, achieving high accuracy with less annotation work.

Contribution

It introduces a novel approach integrating keyword filtering, active learning, and explainable AI to efficiently detect Architecture Technical Debt with reduced manual annotation.

Findings

Active learning with Breaking Ties improves F1-score to 0.72.

Annotation effort is reduced by 49%.

LIME is preferred for explanations due to clarity.

Abstract

Self-Admitted Technical Debt (SATD) refers to technical compromises explicitly admitted by developers in natural language artifacts such as code comments, commit messages, and issue trackers. Among its types, Architecture Technical Debt (ATD) is particularly difficult to detect due to its abstract and context-dependent nature. Manual annotation of ATD is costly, time-consuming, and challenging to scale. This study focuses on reducing labeling effort in ATD detection by combining keyword-based filtering with active learning and explainable AI. We refined an existing dataset of 116 ATD-related Jira issues from prior work, producing 57 expert-validated items used to extract representative keywords. These were applied to identify over 103,000 candidate issues across ten open-source projects. To assess the reliability of this keyword-based filtering, we conducted a qualitative evaluation of a statistically representative sample of labeled issues. Building on this filtered dataset, we applied active learning with multiple query strategies to prioritize the most informative samples for annotation. Our results show that the Breaking Ties strategy consistently improves model performance, achieving the highest F1-score of 0.72 while reducing the annotation effort by 49\%. In order to enhance model transparency, we applied SHAP and LIME to explain the outcomes of automated ATD classification. Expert evaluation revealed that both LIME and SHAP provided reasonable explanations, with the usefulness of the explanations often depending on the relevance of the highlighted features. Notably, experts preferred LIME overall for its clarity and ease of use.