How Far Have LLMs Come Toward Automated SATD Taxonomy Construction?

TL;DR

This paper explores how large language models can assist in semi-automating the creation of SATD taxonomies across various software domains, significantly reducing manual effort and cost.

Contribution

It introduces a structured LLM-driven pipeline for SATD taxonomy construction and demonstrates its effectiveness across multiple domains with minimal time and expense.

Findings

Successfully recovered domain-specific categories

Completed taxonomy generation in under two hours

Cost less than $1 per dataset

Abstract

Technical debt refers to suboptimal code that degrades software quality. When developers intentionally introduce such debt, it is called self-admitted technical debt (SATD). Since SATD hinders maintenance, identifying its categories is key to uncovering quality issues. Traditionally, constructing such taxonomies requires manually inspecting SATD comments and surrounding code, which is time-consuming, labor-intensive, and often inconsistent due to annotator subjectivity. In this study, we investigated to what extent large language models (LLMs) could generate SATD taxonomies. We designed a structured, LLM-driven pipeline that mirrors the taxonomy construction steps researchers typically follow. We evaluated it on SATD datasets from three domains: quantum software, smart contracts, and machine learning. It successfully recovered domain-specific categories reported in prior work, such as Layer Configuration in machine learning. It also completed taxonomy generation in under two hours and for less than $1, even on the largest dataset. These results suggest that, while full automation remains challenging, LLMs can support semi-automated SATD taxonomy construction. Furthermore, our work opens up avenues for future work, such as automated taxonomy generation in other areas.