An Empirical Study on the Effectiveness of Large Language Models for SATD Identification and Classification

TL;DR

This study evaluates the effectiveness of large language models, especially Flan-T5 variants, in identifying and classifying Self-Admitted Technical Debt in software, showing they outperform traditional models in some settings but have limitations in others.

Contribution

It provides a comprehensive empirical analysis of LLMs for SATD tasks, comparing fine-tuning and in-context learning approaches across different model sizes and settings.

Findings

Fine-tuned LLMs outperform non-LLM baselines in SATD identification.

For SATD classification, larger models with contextual info improve performance.

Few-shot in-context learning can surpass fine-tuned smaller models in classification.

Abstract

Self-Admitted Technical Debt (SATD), a concept highlighting sub-optimal choices in software development documented in code comments or other project resources, poses challenges in the maintainability and evolution of software systems. Large language models (LLMs) have demonstrated significant effectiveness across a broad range of software tasks, especially in software text generation tasks. Nonetheless, their effectiveness in tasks related to SATD is still under-researched. In this paper, we investigate the efficacy of LLMs in both identification and classification of SATD. For both tasks, we investigate the performance gain from using more recent LLMs, specifically the Flan-T5 family, across different common usage settings. Our results demonstrate that for SATD identification, all fine-tuned LLMs outperform the best existing non-LLM baseline, i.e., the CNN model, with a 4.4% to 7.2% improvement in F1 score. In the SATD classification task, while our largest fine-tuned model, Flan-T5-XL, still led in performance, the CNN model exhibited competitive results, even surpassing four of six LLMs. We also found that the largest Flan-T5 model, i.e., Flan-T5-XXL, when used with a zero-shot in-context learning (ICL) approach for SATD identification, provides competitive results with traditional approaches but performs 6.4% to 9.2% worse than fine-tuned LLMs. For SATD classification, few-shot ICL approach, incorporating examples and category descriptions in prompts, outperforms the zero-shot approach and even surpasses the fine-tuned smaller Flan-T5 models. Moreover, our experiments demonstrate that incorporating contextual information, such as surrounding code, into the SATD classification task enables larger fine-tuned LLMs to improve their performance.