A First Look at the Self-Admitted Technical Debt in Test Code: Taxonomy and Detection

TL;DR

This paper presents the first large-scale analysis of self-admitted technical debt in test code, classifies its types, and evaluates detection tools and LLMs, revealing current limitations in automatic identification methods.

Contribution

It introduces a taxonomy of SATD in test code and assesses the effectiveness of existing detection tools and LLMs, highlighting significant detection challenges.

Findings

Identified 615 SATD comments in test code from 1,000 projects.

Existing detection tools have moderate recall and poor precision.

LLMs perform poorly in detecting SATD in test code.

Abstract

Self-admitted technical debt (SATD) refers to comments in which developers explicitly acknowledge code issues, workarounds, or suboptimal solutions. SATD is known to significantly increase software maintenance effort. While extensive research has examined SATD in source code, its presence and impact in test code have received no focused attention, leaving a significant gap in our understanding of how SATD manifests in testing contexts. This study, the first of its kind, investigates SATD in test code by manually analyzing 50,000 comments randomly sampled from 1.6 million comments across 1,000 open-source Java projects. From this sample, after manual analysis and filtering, we identified 615 SATD comments and classified them into 14 distinct categories, building a taxonomy of test code SATD. To investigate whether test code SATD can be detected automatically, we evaluated existing SATD detection tools, as well as both open-source and proprietary LLMs. Among the existing tools, MAT performed the best, albeit with moderate recall. To our surprise, both open-source and proprietary LLMs exhibited poor detection accuracy, primarily due to low precision. These results indicate that neither existing approaches nor current LLMs can reliably detect SATD in test code. Overall, this work provides the first large-scale analysis of SATD in test code, a nuanced understanding of its types, and the limitations of current SATD detection methods. Our findings lay the groundwork for future research on test code-specific SATD.