Understanding Self-Admitted Technical Debt in Test Code: An Empirical Study

TL;DR

This empirical study investigates the presence, types, and implications of Self-Admitted Technical Debt in test code, revealing its distribution, categorization, and the effectiveness of machine learning models for automatic classification.

Contribution

It fills the research gap by analyzing SATD in test code, proposing new categories, and developing a CodeBERT-based model for automatic classification of SATD comments.

Findings

SATD is prevalent in test code but not linked to test smells.

New categories of SATD types in test code are identified.

CodeBERT model outperforms others in classifying SATD comments.

Abstract

Developers often opt for easier but non-optimal implementation to meet deadlines or create rapid prototypes, leading to additional effort known as technical debt to improve the code later. Oftentimes, developers explicitly document the technical debt in code comments, referred to as Self-Admitted Technical Debt (SATD). Numerous researchers have investigated the impact of SATD on different aspects of software quality and development processes. However, most of these studies focus on SATD in production code, often overlooking SATD in the test code or assuming that it shares similar characteristics with SATD in production code. In fact, a significant amount of SATD is also present in the test code, with many instances not fitting into existing categories for the production code. This study aims to fill this gap and disclose the nature of SATD in the test code by examining its distribution and types. Moreover, the relation between its presence and test quality is also analyzed. Our empirical study, involving 17,766 SATD comments (14,987 from production code, 2,779 from test code) collected from 50 repositories, demonstrates that while SATD widely exists in test code, it is not directly associated with test smells. Our study also presents comprehensive categories of SATD types in the test code, and machine learning models are developed to automatically classify SATD comments based on their types for easier management. Our results show that the CodeBERT-based model outperforms other machine learning models in terms of recall and F1-score. However, the performance varies on different types of SATD.