Self-Admitted Technical Debt in LLM Software: An Empirical Comparison with ML and Non-ML Software

TL;DR

This study empirically compares self-admitted technical debt in LLM-based systems with ML and non-ML software, revealing unique debt forms and dynamics in LLM development.

Contribution

It is the first empirical analysis of SATD in LLM systems, identifying new debt types and comparing their prevalence and evolution across different software paradigms.

Findings

LLM repositories accumulate SATD at similar rates to ML systems.

LLM repositories remain debt-free longer than ML repositories.

Three new forms of technical debt are identified in LLM development.

Abstract

Self-admitted technical debt (SATD), referring to comments flagged by developers that explicitly acknowledge suboptimal code or incomplete functionality, has received extensive attention in machine learning (ML) and traditional (Non-ML) software. However, little is known about how SATD manifests and evolves in contemporary Large Language Model (LLM)-based systems, whose architectures, workflows, and dependencies differ fundamentally from both traditional and pre-LLM ML software. In this paper, we conduct the first empirical study of SATD in the LLM era, replicating and extending prior work on ML technical debt to modern LLM-based systems. We compare SATD prevalence across LLM, ML, and non-ML repositories across a total of 477 repositories (159 per category). We perform survival analysis of SATD introduction and removal to understand the dynamics of technical debt across different development paradigms. Surprisingly, despite their architectural complexity, our results reveal that LLM repositories accumulate SATD at similar rates to ML systems (3.95% vs. 4.10%). However, we observe that LLM repositories remain debt-free 2.4x longer than ML repositories (a median of 492 days vs. 204 days), and then start to accumulate technical debt rapidly. Moreover, our qualitative analysis of 377 SATD instances reveals three new forms of technical debt unique to LLM-based development that have not been reported in prior research: Model-Stack Workaround Debt, Model Dependency Debt, and Performance Optimization Debt. Finally, by mapping SATD to stages of the LLM development pipeline, we observe that debt concentrates