A Taxonomy of Self-Admitted Technical Debt in Deep Learning Systems

TL;DR

This paper presents a taxonomy of self-admitted technical debt in deep learning systems, highlighting its categories, characteristics, and the inadequacy of current static analysis tools in addressing it.

Contribution

It introduces a novel taxonomy of DL-specific SATD based on empirical analysis of open-source projects, revealing unique categories and the need for specialized support tools.

Findings

DL-specific SATD often relates to suboptimal choices, not bugs.

Current static analysis tools are ineffective for DL SATD.

A taxonomy with 7 categories and 41 subcategories was developed.

Abstract

The development of Machine Learning (ML)- and, more recently, of Deep Learning (DL)-intensive systems requires suitable choices, e.g., in terms of technology, algorithms, and hyper-parameters. Such choices depend on developers' experience, as well as on proper experimentation. Due to limited time availability, developers may adopt suboptimal, sometimes temporary choices, leading to a technical debt (TD) specifically related to the ML code. This paper empirically analyzes the presence of Self-Admitted Technical Debt (SATD) in DL systems. After selecting 100 open-source Python projects using popular DL frameworks, we identified SATD from their source comments and created a stratified sample of 443 SATD to analyze manually. We derived a taxonomy of DL-specific SATD through open coding, featuring seven categories and 41 leaves. The identified SATD categories pertain to different aspects of DL models, some of which are technological (e.g., due to hardware or libraries) and some related to suboptimal choices in the DL process, model usage, or configuration. Our findings indicate that DL-specific SATD differs from DL bugs found in previous studies, as it typically pertains to suboptimal solutions rather than functional (\eg blocking) problems. Last but not least, we found that state-of-the-art static analysis tools do not help developers avoid such problems, and therefore, specific support is needed to cope with DL-specific SATD.