Automated Detection of Algorithm Debt in Deep Learning Frameworks: An Empirical Study

TL;DR

This paper empirically investigates automated detection of Algorithm Debt in deep learning frameworks using various ML/DL models and feature extraction techniques to improve early identification of technical debt.

Contribution

It introduces an empirical study exploring multiple feature extraction methods and advanced language models for detecting Algorithm Debt in deep learning source code comments.

Findings

Feature enrichment improves detection accuracy

Language models like ROBERTA and ALBERTv2 enhance detection performance

Enriched datasets with AD-related terms aid in early debt identification

Abstract

Context: Previous studies demonstrate that Machine or Deep Learning (ML/DL) models can detect Technical Debt from source code comments called Self-Admitted Technical Debt (SATD). Despite the importance of ML/DL in software development, limited studies focus on automated detection for new SATD types: Algorithm Debt (AD). AD detection is important because it helps to identify TD early, facilitating research, learning, and preventing the accumulation of issues related to model degradation and lack of scalability. Aim: Our goal is to improve AD detection performance of various ML/DL models. Method: We will perform empirical studies using approaches: TF-IDF, Count Vectorizer, Hash Vectorizer, and TD-indicative words to identify features that improve AD detection, using ML/DL classifiers with different data featurisations. We will use an existing dataset curated from seven DL frameworks where comments were manually classified as AD, Compatibility, Defect, Design, Documentation, Requirement, and Test Debt. We will explore various word embedding methods to further enrich features for ML models. These embeddings will be from models founded in DL such as ROBERTA, ALBERTv2, and large language models (LLMs): INSTRUCTOR and VOYAGE AI. We will enrich the dataset by incorporating AD-related terms, then train various ML/DL classifiers, Support Vector Machine, Logistic Regression, Random Forest, ROBERTA, and ALBERTv2.