SACS: A Code Smell Dataset using Semi-automatic Generation Approach

TL;DR

This paper presents SACS, a high-quality, semi-automatically generated dataset of over 30,000 labeled code smell samples across three categories, aimed at advancing machine learning-based code smell detection.

Contribution

The study introduces a semi-automatic approach combining automatic rules and manual review to create a large, reliable code smell dataset, addressing data quality issues in existing datasets.

Findings

Created an open-source dataset with over 10,000 samples per code smell category.

Applied structured review guidelines and an annotation tool for manual validation.

Facilitates future research in code smell detection and automated refactoring.

Abstract

Code smell is a great challenge in software refactoring, which indicates latent design or implementation flaws that may degrade the software maintainability and evolution. Over the past of decades, the research on code smell has received extensive attention. Especially the researches applied machine learning-technique have become a popular topic in recent studies. However, one of the biggest challenges to apply machine learning-technique is the lack of high-quality code smell datasets. Manually constructing such datasets is extremely labor-intensive, as identifying code smells requires substantial development expertise and considerable time investment. In contrast, automatically generated datasets, while scalable, frequently exhibit reduced label reliability and compromised data quality. To overcome this challenge, in this study, we explore a semi-automatic approach to generate a code smell dataset with high quality data samples. Specifically, we first applied a set of automatic generation rules to produce candidate smelly samples. We then employed multiple metrics to group the data samples into an automatically accepted group and a manually reviewed group, enabling reviewers to concentrate their efforts on ambiguous samples. Furthermore, we established structured review guidelines and developed a annotation tool to support the manual validation process. Based on the proposed semi-automatic generation approach, we created an open-source code smell dataset, SACS, covering three widely studied code smells: Long Method, Large Class, and Feature Envy. Each code smell category includes over 10,000 labeled samples. This dataset could provide a large-scale and publicly available benchmark to facilitate future studies on code smell detection and automated refactoring.