BioDefect: The First Dataset for Defect Detection in Bioinformatics Software

TL;DR

BioDefect is a novel dataset specifically designed for defect detection in bioinformatics software, including complete source code and improving detection performance significantly.

Contribution

Introduction of BioDefect, the first dataset tailored for bioinformatics software defect detection, addressing limitations of existing datasets and enhancing model performance.

Findings

BioDefect improves defect detection F1-score by 29.61% to 38.04%.

BioDefect includes complete source code repositories for realistic defect scenarios.

Systematic assessment on nine language models demonstrates BioDefect's effectiveness.

Abstract

Software defect detection is a critical task in software engineering. However, no prior studies have specifically addressed defect detection in bioinformatics software. Given that the performance of defect detection tasks is primarily influenced by both models and datasets, our experiments controlled for model-related factors and confirmed the limitations of existing datasets in bioinformatics software. To address this issue, we introduce BioDefect, the first dataset specifically designed for defect detection in bioinformatics software, aiming to overcome the limitations of existing datasets in this context. Unlike prior datasets, BioDefect includes complete source code repositories, preserving the actual contextual information of defective code, thereby more accurately reflecting real-world defect scenarios in bioinformatics software. Additionally, BioDefect mitigates issues related to label inconsistency and data leakage, ensuring high data quality and experimental reliability. To evaluate the effectiveness of BioDefect, we conduct a systematic assessment on nine language models (LMs), including DeepSeek-R1. The results demonstrate that BioDefect significantly enhances defect detection performance for bioinformatics software. Compared to existing datasets, BioDefect achieves an average F1-score improvement of 29.61% to 38.04% across all models, highlighting its superior advantages. This study fills a critical research gap in bioinformatics software defect detection, laying a foundation for future studies in this field and offering new insights for improving bioinformatics software quality assurance.