From Code Changes to Quality Gains: An Empirical Study in Python ML Systems with PyQu

TL;DR

This study analyzes how specific code changes in Python ML systems impact software quality, introducing PyQu to identify quality-enhancing commits with high accuracy across a large dataset.

Contribution

The paper presents PyQu, a novel tool leveraging software metrics to detect quality-improving code changes in Python ML projects, revealing 41% previously unknown improvements.

Findings

PyQu achieves an average F1 score of 0.84 in identifying quality-enhancing commits.

Identified 61 code changes that directly improve software quality, classified into 13 categories.

41% of the detected quality improvements are new and not found by existing tools.

Abstract

In an era shaped by Generative Artificial Intelligence for code generation and the rising adoption of Python-based Machine Learning systems (MLS), software quality has emerged as a major concern. As these systems grow in complexity and importance, a key obstacle lies in understanding exactly how specific code changes affect overall quality-a shortfall aggravated by the lack of quality assessment tools and a clear mapping between ML systems code changes and their quality effects. Although prior work has explored code changes in MLS, it mostly stops at what the changes are, leaving a gap in our knowledge of the relationship between code changes and the MLS quality. To address this gap, we conducted a large-scale empirical study of 3,340 open-source Python ML projects, encompassing more than 3.7 million commits and 2.7 trillion lines of code. We introduce PyQu, a novel tool that leverages low level software metrics to identify quality-enhancing commits with an average accuracy, precision, and recall of 0.84 and 0.85 of average F1 score. Using PyQu and a thematic analysis, we identified 61 code changes, each demonstrating a direct impact on enhancing software quality, and we classified them into 13 categories based on contextual characteristics. 41% of the changes are newly discovered by our study and have not been identified by state-of-the-art Python changes detection tools. Our work offers a vital foundation for researchers, practitioners, educators, and tool developers, advancing the quest for automated quality assessment and best practices in Python-based ML software.