An Empirical Study on Bugs Inside PyTorch: A Replication Study

TL;DR

This study empirically analyzes bugs in the PyTorch deep learning library, comparing their causes, symptoms, and fixes to those in TensorFlow, revealing that PyTorch bugs resemble traditional software bugs more than deep learning-specific issues.

Contribution

It provides a detailed characterization of PyTorch bugs, including their causes, symptoms, and fix patterns, and compares these findings with TensorFlow to highlight similarities and differences.

Findings

PyTorch bugs are similar to traditional software bugs.

Bug causes and fix patterns are consistent across different deep learning libraries.

PyTorch bugs are less related to deep learning-specific issues.

Abstract

Software systems are increasingly relying on deep learning components, due to their remarkable capability of identifying complex data patterns and powering intelligent behaviour. A core enabler of this change in software development is the availability of easy-to-use deep learning libraries. Libraries like PyTorch and TensorFlow empower a large variety of intelligent systems, offering a multitude of algorithms and configuration options, applicable to numerous domains of systems. However, bugs in those popular deep learning libraries also may have dire consequences for the quality of systems they enable; thus, it is important to understand how bugs are identified and fixed in those libraries. Inspired by a study of Jia et al., which investigates the bug identification and fixing process at TensorFlow, we characterize bugs in the PyTorch library, a very popular deep learning framework. We investigate the causes and symptoms of bugs identified during PyTorch's development, and assess their locality within the project, and extract patterns of bug fixes. Our results highlight that PyTorch bugs are more like traditional software projects bugs, than related to deep learning characteristics. Finally, we also compare our results with the study on TensorFlow, highlighting similarities and differences across the bug identification and fixing process.