Understanding Bugs in Multi-Language Deep Learning Frameworks

TL;DR

This study analyzes nearly 1500 bugs in multi-language deep learning frameworks, classifying bug types, assessing their impacts, and highlighting the complexity and prevalence of multi-programming-language bugs to improve bug management.

Contribution

It provides a comprehensive manual analysis of bugs in three major MPL deep learning frameworks, revealing bug types, impacts, and the complexity of MPL bug fixes, which was previously underexplored.

Findings

28.6%, 31.4%, and 16.0% of bugs are MPL bugs in MXNet, PyTorch, and TensorFlow.

MPL bug fixes have higher code change complexity than SPL bug fixes.

MPL bug fixes in PyTorch have longer open times and greater communication complexity.

Abstract

Deep learning frameworks (DLFs) have been playing an increasingly important role in this intelligence age since they act as a basic infrastructure for an increasingly wide range of AIbased applications. Meanwhile, as multi-programming-language (MPL) software systems, DLFs are inevitably suffering from bugs caused by the use of multiple programming languages (PLs). Hence, it is of paramount significance to understand the bugs (especially the bugs involving multiple PLs, i.e., MPL bugs) of DLFs, which can provide a foundation for preventing, detecting, and resolving bugs in the development of DLFs. To this end, we manually analyzed 1497 bugs in three MPL DLFs, namely MXNet, PyTorch, and TensorFlow. First, we classified bugs in these DLFs into 12 types (e.g., algorithm design bugs and memory bugs) according to their bug labels and characteristics. Second, we further explored the impacts of different bug types on the development of DLFs, and found that deployment bugs and memory bugs negatively impact the development of DLFs in different aspects the most. Third, we found that 28.6%, 31.4%, and 16.0% of bugs in MXNet, PyTorch, and TensorFlow are MPL bugs, respectively; the PL combination of Python and C/C++ is most used in fixing more than 92% MPL bugs in all DLFs. Finally, the code change complexity of MPL bug fixes is significantly greater than that of single-programming-language (SPL) bug fixes in all the three DLFs, while in PyTorch MPL bug fixes have longer open time and greater communication complexity than SPL bug fixes. These results provide insights for bug management in DLFs.