Automatic Static Bug Detection for Machine Learning Libraries: Are We There Yet?

TL;DR

This study evaluates the effectiveness of five static bug detection tools on machine learning libraries, revealing their limited success and highlighting areas for improvement to enhance bug detection in such specialized software.

Contribution

The paper provides a comprehensive analysis of static bug detectors' performance on ML libraries, identifying their limitations and suggesting directions for making these tools more effective.

Findings

Static bug detectors detect only 0.01% of bugs in ML libraries.

Flawfinder and RATS are the most effective among the tools tested.

Opportunities are discussed to improve static bug detection in ML software.

Abstract

Automatic detection of software bugs is a critical task in software security. Many static tools that can help detect bugs have been proposed. While these static bug detectors are mainly evaluated on general software projects call into question their practical effectiveness and usefulness for machine learning libraries. In this paper, we address this question by analyzing five popular and widely used static bug detectors, i.e., Flawfinder, RATS, Cppcheck, Facebook Infer, and Clang static analyzer on a curated dataset of software bugs gathered from four popular machine learning libraries including Mlpack, MXNet, PyTorch, and TensorFlow with a total of 410 known bugs. Our research provides a categorization of these tools' capabilities to better understand the strengths and weaknesses of the tools for detecting software bugs in machine learning libraries. Overall, our study shows that static bug detectors find a negligible amount of all bugs accounting for 6/410 bugs (0.01%), Flawfinder and RATS are the most effective static checker for finding software bugs in machine learning libraries. Based on our observations, we further identify and discuss opportunities to make the tools more effective and practical.