TFCheck : A TensorFlow Library for Detecting Training Issues in Neural Network Programs

TL;DR

TFCheck is a TensorFlow library designed to automatically detect training issues in neural network programs, addressing a gap where existing testing techniques assume bug-free training code.

Contribution

The paper introduces a catalog of verification routines and implements them in TFCheck to automatically identify training issues in ML programs.

Findings

TFCheck successfully detects training issues in real-world ML code.

The library identifies bugs in synthetic and mutant training programs.

Case study demonstrates TFCheck's effectiveness in practical scenarios.

Abstract

The increasing inclusion of Machine Learning (ML) models in safety critical systems like autonomous cars have led to the development of multiple model-based ML testing techniques. One common denominator of these testing techniques is their assumption that training programs are adequate and bug-free. These techniques only focus on assessing the performance of the constructed model using manually labeled data or automatically generated data. However, their assumptions about the training program are not always true as training programs can contain inconsistencies and bugs. In this paper, we examine training issues in ML programs and propose a catalog of verification routines that can be used to detect the identified issues, automatically. We implemented the routines in a Tensorflow-based library named TFCheck. Using TFCheck, practitioners can detect the aforementioned issues automatically. To assess the effectiveness of TFCheck, we conducted a case study with real-world, mutants, and synthetic training programs. Results show that TFCheck can successfully detect training issues in ML code implementations.