Coverage Guided Testing for Recurrent Neural Networks

TL;DR

This paper introduces a coverage guided testing approach for LSTM-based RNNs, utilizing novel metrics and genetic algorithms to detect defects more effectively and interpretably than existing methods.

Contribution

It develops a new set of test metrics and a genetic algorithm for systematic RNN testing, with an implementation that outperforms state-of-the-art tools.

Findings

TestRNN outperforms DeepStellar and attack-based methods.

Finer temporal semantics improve defect detection.

Provides interpretable testing results.

Abstract

Recurrent neural networks (RNNs) have been applied to a broad range of applications, including natural language processing, drug discovery, and video recognition. Their vulnerability to input perturbation is also known. Aligning with a view from software defect detection, this paper aims to develop a coverage guided testing approach to systematically exploit the internal behaviour of RNNs, with the expectation that such testing can detect defects with high possibility. Technically, the long short term memory network (LSTM), a major class of RNNs, is thoroughly studied. A family of three test metrics are designed to quantify not only the values but also the temporal relations (including both step-wise and bounded-length) exhibited when LSTM processing inputs. A genetic algorithm is applied to efficiently generate test cases. The test metrics and test case generation algorithm are implemented into a tool TestRNN, which is then evaluated on a set of LSTM benchmarks. Experiments confirm that TestRNN has advantages over the state-of-art tool DeepStellar and attack-based defect detection methods, owing to its working with finer temporal semantics and the consideration of the naturalness of input perturbation. Furthermore, TestRNN enables meaningful information to be collected and exhibited for users to understand the testing results, which is an important step towards interpretable neural network testing.