testRNN: Coverage-guided Testing on Recurrent Neural Networks

TL;DR

testRNN is a novel coverage-guided testing tool designed for LSTM-based recurrent neural networks, enabling verification, validation, and robustness assessment through mutation-based test generation and new coverage metrics.

Contribution

It introduces the first coverage-guided testing approach for LSTMs, including new structural coverage metrics and a tool for internal data flow analysis.

Findings

Empirically evaluates LSTM robustness using new coverage metrics

Provides a mutation-based test case generation method

Helps model designers understand internal data flow

Abstract

Recurrent neural networks (RNNs) have been widely applied to various sequential tasks such as text processing, video recognition, and molecular property prediction. We introduce the first coverage-guided testing tool, coined testRNN, for the verification and validation of a major class of RNNs, long short-term memory networks (LSTMs). The tool implements a generic mutation-based test case generation method, and it empirically evaluates the robustness of a network using three novel LSTM structural test coverage metrics. Moreover, it is able to help the model designer go through the internal data flow processing of the LSTM layer. The tool is available through: https://github.com/TrustAI/testRNN under the BSD 3-Clause licence.