Metamorphic Testing of a Deep Learning based Forecaster

TL;DR

This paper applies metamorphic testing to a deep learning-based system for predicting system outages, uncovering previously unknown issues and effectively detecting injected bugs, thereby enhancing the reliability of such forecasting models.

Contribution

It introduces 19 metamorphic relations for testing deep learning forecasting applications and demonstrates their effectiveness in identifying issues and bugs.

Findings

Uncovered 8 previously unknown issues in the application.

Detected 65.9% of injected bugs through metamorphic relations.

Validated the approach on real and mutated models.

Abstract

In this paper, we present the Metamorphic Testing of an in-use deep learning based forecasting application. The application looks at the past data of system characteristics (e.g. `memory allocation') to predict outages in the future. We focus on two statistical / machine learning based components - a) detection of co-relation between system characteristics and b) estimating the future value of a system characteristic using an LSTM (a deep learning architecture). In total, 19 Metamorphic Relations have been developed and we provide proofs & algorithms where applicable. We evaluated our method through two settings. In the first, we executed the relations on the actual application and uncovered 8 issues not known before. Second, we generated hypothetical bugs, through Mutation Testing, on a reference implementation of the LSTM based forecaster and found that 65.9% of the bugs were caught through the relations.